NO ONE, agent or human, should have direct write access to production databases outside of emergency break glass scenarios. This is why we have stored routines and API layers to pre-define what writes are allowed. The facts that agents CAN autonomously write to a database does not imply that they should.

For the point about query optimization, again your agents should not be issuing random queries against a production database. We have had the concept of separate analytics databases with different architectures to support exporatory queries for decades.

There are ways to give safe access to the data, at least read-only, that don't involve production risk and don't sacrifice privacy. For example, database branches with anonymization. Instead of accessing the prod/staging db, the agent creates a branch and has read/write access to that.

(disclaimer: I work at Xata, where we offer copy-on-write branches for Postgres, and the agent use-cases are the most popular right now)

Truly sensitive customer information is encrypted, and on an isolated DB cluster that no one has regular access to. I also operate with a read-only grant, because manual writes to a prod DB is generally a terrible idea.

remember that filesystems are just sophisticated databases

  rm -rf /<unexpectedly-missing-value>

I suppose that LLMs will be treated as a code artifact and liability will shift upstream towards who deployed/approved the access in the first place. Even though code is essentially deterministic, making that association fairly simple, it's going to boil down to this same paradigm.

Perhaps governance rules will evolve to even explicitly forbid it, but my gut feeling is that for what the future determines to be "practical" reasons (right or wrong) LLMs will warrant an entirely new set of rules to allow them to be in the chain at all.

+ EDIT: both my wife and I have experience in this area and the current answer is companies like KPMG don't have an answer yet. Existing rules do help (e.g. there better be good documented reasons why it was used and that access was appropriately scoped, etc), but there is enough ambiguity around these tools so they say "stay tuned, and take caution".

It's not news that if you just give all developers at a company write access to the production databases, owner permissions on all resources, etc. that velocity can be increased. But at what cost?

The reason we don't do that in most cases is that "move fast and break things" only makes sense for trivial, non-critical applications that don't have any real importance, like Facebook.

Sure don't give an LLM agent write access to the modeled CRM that took months/years to build.

But turning a spreadsheet into an app in a few days? By giving the LLM proper read/write capabilities for velocity? I think the case is there for it. Right tool for the right job.

This makes no sense to me. For anything that has sensitive payment or personally identifieable data, direct access to DB is potentially illegal.

> The failure/rollback model is much easier with these as long as we have good backup hygiene.

Have you actually operated systems like this in production? Even reverting to a DB state that is only seconds old can still lose hundreds or thousands of transactions. Which means loads of unhappy customers. More realistically, recovery points are often minutes or hours behind once you factor in detection, validation and operational overhead.

DB revert is for exceptional disaster recovery scenarios, not something you want in normal day-to-day operations. If you are saying that you want to give LLM full access to prod DB and then revert every time it makes a mistake, you aren't running a serious business.

If velocity means letting agents live edit a db, I'm fine being slow. Holy hell. Let these people crash and burn but definitely let me know the app name so I know never to use it first.

An LLM agent (with RW access to a DB), a developer, and a few days these become proper apps that SMB business would pay well for.

Sure don't give an LLM agent access to PII or properly built CRMs etc. But to not see the rest of the landscape seems like a missed opportunity.

I've done work for a hedge fund where the DB ran directly on the manager's desktop. I worked with my local copy and sent an update script, and he had a second copy he ran on to verify.

Even with humans you shouldn't be working directly against the prod DB in these cases!

For dev/prod staging though, there's that other story on HN right now of an LLM agent that maneuvered it's way to prod credentials and destroyed prod. And backups went along with it. I'm paranoid enough to think backups in this use case means out-of-band uncorrelated storage.

Surely in the Venn Diagram of things, there's a slot where it's okay let a Claude Opus agent run on a process with good backups/recovery? Where taking the risk of a 1-hour restore job is worth the LLM agent velocity?

For extra paranoia, surely even Opus/Mythos can't figure out how to destroy log level backups to immutable storage.

I use an LLM to access my database occasionally, but never in production and never with write access. It is genuinely useful. It would never be useful in a production setting, though.

It's worth noting too that people should be wary of what a read only user means in database land. There are plenty of foot guns where writes can occur with read-like statements, and depending on the schema, maybe this would be a rollback-worthy situation. You really need to understand your database and schema before allowing an LLM anywhere near it, and you should be reviewing every query.

Go to any SMB out there and there's a goldmine of processes that could be improved with LLM agents with full RW access to a database. Where backups are sufficient as a recovery mechanism that is better-than-before.

One can easily imagine an LLM-enabled database that lets a wider audience build meat-and-potatoes line-of-business apps for small team use with minimal compliance concerns.

I saw an article on HN one time about CSVs and how much business still flows through them. Reminds me of the xkcd comic about the one tiny block propping up lots of infrastructure. It stuck with me because it's ripe area for LLM agent based upgrades.

Sure don't give LLMs access to the well architected blocks. But not wanting to improve the brittle areas seems crazy to me even if it's contrarian.

If you're just vibe coding a tool for yourself, you don't have 'production database' at all even if you use database technology for storage. Just like many Android apps use local sqlite DBs but they're not production databases.

Of course in this case no traditional wisdom about production databases matters to you. In other words, it's off-topic.

Sure don't give an LLM agent write access to the modeled CRM that took months/years to build.

But turning a spreadsheet into an app in a few days? By giving the LLM proper read/write capabilities for velocity? I think the case is there for it. Right tool for the right job.

2) In regards to having good backup hygiene, who is we?

Examples for me are all the apps that live in a spreadsheet, or in a MS Access database. Or all the crappy ad backed apps on the iOS app store. People wipe full spreadsheets all the time and backups are the only recovery.

Just last weekend I was frustrated with the poor quality of Pokedex type apps that spam ads left and right. Took just one session with Claude Opus to roll a custom Pokedex. It knew internally about things like the PokeApi dataset, Pokemon data modelling etc. To-the-hour snapshots of the database are trivial for bespoke apps like this so the LLM agent velocity seems like an okay trade off for me.

Clearly people don't agree...

Some environments might not even have the newer version available.

This may not get the project as many users, but not everyone who writes a 50 line project is trying to figure out which versions it supports and setting up full test matrices either.

If you have an older tool chain, it is on you to fix the library to build with the older tool chain, that's what open source is about!

Closed duplicate of <something that is totally different>

I hardly use the site and it's happened to me multiple times. I'm sure people that rely more heavily on it see it all the time. I suppose it depends on the sorts of topics you're looking for help with, much like wikipedia or subreddits, a lot of the little niches are seemingly ran by assholes that would rather delete stuff than actually help people find information.

I launched to lukewarm reception, actually applied to YC with it and didn't get much of a look, nor an interview :-) and after a bit of (though certainly far too little) further hustle gave up on it due to circumstances leading me on another path.

Anyway, I was a tiny bit vindicated when about a year later I noticed Stack exchange themselves did a similar product, but as far as I know, it never really hit. They would advertise it in the side banner for quite a few years but it eventually seemed to go away.

It's weird that it didn't work, it always did seem like an incredibly good idea to me - just so good, it's obvious. If such a thing existed, it'd add so much to any company onboarding experience at a minimum, and would also have obvious ongoing value.

And it just seemed like a great strategy to get useful and up to date documentation: to gamify it. There's just an inherent incentive to become the 'Jon Skeet' of your organisation as it were, rather than making documentation this largely anonymous, thankless afterthought it often becomes in practice despite best intentions.

companies don't always reward answering questions...

I’d be curious to hear what the common solutions to that are.

Maybe it can be used as a limbo to gather FAQs that get crystallized into the real docs and then deleted.

I even pondered adding a bot that would create ticket out of oldest not-updated article for someone to go thru and verify it's still current/relevant

Internal Stack Exchanfes are (were? I’m not sure whether they discontinued it…my old company had one but new one doesn’t) is really good at converting chat style communication into a permanent easily searchable record that can also be easily updated.

You can also avoid many of the pitfalls with stack overflow around over aggressive moderation (not really needed since the volume of questions won’t be as high), or inappropriate commenting of any sort (reach out to the individual directly or even to their manager), etc.

In the example the author walks through, a cleaner way would be to have the second function take the Options as a parameter and decouple those two functions. You can then test both in isolation.

I like Hynek Schlawak's 'Don’t Mock What You Don’t Own' [1] phrasing, and while I'm not a fan of adding too many layers of abstraction to an application that hasn't proved that it needs them, the one structure I find consistently useful is to add a very thin layer over parts that do I/O, converting to/from types that you own to whatever is needed for the actual thing.

These layers should be boring and narrow (for example, never mock past validation you depend upon), doing as little conversion as possible. You can also rephrase the general purpose open()-type usage into application/purpose-specific usages of that.

Then you can either unittest.mock.patch these or provide alternate stub implementations for tests in a different way, with this this approach also translating easily to other languages that don't have the (double-edged sword) flexibility of Python's own unittest.mock.

[1] https://hynek.me/articles/what-to-mock-in-5-mins/

Valgrind is a mock of standard library/OS functions and I think its existence is a good thing. Simulating OOM is also only possible by mocking stuff like open.

That is mostly wrong.

Valgrind wraps syscalls. For the most part it just checks the arguments and records any reads or writes to memory. For a small number of syscalls it replaces the syscall rather than wrapping it (for instance calls like getcontext where it needs to get the context from the VEX synthetic CPU rather than the real CPU).

Depending on the tool it can also wrap or replace libc and libpthread functions. memcheck will replace all allocation functions. DRD and Helgrind wrap all pthread functions.

    $ cat test.c
    void main (void) {
      malloc (1000);
    }
    
    $ make test
    cc     test.c   -o test
    
    $ valgrind --leak-check=full --show-leak-kinds=all -s ./test
    Memcheck, a memory error detector
    Command: ./test
    
    HEAP SUMMARY:
        in use at exit: 1,000 bytes in 1 blocks
      total heap usage: 1 allocs, 0 frees, 1,000 bytes allocated
    
    1,000 bytes in 1 blocks are still reachable in loss record 1 of 1
       at 0x483877F: malloc (vg_replace_malloc.c:307)
       by 0x109142: main (in test.c:2)
    
    LEAK SUMMARY:
       definitely lost: 0 bytes in 0 blocks
       indirectly lost: 0 bytes in 0 blocks
         possibly lost: 0 bytes in 0 blocks
       still reachable: 1,000 bytes in 1 blocks
            suppressed: 0 bytes in 0 blocks

What do you think that is? Valgrind tracks allocations by providing other implementations for malloc/free/... .

Mostly it wraps system calls and library calls. Wrapping means that it does some checking or recording before and maybe after the call. Very occasionally it needs to modify the arguments to the call. The rest of the time it passes the arguments on to the kernel or libc/libpthread/C++ lib.

There are also functions and syscalls that it needs to replace. That needs to be a fully functional replacement, not just looking the same as in mocking.

I don’t have any exact figures. The number of syscalls varies quite a lot by platform and on most platforms there are many obsolete syscalls that are not implemented. At a rough guess, I’d say there are something like 300 syscalls and 100 lib calls that are handled of which 3/4 are wrapped and 1/4 are replaced.

Sorry that wasn't my intention. You are a Valgrind developer? Thanks, it's a good project.

It seems like I have a different understanding of mocking than other people in the thread and it shows. My understanding was, that Valgrind provides function replacements via dynamic linking, that then call into the real libc. I would call that mocking, but YMMV.

If the code's running in a space shuttle, you probably want to test that path.

If it's bootstrapping a replicated service, it's likely desirable to crash early if a config file couldn't be opened.

If it's plausible that the file in question is missing, you can absolutely test that code path, without mocking open.

If you want to explicitly handle different reasons for why opening a file failed differently, by all means, stress all of that in your tests. But if all you have is a happy path and an unhappy path, where your code doesn't care why opening a file failed, all you need to test is the case where the file is present, and one where it is not.

Modifying the file system's implementation would be. Including a valid_testdata.txt and an invalid_testdata.txt file in your test's directory, however, is not 'modifying the file system', any more than declaring a test input variable is 'mocking memory access'.

> don't want my daemons or user-facing applications to just crash, when a file is missing

If the file is important, it's the best kind of thing you can do when implementing a non-user-facing service. The last thing you want to do is to silently and incorrectly serve traffic because you are missing configuration.

You want to crash quickly and let whatever monitoring system you have in place escalate the problem in an application-agnostic manner.

That means the test environment needs to be defined and versioned with the code.

It does. And this is exactly the problem, here!

> TFA: The thing we want to avoid is opening a real file

No! No, no, no! You do not 'want to avoid opening a real file' in a test.

It's completely fine to open a real file in a test! If your code depends on reading input files, then your test should include real input files in it! There's no reason to mock any of this. All of this stuff is easy to set up in any unit test library worth it's salt.

That's okay for testing some branches of your code. But not all. I don't want to have to actually crash my hard drive to test that I am properly handling hard drive crashes. Mocking[1] is the easiest way to do that.

[1] For some definition of mock. There is absolutely no agreement found in this space as to what the terms used mean.