Its pretty good at SQL

Eh 🤷‍♂️. Call it anecdotal if you want, but this is my experience:

The imbeciles who manage our ERP silently pushed some update that must have removed some indexes because suddenly several complex queries were one or two orders of magnitude slower.

While I got to work on optimizing them, a colleague who’s outsourced most of his thinking to LLMs got what according to him was “the most powerful model” (some flavour of claude, I assume) to take a look at one particular affected query relevant to the project he was working on.

To be fair it went for more or less the same approach I was taking (storing intermediate results in table variables, spending memory to save speed, though it used temporary tables instead for some reason), but in a significantly more inelegant and inefficient way than I would have (for instance, it put SELECT *s — from tables with hundreds of columns! — into the temporary tables, when we only needed a few columns; an absurd waste of memory; it also missed several other optimizations, being apparently happy with “good enough” instead of trying to be thorough).

Happy with the result he pushed it into production, while I kept working on the rest of the queries, managing to also bring them back to reasonable times.

Couple days later, ticket comes in about that specific query being slow again, for particular parameters. I’m not sure what was wrong (the design of the database is absolute unadulterated shit, but it belongs to the ERP anthropoids and we can look but not touch, and have to work around it), but in some specific cases some joins produced many duplicate results, leading to massive slowdowns. While diagnosing that I also noticed that some of the values it produced weren’t correct.

So I rewrote the whole thing based on the work I’d done with the other queries, and now it works properly and efficiently, and significantly faster (at least until the ERP assholes break something again).

TL;DR, sure, the machine quickly produced an apparently working solution (I don’t know what the cost of the tokens was, though), in much less time than I could, but the result was ugly, poorly thought out (which, I mean, since it was made by something that can’t think and just statistically processes language, I’m not sure to which extent can be a criticism of the technology, but is a criticism of those using it), less efficient than mine (and I wouldn’t call myself any kind of SQL master or guru, I just meddle), and failed in some particular cases. I was not impressed.