Had a similar experience with a question about google's googleads python lib. It told me how to do it, didn't work and it turns out it was impossible todo because of how it is structured so it just wasted 2-3 days of work.

This matches my experience when I've used midsize models connected to read-only tools.

It does not match my experience using a big model in an environment where it can generate and execute code when given instructions to base its answer on that. That pattern almost always results in a factual answer, and often, but not always a good one. The downside is you need to set up such an environment or pay a provider for one, and it's slow. If you have something else to do while you wait, it may improve your efficiency. If you don't, you might as well do the experiment yourself.

I sometimes wonder if some folks who hate AI use it poorly to show that it "fails half the time".

The AI has access to roughly the same resources as you - if you can't find docs or anything then it doesn't either. So yeah - for incompletely documented APIs it will perform poorly. It isn't magic (despite some marketing claims)...

But if you don't want to spend 20 mins digging through AWS's API docs to figure out how to list all EC2 instances and their volume mounts to tally disk usage or something it will produce workable code very quickly.

Funny that I'm seeing negative view on this post, and it's also closer to what I've found in the wild. Often (in special outside of the web/rest api world), the LLM fails far more often, like at least 3 in 10 is pure garbage, while the other 7 is maybe correct, but still needs refactor to be in the shape to use in the codebases that I've.