I suspect it mostly relates how much code base there is on internet about the topic. For instance if you make it use a niche library, it is quite common that it makes up methods that don't exist in that library but exists in related libraries. When I point this out, it also hallucinates saying "It was removed after version bla". I also may not be using the most cutting edge LLM (mix of freely available and open source ones).

The other day I asked it whether if there is a python library that can do linear algebra over F2, for which it pointed me to the correct direction (Galois) but when I asked it examples of how to do certain stuff it just came up with wrong functions over and over again:

In the end it probably was still faster than google searching this but all of these errors happened one after the other in the span of five minutes, so yeah. If I recall correctly, some of its claims about these namespaces, versions etc were also hallucinated. For instance vstack also does not exist in Galois but it does exist in a very popular package called numpy that can do regular linear algebra (and which this package also uses behind the scenes).

I really don't feel like getting in depth about work on the weekend, sorry.

It's the models that make the difference. Up until like Nov it's all been really shit

It's the language and the domain. They work pretty well for the web and major languages (like top 15).

As soon as you get away from that they get drastically worse.

But I agree they're still unambiguously useful despite their occasional-to-regular bullshitting and mistakes. Especially for one-off scripts, and blank-page starts.