Bullshit factory is very apt. I was using it for an open book exam and it gave answers entirely skewed to the way the question was asked.

For example, if I asked “is X bacteria a pathogen in Y disease”, it would say yes, it was a very bad pathogen.

If I asked “what effects does X bacteria have in this body system”, it said it was a beneficial bacteria.

Never trust the AI summary, you have to fully read the studies.

It does lie and hallucinate a lot, especially with biased context in the question (the bullshit part). The (biased) knowledge is hiding somewhere in its weights, it is just that it is sometimes quite hard to recover.

Your 40% depends a lot on how you ask the questions and the field of these questions. Humanity's last exam is a morr obiective benchmark for measuring the wide knowledge of LLMs.