I can ask it how relativity works and it will tell me, I can ask it to explain something differently and it will, I can ask it to eli12 and it will dumb it down. Of course there is a maximum compression, that doesn't make what these learn not useful.
And you can train it to do whatever you want, be confident, be persuasive, be accurate, nothing inherent about the model makes it go one way or another. And if you glue them into another system containing verified facts (database or a wiki or something) then you can very easily get very high accuracy
And we are literally 5 years in since chatgpt dropped and these things have gone from "can finish the line of the function your working on" to "can do basically anything on a computer code wise" ..METR stopped doing their horizon benchmark because it got too expensive to develop longer horizon tasks and frontier models have saturated what they already have
I think this is something anti ai crowd miss a lot too, these models are currently as bad as they'll ever be, all the issues people have are just bugs that can and will be fixed ... Or left in depending on the wants of the wannabe overlords running the thing