So just because I didn’t write a book means I could never understand it from reading it?
Reading a book does not give you the same knowledge as the author.
The author may have had to choose between two different pieces of exclusive information to add to the book, they may have written information then had to remove it for some reason, or may have had to exclude certain information it altogether.
You would never know if they did, or the reasons behind why they did what they did if you only read the book.
Are editors useless in the world of book publishing? Then what is the point of code review?
The difference between your book analogy and human code review is you can actually talk with the author in a bi-directional channel to gain a deeper understanding of what they did, and why they did it.
Book editors and human code reviewers are the same in that respect, they develop the deeper understanding that simply reading does not provide.
When it comes to LLMs: that bi-directional channel doesn't exist, only the illusion of one.
LLMs don't have the capacity to think things through, their reasoning is actually post-hoc justifications for their previous output, not a result of a priori thought. There is no actual understanding that could be gleaned because it literally doesn't exist.
To train an LLM you need more data and feedback than can be manually tagged, so tags and feedback are generated automatically. This means that they are not being trained against ground truths, they're being trained against a confidence checker. To an LLM, there is literally no difference between a correct answer and a confident answer.
That's why they seem so stupid when they give an answer that is obviously incorrect.
They are confidence machines, they produce confident sounding answers.
The problem is that human psychology is wired not to be discern the difference when the output's falseness isn't immediately obvious.