Simple questions to an LLM like: “review this procedure, is it internally self consistent” will find issues, real issues - usually minor - but undeniable when you read the review, the problem it describes is absolutely present in the document, page 14 says “X” and page 23 says “not X” and the context is basically identical - so which are we really supposed to do: “X” or “not X”? Well, that depends. Depends on what? Additional context and detail that the already too long 50 page procedure doesn’t address.
lol and I find issues like this in LLM output of only 1 or 2 pages. LLMs can't even be consistent across paragraphs, much less hundreds of pages of context window. what an insane claim.