Chain of thought doesn’t quite align with this. It’s possible to predict something not in training if there are sufficient facts in the history to align the probability toward a conclusion. (the “teach a man to fish” approach) Reasoning chains were introduced to take advantage of this, but it’s also possible to prime facts through external sourcing that can be used to extend the predictions. (web searching, document traversal, etc.)

Your larger point remains accurate though. Introspection capabilities are functional but quite limited, and training bias can be quite pathological in a vacuum.

This binary distinction you're making doesn't exist in any serious scientific literature. I literally have a degree in information technology and have been following this subject for over 30 years now - there's no real distinction between "knowledge" and intelligence. In fact, compression itself is literally intelligence.