That stops it from making stuff up

No it doesn't. That's simply not how LLMs work. They're "making stuff up" 100% of the time. If the training data is good, the stuff they're making up more or less matches the training data. If the training data isn't good, they'll make up stuff that sounds plausible.