Calculations will happily tell you that an acutely toxic drug is the best way to cure cancer.

The reason why that does not lead to catastrophe is that there are many checks and safety nets in place in order not to blindly trust any result.

The exact same approach can be applied to an LLM.