The software is not there. These systems don't actually think, they just pick one of the most likely next words (actually tokens), given the words they already have, in a loop until the most likely next token is 'end my turn, and let the user type'.

They do this in a way that appears surprisingly "smart" because there's a ton of data backing the calculation of which tokens are most likely, and they are trained to make humans think they are giving a good answer. (An extremely different goal from "Give a good answer.")

But they still make a ton of mistakes, known as hallucinations. This is unavoidable because of the statistical nature of the process. And because they aren't actually thinking and can't "care" if they generate something false.