You can make external tools available to the LLM and then provide it with instructions for when/how to use them.
So, for example, you'd describe to it that if someone asks it about math or chess, then it should generate JSON text according to a given schema and generate the command text to parametrize a script with it. The script can then e.g. make an API call to Wolfram Alpha or call into Stockfish or whatever.

This isn't going to be 100% reliable. For example, there's a decent chance of the LLM fucking up when generating the relatively big JSON you need for describing the entire state of the chessboard, especially with general-purpose LLMs which are configured to introduce some amount of randomness in their output.

But well, in particular, ChatGPT just won't have the instructions built-in for calling a chess API/program, so for this particular case, it is likely as dumb as auto-complete. It will likely have a math API hooked up, though, so it should be able to calculate a logarithm through such an external tool. Of course, it might still not understand when to use a logarithm, for example.

At first it blamed its poor performance on the icons used, but then they switched to chess notation and it still failed hard

It's not even that. It's not a chess AI or a AGI (which doesn't exist). It will speak and pretend to play, but has no memory of the exact position of the pieces nor the capability to plan several steps ahead. For ask intended and porpoises, it's like asking my toddler what's the time (she always says something that sounds like a time, but doesn't understand the concept of hours or what the time is)

The fact that somebody posted this on LinkedIn and not only wasn't shamed out of his job but there are several articles about it is truly infuriating.

SUVs aren't marketed as grass mowers. LLMs are marketed as AI with all the answers.

Made people click though didnt it.

It depends. Have you used it? If not - Yes! It does do . . . all the things.

If you have used it, I’m sorry that was incorrect. You simply need to pay for the upgraded subscription. Oh, and as a trusted insider now we can let you in on a secret - the next version of this thing is gonna be, like, wow! Boom shanka! Everyone else will be so far behind!

Clearly you didn't swing the hammer hard enough

When all you have (or you try to convince others that all they need) is a hammer, everything looks like a nail. I guess this shows that it isn't.

One of those Fisher-Price plastic hammers with the hole in the handle?

It doesn’t work without 200 hours of un-fucking

It probably consumes as much energy as a family house for a day just to come up with that program. That's what happens.

In fact, I did a Google search and didn't have any choice but to have an "AI" answer, even if I don't want it. Here's what it says:

Each ChatGPT query is estimated to use around 10 times more electricity than a traditional Google search, with a single query consuming approximately 3 watt-hours, compared to 0.3 watt-hours for a Google search. This translates to a daily energy consumption of over half a million kilowatts, equivalent to the power used by 180,000 US households. 

Little disappointed more people didn’t get this.

Hey I don't mean to ruin your day, but maybe you should Google what you just commented...