Well, not sure what you mean by hallucinating, but there are some LLMs (chatgpt3.5-turbo-instruct) that could play chess at a good amateur level (1750 elo), and by studying them, some people found parameters in the model that seemed to represent the current state of the chess board (as in, they corresponded to the game, and changing them artificially made the LLM play as if the game as always been that way).

it’s never recalling information, it’s always hallucinating

With the exception of information you have put within the context window

https://generativeai.pub/what-is-the-context-window-ais-memory-limit-ce8a966f954a