Your exchange makes me think about the chinese room thought experiment.
The person inside the room has instructions and a dictonary they uses to translate chinese symbols into english words. They never leave the room and never interact with anyone. They just translate single words.
They don't understand chinese, but the output of the system (the room) gives the impression that there is thinking behind the process. If I remember correctly, it was an argument against the Turing test. The claim was that computers could be extremely efficient into constructing anwsers that seems to be backed by human consciousness/thinking.