You hit the nail on the head. They produce output that mimics the appearance of a thoughtful response, but isn't that at all. LLMs do not actually think and do not have any concept of truth.

This is probably why things like ClickUp naming their AI tool "Brain" annoys me so much. It's designed partially as a way for organizations to get aggregated access to the major LLMs. So yeah, my former coworkers are getting LLM output from "Clickup Brain." What a marketing scam.

I've been wondering how people's attitudes toward LLMs would shift if society collectively changed the language we used about them to be more accurate. Maybe there wouldn't be so many people claiming "AI is great for research" and whatnot. Even then, though, I doubt people would fully get past the human tendency to trust confident-sounding language.