I completely agree that LLMs aren't intelligent. On the other hand, I'm not sure most of what we call intelligence in human behavior is any more intelligent than what LLMs do.

We are certainly capable of a class of intelligence that LLMs can't even approach, but most of us aren't using it most of the time. Even much (not all) of our boundary pushing science is just iterating algorithms that made the last discoveries.

Not to be pedantic, but the original use of the word intelligence in this context was “gathered digested information.”

Unfortunately, during the VC funding rounds for this, “intelligence” became the “thinky meat brain” type, and a marketing term associated with personhood, and the intense personalization along with it.