I think a more apt comparison would be a complicated magic 8 Ball, since it actually gives answers that seem to be relevant to the question, but your interpretation does the actualy mental work.

Talking about rubber duck intelligence, there is a two step "thinking then respond" that recent iterations of llms have started using. It is literally a rubber duck during the thinking phase. I downloaded a local llm with this feature and had it run and the cli did not hide the "thinking" once done. The end product was better quality than if it had tried to spit an answer immediately (I toggled thinking off and it definitely was dumber, so I think you are right for the generation of llms before "thinking"

Well a local model responding to a prompt on less than 20gb of vram (gaming computer) costs less power than booting up any recent AAA high fps game. The primary cost in power is r&d. Training the next model to be "the best" is an Arms race. 90% of power consumption is trying to train the next model in 100 different ways. China was able to build off of chat cpt tech and built a model that has similar abilities and smartness for only 5 million. I think I won't update my local model until there is actually more abilities in the next one.

You are right and I have seen some people try some clumsy solutions:

Have the llm summarize the chat context ( this loses information, but can make the llm appear to have a longer memory)

Have the llm repeat and update a todo list at the end of every prompt (this keeps it on task as it always has the last response in memory, BUT it can try to do 10 things and fails on step 1 but doesn't realize jt)

Have a llm trained with really high quality data then have it judge the randomness of the internet. This is meta cognition by humans using the llm as a tool for itself. It definitely can't do it by itself without becoming schizophrenic, but it can make some smart models from inconsistent and crappy/dirty datasets.

Again, you are right and I hate using the syncophantic clockwork-orange llms with no self awareness. I have some hope that they will get better.

I'm now curious what you mean by self updating model? For a model to be made, it needs billions of data points for it to make the first. A second can be made with the first judging the quality of the input into the 2nd. Some models already do this sifting in prep for the next model creation.

I think of it like humans: we have billions of sensory signals each day, and we judge what is important based on genetics, culture, and our chosen interpretation of morality e.g. hedonism considers effort/discomfort. If a llm has a billion sensory signals each day and has application specific hardware like our genetics then would the hardware finally allow you to call it intelligent?

I am turning into a philosopher in this comment thread! Soo... when is a chair a chair and not a stool?

LLMs are a completely separate technology from what AGI will be. There is no path from LLM to AGI that isn't a complete redo.