Going off slightly on a tiny tangent: if a human is not merely a set atoms (which get continually replaced anyway) but the PATTERN of them, then a molecular-level copy is essentially the same person. Or at least as functionality indistinguishable from them as makes no difference. Yes obviously the one who existed first "came first", which is a tautology hence uninteresting.

I think the issue with LLMs, for most people who aren't hip-deep into the techbro culture that seems to be basically worshiping the idea of playing the role of a god to create a new technological lifeform, is not whether they think or not, but whether they are USEFUL to them or not. And there's the rub: when an "AI" can search the Internet for you - especially nowadays when it is so enshittified, burying information behind multiple clicks, requests to sign-up to newsletters, JavaScript and even CSS interactivity that fights you at every step of the way trying to glean information from a website - and return something useful, like a recipe to cook food, then people enjoy using it! Until it turns out that the recipe is fatal to humans that consume the product - e.g. when it contains glue in it (arguably producing the best picturesque foodstuff products, suitable for visual presentation rather than edible consumption), or razor blades (that one representing straight-up poisoning the well scenarios, but since those can often represent sarcasm - even in material that long predates the emergence of LLMs hence was genuinely meant for a fellow human to read - it is not always so easy to detect).

My primary issue lies with how modern LLMs offers the form of an answer, while cheapening out on the substance. It thus flips the standard mode of evaluation, making it more rather than less difficult for someone to use it to find information. e.g. if you come across a page on the internet that is full of spelling errors, then the chances of its content being accurate in spite of that is next to nil. However, ChatGPT has straight up told people to do things that would literally kill them. I am used to an older model of computation where if you ask a computer what 1+1 is it will deterministically always tell you "2" (aside from issues such as hardware failure, including running low on battery power supply), whereas LLMs simply work according to an entirely different model, where the answer might sometimes be 3, or 1 again, or the message to KYS, etc. And I don't think that most people - especially literal children - realize the switch. They are bullshit generators - which I mean to say not only that in the derogatory sense, but more foundationally that is literally what they were designed to be, in being able to provide an answer in a certain format and when an answer is not forthcoming then to simply make shit up... by design.

So I would guess that the "is it thinking" questions might come more from those techbros who are in love with the idea of technological emergence of sentience, whereas most common folk simply want to be able to find information on the internet, especially now that SEOs have made Google notably less useful at that task. And for them, hearing about whether a tool is at the "genius" level or not is a proxy for how much they can place trust in the outcome of a query. Which is to say, not much - a position that Sam Altman is expressing himself so by no means a niche or even exclusively outsider concern.