OC below by @HaraldvonBlauzahn@feddit.org

What called my attention is that assessments of AI are becoming polarized and somewhat a matter of belief.

Some people firmly believe LLMs are helpful. But programming is a logical task and LLMs can't think - only generate statistically plausible patterns.

The author of the article explains that this creates the same psychological hazards like astrology or tarot cards, psychological traps that have been exploited by psychics for centuries - and even very intelligent people can fall prey to these.

Finally what should cause alarm is that on top that LLMs can't think, but people behave as if they do, there is no objective scientifically sound examination whether AI models can create any working software faster. Given that there are multi-billion dollar investments, and there was more than enough time to carry through controlled experiments, this should raise loud alarm bells.

you are viewing a single comment's thread
view the rest of the comments
[+] -7 points 1 year ago* (last edited 1 year ago) (25 children)

What called my attention is that assessments of AI are becoming polarized and somewhat a matter of belief.

Proceed to write a belief as a statement in the following paragraph

If you think LLMs doesnt think (I won't argue that they arent extremely dumb), please define what is thinking, before continuing, and if your definition of thinking doesn't apply to humans, we won't be able to agree.

  • source
  • hideshow 25 child comments
  • [–] 19 points 1 year ago (13 children)

    The burden of proof is on those who say that LLMs do think.

  • source
  • parent
  • hideshow 13 child comments
  • [–] -1 points 1 year ago* (last edited 1 year ago) (12 children)

    I asked for your definition, I cannot prove something if we do not agree on a definition first.
    You also missread what I said, I did not said AI were thinking.
    The burden of proof is on the one who made an affirmation.
    I'm not the one who made an affirmation which field experts doesn't know the answer.
    But depending of your definition of thinking, some can be answered.

  • source
  • parent
  • hideshow 12 child comments
  • [–] 0 points 1 year ago* (11 children)

    I don't think y'all are disagreeing but maybe this sentence is somewhat confusing:

    If you think LLMs doesnt think (I won’t argue that they arent extremely dumb), please define what is thinking,

    Maybe the "doesnt" shouldn't be there.

  • source
  • parent
  • hideshow 11 child comments
  • [–] 0 points 1 year ago* (10 children)

    No it is here because that's what they claim.
    Nobody yet know how it work, we don't know how LLMs process information.
    Anyone who claim it really think, or it isn't thinking, is believing, this is not something the current ML field know.

  • source
  • parent
  • hideshow 10 child comments
  • [–] 4 points 1 year ago (9 children)

    Well, the neural network is given a prefix (series of tokens) and a token, and it spits out how likely is it that the token follows the prefix. Text is generated by calculating this probability for all known tokens, then picking one random, weighted based on the calculated probabilities.

  • source
  • parent
  • hideshow 9 child comments
  • [–] 0 points 1 year ago (8 children)

    And the brain is made out of neurons that sends electric signals between them and operate muscles.
    That doesnt explain how the brain think.

  • source
  • parent
  • hideshow 8 child comments
  • [–] 2 points 1 year ago (7 children)

    It allows us to conclude that an LLM doesn't "think" about what it is saying. Based on the mechanics, the LLM doesn't even know it's a participant in the conversation.

  • source
  • parent
  • hideshow 7 child comments
  • [–] 3 points 1 year ago (5 children)

    By that logic we also conclude that the human brain doesn't "think" about what it is saying.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 0 points 1 year ago (4 children)

    That does not follow. I can't speak for you, but I can tell if I'm involved in a conversation or not.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 1 point 1 year ago (2 children)

    And how do you know LLMs can't tell that they are involved in a conversation?

    Unless you think there is something non-computational in the human brain, then you must accept that computers are - in theory - capable of thinking. With the right software and sufficiently powerful hardware.

    Given that truth (which I think you can only avoid through religion or quantum quackery), you can't just say "it's only maths; it can't be thinking" because we know that maths can think.

    Do LLMs "think"? The definition of "think" is wooly enough and we understand them little enough that it's quite an assertion to say that they definitely don't.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 1 year ago (1 child)

    And how do you know LLMs can’t tell that they are involved in a conversation?

    It has no memory, for one. What makes you think that it does know its in a conversation?

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 1 year ago

    It has no memory, for one.

    It has very short term memory in the form of it's token context. Especially with something like Meta's Coconut.

    What makes you think that it does know its in a conversation?

    I don't really. Yet. But I also don't think that it is fundamentally impossible for LLMs to think, like you seem to. I also don't think the definition of the word "think" is so narrow that it requires that level of self-awareness. Do you think a mouse is really aware it is a mouse? What about a spider?

  • source
  • parent
  • [–] 6 points 1 year ago (5 children)

    I don't think the current common implementation of AI systems are "thinking" and I'll base my argument on Oxford's definitions of words. Thinking is defined as "the process of using one's mind to consider or reason about something". I'll ignore the word "mind" and focus on the word "reason". I don't think what AIs are doing counts as reasoning as defined by Oxford. Let's go to that definition: "the power of the mind to think, understand, and form judgments by a process of logic". I take issue with the assertion that they form judgments. For completeness, but I don't think it's definition is particularly relevant here, a judgment is: "the ability to make considered decisions or come to sensible conclusions".

    I think when you ask an LLM how many 'r's there are in Strawberry and questions along this line you can see they can't form judgments. These basic but obscure questions are where you see that the ability to form judgements isn't there. I would also add that if you "form judgments" you probably don't need to be reminded you formed a judgment immediately after forming one. Like if I ask an LLM a question, and it provides an answer, I can convince it that it was wrong whether or not I'm making junk up or not. I can tell it it made a mistake and it will blindly change it's answer whether it made a mistake or not. That also doesn't feel like it's able to reason or make judgments.

    This is where all the hype falls flat for me. It feels like sometimes it looks like a concrete wall, but occasionally that concrete wall is made of wet paper. You can see how impressive the tool is and how paper thin it is at the same time. It's cool, it's useful, it's fake, and that's ok. Just be aware of what the tool is.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 1 point 1 year ago (4 children)

    I think when you ask an LLM how many 'r’s there are in Strawberry and questions along this line you can see they can’t form judgments.

    Like a LLMs you are making the wrong affirmation based lacking knowledge.
    Current LLMs input, and output tokens, they dont ever see the individual letters, they see tokens, for straberry, they see 3 tokens:

    They dont have any information on what characters are in this tokens. So they come up with something. If you learned a language only by speaking, you'll be unable to write it down correctly (except purely phonetical systems), instead you'll come up with what you think the word should be written.

    I would also add that if you “form judgments” you probably don’t need to be reminded you formed a judgment immediately after forming one.

    You come up with the judgment before you are aware of it: https://www.unsw.edu.au/newsroom/news/2019/03/our-brains-reveal-our-choices-before-were-even-aware-of-them--st

    can tell it it made a mistake and it will blindly change it’s answer whether it made a mistake or not. That also doesn’t feel like it’s able to reason or make judgments.

    That's also how the brain can works, it come up with a plausible explanation after having the result.
    See the experience which are spoken about here: https://www.youtube.com/watch?v=wfYbgdo8e-8

    I showed the same behavior in humans of some behavior you observed in LLMs, does this means that by your definition, humans doesnt think ?

  • source
  • parent
  • hideshow 4 child comments
  • [–] 7 points 1 year ago (3 children)

    If the LLM could reason, shouldn't it be able to say "my token training prevents me from understanding the question as asked. I don't know how many 'r's there are in Strawberry, and I don't have a means of finding that answer"? Or at least something similar right? If I asked you what some word in a language you didn't know, you should be able to say "I don't know that word or language". You may be able to give me all sorts of reasons why you don't know it, and that's all fine. But you would be aware that you don't know and would be able to say "I don't know".

    If I understand you correctly, you're saying the LLM gets it wrong because it doesn't know or understand that words are built from letters because all it knows are tokens. I'm saying that's fine, but it should be able to reason that it doesn't know the answer, and say that. I assert that it doesn't know that it doesn't know what letters are, because it is incapable of coming to that judgement about its own knowledge and limitations.

    Being able to say what you know and what you don't know are critical to being able to solve logic problems. Knowing which information is missing and can be derived from known things, and which cannot be derived is key to problem solving based on reason. I still assert that LLMs cannot reason.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 0 points 1 year ago* (last edited 1 year ago) (2 children)

    I’m saying that’s fine, but it should be able to reason that it doesn’t know the answer, and say that.

    That is of course a big problem. They try to guess too much stuff, but it's also why it kinda works. Symbolics AI have the opposite problem, they are rarely useful, because they can't guess stuff, they are rooted in hard logic, and cannot come up with a reasonable guess.
    Now humans also try to guess stuff and sometimes get it wrong, it's required in order to produce results from our thinking and not be stuck in a state where we don't have enough data to do anything, like a symbolic AI.

    Now, this is becoming a spectrum, humans are somewhere in the middle of LLMs and symbolics AI.
    LLMs are not completely unable to say what they know and doesnt know, they are just extremely bad at it from our POV.

    The probleme with "does it think" is that it doesn't give any quantity or quality.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 1 year ago (1 child)

    Is the argument that LLMs are thinking because they make guesses when they don't know things combined with no provided quantity or quality to describe thinking?

    If so, I would suggest that the word "guessing" is doing a lot of heavy lifting here. The real question would be "is statistics guessing"? I would say guessing and statistics are not the same thing, and Oxford would agree. An LLM just grabs tokens based on training data on what word or token most likely comes next, it will just be using what the statistically most likely next token or word is. I don't think grabbing the highest likely next token counts as guessing. That feels very algorithmic and statistical to me. It is also possible I'm missing the argument still.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 1 year ago

    Is the argument that LLMs are thinking because they make guesses

    No, it's that you can't root the argument that they don't think over the fact they make stuff up, because humans too. You could root it in the amount of things it guess wrong, but it's extremely hard to measure.
    Again, I'm not claiming that they think, but that we don't know until one or the other is proven.
    Right now, thinking one, or the other is true, is belief.

  • source
  • parent
  • [–] 8 points 1 year ago (2 children)

    Since LLMs runs on CPUs with a lot of memory, do you agree that my calculator is thinking?

  • source
  • parent
  • hideshow 2 child comments
  • [–] -3 points 1 year ago

    You think computation is thinking ?
    I asked for your definition of thinking.
    The OP talked about belief, then made a statement using a word that is not precisely defined.
    If you think computation is thinking then by your definition the LLM is thinking.
    But that's your definition of thinking.

  • source
  • parent
  • [–] 3 points 1 year ago (1 child)

    ' Please succinctly answer a question of philosophy that has plagued mankind for thousands of years. can't? I win'

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 1 year ago

    Claiming LLMs can't think with the current informations available, and calling that not a belief, is claiming to have a response to this philosophy question.
    The only sensible answer is saying you don't know, or being aware and communicating that your statement is a belief.

  • source
  • parent