Thought today's smbc comic was too funny not to share. Source

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 2 months ago (34 children)

Seems like a pretty stupid argument. If I sat in a room and wrote what a native speaker told me to write, I wouldn't understand the conversation, but the native speaker would. How is that different than doing what an LLM says?

  • source
  • parent
  • hideshow 34 child comments
  • [–] 30 points 2 months ago (8 children)

    The argument predates LLMs by decades, and in other forms it has existed for centuries. It doesn't directly corellate to modern LLMs. It is an example of how a mechanism could mimic human intelligence without actually being intelligent. It's a thought experiment. LLMs are not lookup tables, they are neural network based statistical models, but a similar argument can be made.

  • source
  • parent
  • hideshow 8 child comments
  • [–] 1 point 1 month ago

    Is there any actual material difference between what they're describing (the computer equivalent of a philosophical zombie, I would say), and a true agi? Their point seems rather vacuous to me (all I did was read the Wikipedia article).

  • source
  • parent
  • [+] -7 points 2 months ago (4 children)

    That's like saying the neural network in your head told you how to respond to my inquiry, so it's not intelligent.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 14 points 1 month ago (2 children)

    Welcome to the wacky world of "is consciousness an illusion, and are we actually deterministic machines" philosophy.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 8 points 1 month ago

    Well the neurons themself might be dumb cells, but together they can be quite smart, everyone is trying to draw a line of how many connections when a neural connection becomes sentient if it ever does become at all, which is a century old problem

    Are sparrows conscious? Cockroaches? Elephants?

  • source
  • parent
  • [–] 25 points 2 months ago (15 children)

    In the second scenario, there is no β€œnative speaker” that is telling you what to write. There is a list of instructions that you are following that is the same as the instructions that the computer was following with the first scenario.

    The argument is that if, by following the exact instructions that the computer is following, you still do not understand the Chinese conversation, why would you say that the machine understands?

    I haven’t thought about the idea enough to say I agree or disagree, but I hope I understood your comment and that the explanation helps.

  • source
  • parent
  • hideshow 15 child comments
  • [–] 4 points 1 month ago (8 children)

    I'd like to add a rebuttal I tend to agree with: You're only a component of a larger system, and the whole system (you + the room + your list of instructions) does encode some understanding, otherwise a chinese speaker wouldn't be able to interact with the room in any meaningful way.

    In almost any system, you can look at a specific part and argue that part doesn't fulfill a property that the entire system taken as one does.

  • source
  • parent
  • hideshow 8 child comments
  • [–] 10 points 1 month ago (3 children)

    But in this case you is unnecessary if a mechanism can perform that part, and understanding then completely doesn't rely on consciousness. This is a bit weird way to think about it, but it seems to be a semantic difference

  • source
  • parent
  • hideshow 3 child comments
  • [–] -1 points 1 month ago (2 children)

    No, you're still a necessary part of the system, since otherwise there is no interaction possible. And yes, I don't think understanding relies on consciousness.

    Where is the semantic difference?

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 1 month ago (1 child)

    Difference in if that is required and if β€˜understanding’ is a quality that a system can have. For instance I feel that a gravity coin sorter is made with understanding but does not have understanding

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 1 month ago

    It's not really possible to give a concrete answer without a concrete definition of "understanding". But unless you assume supernatural explanations, either systems must be able to "understand", or humans are also incapable of "understanding", since we ourselves are physical systems too.

    For instance I feel that a gravity coin sorter is made with understanding but does not have understanding

    I get the example and agree that such a coin sorter does not meaningfully have an understanding of gravity. But IMO it does have a (minimal) understanding of the different coins you're sorting, since it contains a mechanism for differentiating between their sizes.

    Imagine we replaced the gravity mechanism with a bunch of actuators controlled by organic neurons, which are connected to something like an eye. The neurons fire and cause the actuators to sort the coins. Now, if say the neurons just react to brightness levels it'd be hard to argue there's an understanding. But if the neurons are part of a real human brain, they can definitely understand the coins etc. and sort based on their understanding.

    As you take away more parts of said brain, when does it lose the understanding?

  • source
  • parent
  • [–] 4 points 1 month ago (3 children)

    It's still the same argument. If the conversation is carried on your end by a bunch of if-else instructions, there's no consciousness to be found.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 0 points 1 month ago (2 children)

    Yes, I think it proves that understanding does not require consciousness.

  • source
  • parent
  • hideshow 2 child comments
  • [–] -5 points 2 months ago (5 children)

    No, it doesn't make any sense at all. If I couldn't understand when a native speaker was replying, how does me not understanding when a neural network does it make it any less significant?

  • source
  • parent
  • hideshow 5 child comments
  • [–] 15 points 2 months ago (4 children)

    In the Chinese Room scenario, cards with Chinese written on them are inserted into a box.

    Inside there is a man who doesn't understand Chinese but has a lookup list of which symbols to write as a response to every possible sentence you can write on the inserted card.

    For anyone observing from outside the box it looks as if whatever is in the box is fluent in Chinese and can hold a conversation. But the man inside does not understand anything about Chinese.

    This is used as an analogy to explain that machines, very much like the man in the box, do not actually understand the data they are fed, nor the data they output, no matter how human the response might seem.

    It was an argument against the Turing test, which is about whether machines showing behaviour indistinguishable from a human is a sign that they can think.

    The Chinese Room argument is that they must also show understanding of the data itself, rather than just human-like responses, to say that they think.

  • source
  • parent
  • hideshow 4 child comments
  • [–] -3 points 2 months ago (3 children)

    That's not what I understood from the Wikipedia article. In any case, an LLM doesn't look up responses from a table.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 9 points 1 month ago (2 children)

    An LLM is a series of immutable nodes that pass results on to the next series of nodes until an output is generated, a functionality that could be perfectly imitated with a very large number of people seeing a series of values passed to them from other people, doing a bit of rigidly defined basic math with them, and passing it on to the next set of people who play the part of the next set of nodes.

    The fact that these nodes are immutable, have no persistent state, and only process information in one direction (input to output, and no more) means that an LLM has no persistence of experience. You could potentially argue that the information, as it exists, is a form of fleeting consciousness, but it would be so incredibly alien compared to the lived experience of humans that it would be impossible to relate to in any meaningful way - and it would also mean that every time an LLM is run, the consciousness is created and destroyed for every single word that it outputs... unless you can somehow argue that the consciousness is contained in the set of tokens that gets fed back into the LLM with each word it outputs, which would also mean that any string of text is the equivalent of latent consciousness, which is patently absurd.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 1 month ago

    You could potentially argue that the information, as it exists, is a form of fleeting consciousness, but it would be so incredibly alien compared to the lived experience of humans that it would be impossible to relate to in any meaningful way

    Curiously, one of the approaches to the consciousness, and a more Zen-ish one, is that clinging to a persistent β€˜self’ is erroneous and it's better seen as a process rather than an entity. Granted, it still assumes a β€˜self’ that is changed by each passing experience.

  • source
  • parent
  • [–] 8 points 2 months ago* (6 children)

    🀷 πŸˆβ€

    You have a cat. The cat meows at you. You do nothing. The cat keeps meowing at you. You offer it food as a guess. The cat stops meowing.

    The next day, the cat starts meowing at you again, but slightly different. You give it food. It keeps meowing. You give it a belly rub, the cat stops meowing.

    You have inadvertently learned the correct response to two types of cat meows. However, you do not speak cat. Nor does the cat speak human. You are just giving a response it expects for a given input.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 14 points 1 month ago* (last edited 1 month ago) (2 children)

    The difference between this and the Chinese Room is that you are attempting to understand what the cat is saying, and making a response that both you and the cat understand. You are attempting to communicate. The Chinese Room is many people who do not understand the input, do not understand the output, and have neither the capability nor the will to do either.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 2 points 1 month ago (1 child)

    ...but does the system that is the Chinese Room (the entire system taken as a whole) understand Chinese?

    Any given sufficiently small piece of your brain does not understand English, does no individual neuron in the language center of your brain understanding English mean that you do not understand English? If that understanding is held in the organization and structure of the brain but not in the individual parts, then why can't the Chinese Room understand Chinese despite no individual component understanding Chinese? That's a surprisingly difficult question to answer in a coherent fashion without asserting that knowledge in an organic brain is magical in some fashion, and nothing else counts.

    Wait until you consider the concept of philosophical zombies, which is all about pointing out the edges and contours of the hard problem of consciousness. The starting point of that discussion is essentially "Is someone who responds exactly like a regular person would but has no actual subjective experience conscious?" and gets stranger from there.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 3 points 1 month ago

    I always thought it was a bit tragically amusing that LLMs can fool people so readily, because it was essentially proof that P-zombies can exist. And we invented them.

    As for the Chinese Room and understanding Chinese, my argument has always been that the chart contains an understanding of Chinese - but as the chart itself cannot think, it does not understand Chinese in the way a person does, and as each person who is in the Chinese Room does not - and cannot - glean meaning from the input they receive or the output that they send out, they have no understanding at all. Understanding also implies being able to adapt the knowledge to new situations, and as the chart that drives the Chinese Room is immutable and cannot accomodate novel scenarios, it does not understand Chinese.

  • source
  • parent
  • [–] 3 points 1 month ago*

    Also doesn't cat only meows when trying to communicate with different species? They are not speaking cat either, they are just hoping you pick up what they want. (in this case it actually shows intent, something a LLM or said chinese room is not capable of doing)

  • source
  • parent
  • [–] 3 points 1 month ago

    Yeah. It basically states that because all your neurons and synapses in the human brain follow quantum physics and can be simulated by a formal program, and do not know what they are doing or know meaning, humans are not sentient but merely simulate being a mind.

    It's really infuriating how one of the dumbest and self evident fallacious thought experiments gets so famous. Because idiots keep repeating it like the sub-sentient LLMs they are.

    Because that is what we are really learning from LLMs. It's not shocking that LLMs aren't sentient, it's shocking that we now know what minds can talk seemingly intelligent without having to be sentient. Which explains the current state of the world.

  • source
  • parent