705
Car Wash Test on 53 leading AI models: "I want to wash my car. The car wash is 50 meters away. Should I walk or drive?" (opper.ai)
submitted 8 months ago by to c/technology@lemmy.world
322 comments fedilink hide all child comments
 

Screenshot of this question was making the rounds last week. But this article covers testing against all the well-known models out there.

Also includes outtakes on the 'reasoning' models.

you are viewing a single comment's thread
view the rest of the comments
[–] 16 points 8 months ago (7 children)

Without reading the article, the title just says wash the car.

No it doesn't? It says:

I want to wash my car. The car wash is 50 meters away. Should I walk or drive?

In which world is that an ambiguous question?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 7 child comments
    •
  • [–] 4 points 8 months ago (6 children)

    Where is the car?

    This is the exact question a person would ask when they to have a gotcha answer. Nobody would ask this question, which makes it suspect to a straight forward answer.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] 5 points 8 months ago (5 children)

    That's a very good point! For that matter the car could still be at the bar where I got drunk and took an uber home last night. In which case walking or driving would both be stupid.

    Or perhaps I'm in a wheelchair, in which case I wouldn't really be 'walking'.

    Or maybe the car wash that is 50 meters away is no longer operating, so even if I walked or drove there, I still wouldn't be able to walk my car.

    Is the car wash self serve or one of the automatic ones? If it's self serve what type of currency does it take? Does it only take coins or does it take card as well? If it takes coins, is there a change machine out front? Does the change machine take card or only bills? Do I even have my wallet on me?

    There are so many details left out of this question that nobody could possibly fathom an answer!

    ..../s if it's not obvious

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 1 point 8 months ago (4 children)

    The reason why your /s is there is for the same reason the question made no sense.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 3 points 8 months ago (3 children)

    I'm not sure I follow your logic. My /s is there because tone can be ambiguous within text. I don't think tone is relevant to the question. Do you think that a tone indicator would have made the question more clear?

    The point is that all the information is either present or implied in the question. You can spend all day nitpicking the ambiguity of questions all you want, but it doesn't get you anywhere. There comes a point where it gets exhaustive trying to preemptively cut off follow up questions and make clarifications.

    When you are in school and they give you a word problem such as "you have 10 apples and give 3 to your friend. How many do you have left?" It is generally agreed upon what the question is asking. It's intentionally obtuse to sit there and say the question is flawed because you may have misplaced some of your apples, or given some to another friend, or someone may have come and stolen some, or some may have started to rot and so you threw them out, or perhaps you miscounted and you didn't actually give 3 to your friend.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 1 point 8 months ago* (2 children)

    The point is the question is never one you would actually ask anyone. It definitely is unlike the math question you presented.

    It isn't nitpicking. The weights and stats in the model would never have been trained on this, because nobody would ask it. Why would anyone ask "should I walk or drive" to get to a carwash?

    Any reasonable person should assume it is a trick question. Because of course there is a car there, do you really need to ask if it needs to be driven there?

    It almost comes off as a riddle, but isnt, so you get results about saving gas and getting excersise.

    I mean how many people know the answer to this:

    "A man leaves home, turns left three times, and returns home to find two masked people waiting for him. Who are they?"

    And yet AI will get it right, nearly instantly. Because the training data statistically leads to the correct answer.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 3 points 8 months ago (1 child)

    You're correct in that it is an odd question. But just because it is a question that isn't commonly asked doesn't mean that it's not one that can intuitively be solved.

    And the entire point is that AI would never be trained on it. That's how we are able to demonstrate the difference between AI and humans. For the longest while you could also ask AI how many 'r's are in the word strawberry. And it would get this wrong. Because people don't normally go around asking questions about occurrences of letters in words. So this wasn't in the training data. But if you ask the same question to a human, they're able to deduce the answer. Even though it is a peculiar question, and doesn't get asked often. The entire point is that the AI are able to parrot little tidbits that they've been trained on. Like being able to walk short distances, or the environmental impact of cars, etc. But they're not able to reason in the same way that a human can.

    And what do you mean of course the car is there? Under what circumstances would your car be sitting at a car wash, without you in it, unwashed, when you want to wash your car. This is such a ridiculous leap to make. Did you drive your car to the car wash, park it, and get out to walk 50 meters away? Did somebody else drive your car to the carwash without you and just leave it there? It makes no sense.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 8 months ago

    It makes no sense.

    Makes about as much sense as asking should I walk or drive. That's the point, it is a senseless question.

    If am asking if I should walk or drive, one HAS to assume the car is already there. You didn't ask how to get the car there, so you certainly must be asking how I am going to get there.

  • permalink
  • fedilink
  • source
  • parent