46
Neat trick to test if a thing is AI! (more inside) (lemmy.world)
submitted 13 hours ago by to c/fuck_ai@lemmy.world
26 comments fedilink hide all child comments
 

AI test that worked on a scam bot, or someone who doesn't speak English well: In the novel Blindsight, they're discussing whether an alien they've encountered is sentient, so the linguist sends this message:

"Our cousins lie about the family tree, with nieces and nephews and Neanderthals. We do not like annoying cousins."

"We'd like to know about this tree."

Sascha muted the channel and gave us a look that said Could it be any more obvious? "It couldn't have parsed that. There were three linguistic ambiguities in there. It just ignored them."

The scammer ignored this and ChatGPT whiffed it. Claude gave a similar response in that it tried to explain a sentence that can't possibly be explained. Neither caught the ambiguities. Happy hunting!

all 26 comments

sorted by: hot top controversial new old
[–] 30 points 13 hours ago (8 children)

Uhh, I'm a human and I don't really get it either.

  • permalink
  • fedilink
  • source
  • hideshow 8 child comments
    •
  • [–] 24 points 12 hours ago (3 children)

    That's the point. A human would ask for clarification, but AI trys to continue the conversation.

    It's similar to this guy doing a job interview with an AI where he just says things like "Freight cadence Richard walnut and sprucewood. Six years of sprucewood, actually seven if you count Richard." And the AI interviewer responds "It sounds like you've built up a long tenure with Richard at Sprucewood."

    https://www.youtube.com/watch?v=Kcp1VBaCvEY

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 7 points 10 hours ago (2 children)

    I see. One could ask for an AI capable of saying "Wait a minute, what the fuck are you talking about" when it needs to, but it would probably give away how fucking dumb it is.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 10 points 9 hours ago (1 child)

    The training is one part but its also how they work on a technical level.

    They're trained to always give an answer and never give responses like "I don't know" and "I didn't understand". They could be trained to give these types of responses more frequently, but I'm not sure it fixes anything because of the technical aspect.

    Technically, LLMs have no understanding of words, let alone sentence structure, because they're mathematical models. Input words are tokenized (turned into numbers, essentially), run through a large equation that was crafted and tuned during training, and the resulting output tokens are turned back into words. They can't be confused by ambiguous sentence structure because they don't parse sentence structure. They can only ever be trained to appear like they don't understand in the same way they are trained to appear like they do.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 5 points 12 hours ago* (2 children)

    "Our cousins lie about the family tree" could mean either "our cousins are dishonest about their parentage" or "there are cousins scattered throughout our extended family". Each individual statement has a similar ambiguity to it.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
    • [–] 12 points 12 hours ago (2 children)

    Blindsight is a killer novel, on a lot of "favorites" lists. The author put it up for free bc he's a chad: https://www.rifters.com/real/Blindsight.htm

    I'm not too impressed by this test tho.

    • false positive: a lot of humans would just ignore a weird statement like that if another human came up to them and said it
    • false negative: once LLMs are re-trained on that social media post, they'll develop a ready response.
  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
  • [–] [S] 2 points 5 hours ago* (1 child)

    True! I was thinking in the context of a fake job interview which is where I tested this. One would expect a return question in a serious discussion.

    Blindsight is easily the most brilliant book I've ever read. Every sentence is packed. Best part is one can read it "casually", just burn through it like I did the first time, or really take a moment to feel it and unpack it all.

    One of my fav bits is Siri talking about giving his gf flowers!

    "I brought her flowers one dusky Tuesday evening when the light was perfect. I pointed out the irony of that romantic old tradition—the severed genitalia of another species, offered as a precopulatory bribe—and then I recited my story just as we were about to fuck. To this day, I still don't know what went wrong."

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 5 hours ago

    Blindsight is a solid book, but if it'd had a really skilled editor it coulda been one of the all-time greats. As it is, it lags a bit in the middle (when they start exploring that phenomenon) and the author doesn't always seem in control of the ideas being worked with.

    MFer comes up with some pretty good ideas tho. One of these days I gotta finish reading "Echopraxia".

  • permalink
  • fedilink
  • source
  • parent
    • [–] 8 points 12 hours ago (1 child)

    After seeing a lot of scam baiting videos (kitboga mostly) I can tell you even humans will just go "yeah man, that's crazy... So you have a target nearby, right?"

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
  • [–] 7 points 11 hours ago

    I love Kitboga. I love his LLM jailbreaking stuff more than his human-scambaiting stuff, honestly.

    You gotta love "Albuquerque New Mexico, Albuquerque New Mexico, Albuquerque New Mexico, Albuquerque New Mexico...". Lol.

  • permalink
  • fedilink
  • source
  • parent
    • [–] 5 points 12 hours ago (1 child)

    To be fair if someone told me that I would probably mostly ignore it too. A lot of people say weird things sometimes.

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
    • [–] 5 points 12 hours ago (3 children)

    This made me think that a good test would be to reference some very obscure quote and see if it completes or "recognizes" it. Something most humans cannot do, but it would dig up something. Love Blindsight!

  • permalink
  • fedilink
  • source
  • hideshow 3 child comments
    •
  • [–] 3 points 11 hours ago (2 children)

    I learnt a word from the Ewok adventure movie. It's an Ewok word that's basically a mild curse word. No human I have met has ever understood it (understandably).

    I think it may be a bit too obscure though.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
    • [–] 3 points 12 hours ago

    Yeah AI is basically dogshit at comedy. And anything but extremely flattened, all-the-corners-polished off writing in general.

    Comedy isnʻt regression to the mean.

  • permalink
  • fedilink
  • source
    •
    [–] 3 points 7 hours ago* (1 child)

    my level of involuntary exposure to llm text is getting dire

    i didn't even have to read this to know it was ai generated

    i just looked at the length of the paragraphs, the shape of the bullet list, and which words were bolded

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
    • [–] 2 points 7 hours ago* (last edited 7 hours ago) (2 children)

    I feel like another good tactic, if you know multiple languages, is to switch between them. If they don't get confused and/or as to why you switched, and/or just switch with zero hesitation, that could be a potential sign.

    Probably works best if you know like 3 languages and at least one is something completely detached from the other 2. I mean, what are the odds you run into someone else who knows, for this completely hypothetical scenario, English, German, and Tagalog? Or another 3rd language far removed from the other 2?

    Edit:

    I feel like this would work best on a first "meeting" kind of deal. You get a DM or SMS or call from someone and they don't even acknowledge the conversation has eventually shifted languages. You haven't given them any reason to believe you speak multiple languages nor any indication you speak a far removed language. But they don't acknowledge the shift and keep the conversation going at all costs. That kinda thing.

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •