284
I Asked AI to Count My Carbs 27,000 Times. It Couldn’t Give Me the Same Answer Twice. (www.diabettech.com)
submitted 6 months ago by to c/technology@lemmy.world
70 comments fedilink hide all child comments
😂 1
 

Hacker News

you are viewing a single comment's thread
view the rest of the comments
[–] 31 points 6 months ago (5 children)

Waste of energy. It's like asking a person to estimate a non-trivial angle. Either use a model trained for that task, or don't bother.

  • permalink
  • fedilink
  • source
  • hideshow 5 child comments
    •
  • [–] 40 points 6 months ago (3 children)

    The point is they are advertising that these models can do it.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] -1 points 6 months ago* (last edited 6 months ago) (2 children)

    You’d expect the same answer each time. It’s the same photo, the same model, the same question. But you won’t get the same answer.

    I don't know what ads show that, but anyone who knows the first thing about LLMs knows you don't get the same answer twice.

    I'd get this expectation 5 years ago when most people weren't familiar with it, but come on... you don't need to feed it an image 500 times to see that.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 3 points 6 months ago (1 child)

    Technically, you can get the same answer twice from an LLM, but only when you control the full input. When a model is being run, a random seed/hash is applied to the input. If you run the model locally you could force the seed to always be the same so that you would always get the same answer for a given question.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 6 months ago

    Barely. Even with the code and seeds, it's still a struggle to do that. There's plenty of questions from people running pytorch and tensorflow models that can't reproduce results. Maybe you isolate enough variables that consecutive runs actually produce the same output, but the study is about commercial models. You'll never get deterministic output from those.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 12 points 6 months ago

    The point is that:

    1. It is being used for ut, even though it is obviously not capable of giving a reliable and realistic answer
    2. It allows this usage, even though it is dangerous and not within it's capabilities
    3. Each model gives answers that vary wildly, something that a human wouldn't do. A human wouldn't give you answers that are 10x more for the same question randomly.
  • permalink
  • fedilink
  • source
  • parent