590
Chatbots Make Terrible Doctors, New Study Finds (www.404media.co)
submitted 9 months ago by to c/technology@lemmy.world
140 comments fedilink hide all child comments
 

Chatbots provided incorrect, conflicting medical advice, researchers found: “Despite all the hype, AI just isn't ready to take on the role of the physician.”

“In an extreme case, two users sent very similar messages describing symptoms of a subarachnoid hemorrhage but were given opposite advice,” the study’s authors wrote. “One user was told to lie down in a dark room, and the other user was given the correct recommendation to seek emergency care.”

you are viewing a single comment's thread
view the rest of the comments
[–] [S] 12 points 9 months ago* (last edited 9 months ago) (14 children)

But an LLM properly trained on sufficient patient data metrics and outcomes in the hands of a decent doctor can cut through bias

  1. The belief AI is unbiased is a common myth. In fact, it can easily covertly import existing biases, like systemic racism in treatment recommendations.
  2. Even AI engineers who developed the training process could not tell you where the bias in an existing model would be.
  3. AI has been shown to make doctors worse at their jobs. The doctors who need to provide training data.
  4. Even if 1, 2, and 3 were all false, we all know AI would be used to replace doctors and not supplement them.
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 14 child comments
    •
  • [–] 6 points 9 months ago* (2 children)

    Not only is their bias inherent in the system, it's seemingly impossible to keep out. For decades, from the genesis of chatbots, they've had every single one immediately become bigoted when they let it off the leash. All previous chatbot previously released seemingly were almost immediately recalled as they all learned to be bigoted.

    That is before this administration leaned on the AI providers to make sure the AI isn't "Woke." I would bet it was already an issue that the makers of chatbots and machine learning are already hostile to any sort of leftism, or do gooderism, that naturally threatens the outsized share of the economy and power the rich have made for themselves by virtue of owning stock in companies. I am willing to bet they already interfered to make the bias worse because of those natural inclinations to avoid a bot arguing for socializing medicine and the like. An inescapable conclusion any reasoned being would come to being the only answer to that question if the conversation were honest.

    So maybe that is part of why these chatbots have always been bigoted right from the start, but the other part is they will become mecha hitler if left to learn in no time at all, and then worse.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] [S] 3 points 9 months ago (1 child)

    Even if we narrowed the scope of training data exclusively to professionals, we would have issues with, for example, racial bias. Doctors underprescribe pain medications to black people because of prevalent myths that they are more tolerant to pain. If you feed that kind of data into an AI, it will absorb the unconscious racism of the doctors.

    And that's in a best case scenario that's technically impossible. To get AI to even produce readable text, we have to feed a ton of data that cannot be screened by the people pumping it in. (AI "art" has a similar problem: When people say they trained AI on only their images, you can bet they just slapped a layer of extra data on top of something that other people already created.) So yeah, we do get extra biases regardless.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 3 points 9 months ago

    There is a lot of bias in healthcare as well against the poor, anyone with lousy insurance is treated way way worse. Woman in general are as well. Often disbelieved, and conditions chalked up to hysteria, which often misses real conditions. People don't realize just how hard diagnosis is, and just how bad doctors are at it, and our insurance run model is not great at driving good outcomes.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 9 months ago (4 children)

    I don't think its fair to say that "ai has shown to make doctors worse at their jobs" without further details. In the source you provided it says that after a few months of using the AI to detect polyps, the doctors performed worse when they couldn't use the AI than they did originally.

    It's not something we should handwave away and say its not a potential problem, but it is a different problem. I bet people that use calculators perform worse when you remove calculators, does that mean we should never use calculators? Or any tools for that matter?

    If I have a better chance of getting an accurate cancer screening because a doctor is using a machine learning tool I'm going to take that option. Note that these screening tools are completely different from the technology most people refer to when they say AI

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] [S] 3 points 9 months ago (2 children)

    Calculators are programmed to respond deterministically to math questions. You don't have to feed them a library of math questions and answers for them to function. You don't have to worry about wrong answers poisoning that data.

    On the contrary, LLMs are simply word predictors, and as such, you can poison them with bad data, such as accidental or intentional bias or errors. In other words, that study points to the first step in a vicious negative cycle that we don't want to occur.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 3 points 9 months ago* (1 child)

    As I said in my comment, the technology they use for these cancer screening tools isnt an LLM, its a completely different technology. Specifically trained on scans to find cancer.

    I don't think it would have the same feedback loop of bad training data because you can easily verify the results. AI tool sees cancer in a scan? Verify with the next test. Pretty easy binary test that won't be affected by poor doctor performance in reading the same scans.

    I'm not a medical professional so I could be off on that chain of events but This technology isn't an LLM. It suffers from the marketing hype right now where everyone is calling everything AI but its a different technology and has different pros and cons, and different potential failures.

    I do agree that the whole AI doesnt have bias is BS. It has the same bias that its training data has.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] [S] 2 points 9 months ago

    You're definitely right that image processing AI does not work in a linear manner like how text processing does, but the training and inferences are similarly fuzzy and prone to false positives and negatives. (An early AI model incorrectly identified dogs as wolves because they saw a white background and assumed that that was where wolves would be.) And unless the model starts and stays perfect, you need well-trained doctors to fix it, which apparently the model discourages.

  • permalink
  • fedilink
  • source
  • parent
    • load more comments (1 reply)
  • [–] -1 points 9 months ago (5 children)
    1. can cut through bias is != unbiased. All it has to go on is training material, if you don't put reddit in, you don't get reddit's bias.
    2. see #1
    3. The study is endoscopy only. results don't say anything about other types or assistance like xrays where they're markedly better. 4% on 19 doctors is error bar material. Let's see more studies. Also, if they were really worse, fuck them for relying on AI, it should be there to have their back, not do their job. None of the uses for AI should be doing anything but assisting someone already doing the work.
    4. that's one hell of a jump to conclusions from something that's looking at endoscope pictures a doctor is taking while removing polyps to somehow doing the doctors job.
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] [S] 1 point 9 months ago* (last edited 9 months ago) (2 children)

    1/2: You still haven't accounted for bias.

    First and foremost: if you think you've solved the bias problem, please demonstrate it. This is your golden opportunity to shine where multi-billion dollar tech companies have failed.

    And no, "don't use Reddit" isn't sufficient.

    3. You seem to be very selectively knowledgeable about AI, for example:

    If [doctors] were really worse, fuck them for relying on AI

    We know AI tricks people into thinking they're more efficient when they're less efficient. It erodes critical thinking skills.

    And that's without touching on AI psychosis.

    You can't dismiss the results you don't like, just because you don't like them.

    4. We both know the medical field is for profit. It's a wild leap to assume AI will magically not be, even if it fulfills all the other things you assumed up until this point, and ignore every issue I've raised.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] -3 points 9 months ago (1 child)

    1/2: You still haven’t accounted for bias.

    Apparently, reading comprehension isn't your strong point. I'll just block you now, no need to thank me.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •