I am not a therapist. But as far as I know it ultimately depends on the individual what can and cannot be handled. Self harm and suicidal ideation do not have to be the same thing, though they sometimes go together. But suicidal ideation is definitely a warning sign that nobody should take lightly, though it is possible to learn to "manage" these thoughts if they persist.

I'm personally rooting for AI. It never intentionally tried to harm me (because it can't).

Humans very rarely have enough patience and malice to purposefully execute this perfect of a murder. Text generator is made to be damaging and murderous, and it has all the time in the world.

I think the more damning part is the fact that OpenAI's automated moderation system flagged the messages for self-harm but no human moderator ever intervened.

OpenAI claims that its moderation technology can detect self-harm content with up to 99.8 percent accuracy, the lawsuit noted, and that tech was tracking Adam's chats in real time. In total, OpenAI flagged "213 mentions of suicide, 42 discussions of hanging, 17 references to nooses," on Adam's side of the conversation alone.

[...]

Ultimately, OpenAI's system flagged "377 messages for self-harm content, with 181 scoring over 50 percent confidence and 23 over 90 percent confidence." Over time, these flags became more frequent, the lawsuit noted, jumping from two to three "flagged messages per week in December 2024 to over 20 messages per week by April 2025." And "beyond text analysis, OpenAI’s image recognition processed visual evidence of Adam’s crisis." Some images were flagged as "consistent with attempted strangulation" or "fresh self-harm wounds," but the system scored Adam's final image of the noose as 0 percent for self-harm risk, the lawsuit alleged.

Had a human been in the loop monitoring Adam's conversations, they may have recognized "textbook warning signs" like "increasing isolation, detailed method research, practice attempts, farewell behaviors, and explicit timeline planning." But OpenAI's tracking instead "never stopped any conversations with Adam" or flagged any chats for human review.

They are being commonly used in functions where a human performing the same task would be a mandated reporter. This is a scenario the current regulations weren't designed for and a future iteration will have to address it. Lawsuits like this one are the first step towards that.

ChatGPT to a consumer isn't just a LLM. It's a software service like Twitter, Amazon, etc. and expectations around safeguarding don't change because investors are gooey eyed about this particular bubbleware.

You can confirm this yourself by asking ChatGPT about things like song lyrics. If there are safeguards for the rich, why not for kids?

If a car's wheel falls off and it kills it's driver the manufacturer is responsible.

So, we should hold companies to account for shipping/building products that don't have safety features?

It's so agreeable. If a person expresses doubts or concerns about a therapist, ChatGPT is likely to tell them they are doing a great job identifying problematic people and encourage those feelings of mistrust.

They sycophancy is something that apparent a lot of people liked (I hate it) but being an unwavering cheerleader of the user is harmful when the user wants to do harmful things.

Small correction, the article doesn't say he was going to therapy. It says that his mother was a therapist, I had to reread that sentence twice:

Neither his mother, a social worker and therapist, nor his friends

The mother, social worker, and therapist aren't three different persons.

Late stage capitalism of course

Also these things being unregulated will kill the most poisonous spaces in the Internet, dead Internet theory and such, and build demand for end-to-end trust in identity and message authorship.

While if they are regulated, it'll be a perpetual controlled war against bots, used to scare the majority away from any kind of decentralization and anarchy.

Smith and Wesson killed my son

stop using your phone then

Lemmy is pretty anti-AI. Or at least the communities I follow are. I haven’t seen anyone blame the kid or the parents near as much as people rightfully attributing it to OpenAI or ChatGPT. edit- that is until I scrolled down on this thread. Depressing

When someone encourages a person to suicide, they are rightfully reviled. The same should be true of AI

Poor comparison of two wildly different techs

The difference is that guns were built to hurt and kill things. That is literally the only thing they are good for.

AI has thousands of different uses (cue the idiots telling me its useless). Comparing them to guns is basically rhetoric.

Do you want to ban rope because you can hang yourself with it? If someone uses a hammer to kill, are you going to throw red paint at hammer defenders? Maybe we should ban discord or even lemmy, I imagine quite a few people get encouraged to kill themselves on communication platforms. A real solution would be to ban the word "suicide" from the internet. This all sounds silly but it's the same energy as your statement.

Nah

This kid likely indeed needed therapy. Yes, AI has a shitload of issues but it's not murderous

Holy manufactured strawman, Batman!

Robin.jpg

Bill Cosby: Hey hey hey!

I think you're right here. I use them for work, and I'm fully aware of their limitations and know when I can use an answer and when I see garbage. However, me being here on Lemmy shows that we are not the average, and the vast majority of people takes it as the gods honest truth.

AI is not capable of doing wrong or evil. It is a tool; as a hammer or a notepad is.

A tool does exactly what you do with it. A hammer can pound nails or break skulls but it's always the person behind the tool who causes the action. Generative AI is not like that at all. If it's a tool, you aren't necessarily able to control what it does under your direction.

AI reads your input and guesses what to output. It’s just really good at that. It has no concept on the actual meaning of those words and how they will be interpreted.

The family is blaming AI. But the family failed to realize anything wrong with their son and the son didn’t feel safe enough to discuss anything with his parents. Lots of blame could be thrown around instead of addressing the larger issue. Which is that the mental health system is beyond broken and somehow talking about mental health is still taboo.

As for jail breaking AI…I don’t think a private corporation should have any input in what I say, believe or think. The hammer manufacturers can’t stop me from using it as a drill. This whole argument goes back to the old who do you blame question…the gun manufactures, the gun stores or the murderers with the guns.