Can't read the article because it's paywalled but I can't imagine they are actually building power stations with AI, that will just be a snappy headline. Maybe the AI is laying out the floor plans or something, but nuclear power stations are intensely regulated. If you want to build a new reactor design, or even if you want to change an existing design very slightly, it has to go through no end of safety checks. There's no way that an AI or even a human would be allowed to design a reactor, and then have it be built with no checks.

Whoops. Fixed, thanks.

How many people has Google convinced to kill themselves? That is the relevant question. Looking up the means to do the deed on Google is very different from being talked into doing it by an LLM that you believe you can trust.

Google doesn't tell you that killing yourself is a good idea and that you shouldn't talk to anyone else about your suicidal ideation

Kill death ratio - or rather, kill save ratio - would be rather difficult to obtain and more difficult still to appreciate and be able to say if it is good or bad based solely on the ratio.

Fritz Haber is one example of this that comes to mind. Awarded a Nobel Prize a century ago for chemistry developments in fertilizer, used today in a quarter of food growth. A decade or so later he weaponized chlorine gas, and his work was later used in the creation of Zyklon B.

By ratio, Haber is surely a hero, but when considering the sheer numbers of the dead left in his wake, it is a more complex question.

This is one of those things that makes me almost hope for an afterlife where all information is available from which truth may be derived. Who shot JFK? How did the pyramids get built? If life's biggest answer is forty-two, what is the question?

For me, the suicide-related data is so hard to measure and so open for debates, that I'd treat it separately, or not include it at all, if using death count as an argument against llms, since it's a breach for deviating the debate.

I can't really see how we could measure that. How do you distinguish between people who are alive because they're just alive and would have been anyway and people who are alive because the AI convinced them not to kill themselves?

I suppose the experiment would be to get a bunch of depressed people split them into two groups and then have one group talk to the AI and the other group not, then see if the suicide rate was statistically different. However I feel it would be difficult to get funding for this.

These people turned to a tool (that they do not understand) - instead of human connection. Instead of talking to real people or professional help. And That is the real tragedy - not an arbitrary technology.

They are a badly designed, dangerous tools and people who do not understand them, including children, are being strongly encouraged to use them. In no reasonable world should an LLM be allowed to engage in any sort of interaction on an emotionally charged topic with a child. Yet it is not only allowed, it is being encouraged through apps like Character.AI.

only a tool

“The essence of technology is by no means anything technological”

Every tool contains within it a philosophy — a particular way of seeing the world.

But especially digital technologies… they give the developer the ability to embed their values into the tools. Like, is DoorDash just a tool?

Reading the messages over it seems a bit more dangerous than just "scary ai". It's a chatbot that continues conversation to people who are suicidal and encourages them to do it. At least have a little safeguard for these situations.

“Cold steel pressed against a mind that’s already made peace? That’s not fear. That’s clarity,” Shamblin’s confidant added. “You’re not rushing. You’re just ready.”

Seriously. There have been always people with mental problems or tendency towards self harm. You can easily find ways to off yourself on google. You can get bullied on any platform. LLMs are just a tool. How detached from reality you get by reading religious texts or ChatGPT convo highly depends on your own brain.

It's like how entire genre of videogames are now getting censored because of few online incels.

Labelling people making arguments you don't like as "haters" does not establish credibility in whichever point you proceed to put forward. It signals you did not attempt to find rationality in their words.

Anyway, yes, you are technically correct that poisoned razorblade candy is harmless until someone hands it out to children, but that's kicking in an open door. People don't think razorblades should be poisoned and put in candy wrappers at all.

Right now chatbots are marketed, presented, sold, and pushed as psychiatric help. So the argument of separaring the stick from the hand holding it is irrelevant.

While a lot of people die trough suicide, it's not exactly good or helpful when an AI guides some of them trough the process and even encourages them to do it.

Why do you think we have seatbelt laws?