To argue that ChatGPT et al. can't cause direct harm (i.e. irradiate people) misses the larger point I think. To me, that's slicing up the ethical responsibility of a system until no component bears ownership. Extending that logic to Therac-25 might be saying a function or variable can't irradiate people. Sure, not by itself. But it existed within a system that reliably produced harm.

From the Therac-25 wiki: "These accidents highlighted the dangers of software control of safety-critical systems."

I'd argue that mental health conversations are safety critical systems. Another safety critical system might be a basic search engine, if a user turns to it with real intent to do harm.

There's a reason we put safety warnings on products, regulate access to chemicals, etc, etc. I'm not arguing that we should do the same for LLMs, just showing that the Therac-25 example is rookie numbers compared to the reach and risks of modern software.