Dangerous information like library books. Like the text of any Tom Clancy novel. These models, online or offline, just pick words based on what you write. You choose what comes out. If you ask for vile pornography, you don’t get to blame the company for offending you, and if some prick asks for tactical advice ‘in Minecraft,’ the company did not proactively radicalize him. The chatbot can only repackage what’s readily available to any pirate.
The majority of that text is being used in violation of copyright, but that's beside the point.
My point is that if I were to have see kid online say they were gonna shoot up a school, and then I spent the next week coaching them how to do so, I would be responsible for my involvement with the eventual shooting. Operating a service which does the same thing should entail the same responsibilities. OpenAI's failure to control their service does not absolve them from responsibility, if anything it shows how knowingly irresponsible they are.
They can easily implement stronger limits on what their models are allowed to say to a user BUT that would makes their service feel more like a computer and less like a person. A simple regex and blacklist of words and phrases could restrict topics. A second agent that only looks at the model output could be used as a nanny. Are you saying that it is reasonable that a teenager was able to bypass all the safeties of a billion dollar business?
And if a local model lets that child slap together a program that surpasses what I could do, fresh from a college education in computer science - good. We’ve been trying to make that happen for sixty years.
That says more about you than the LLM. LLMs can be used as an elaborate autocomplete but fully LLM written projects are known as slop for a reason, each small part may be complete and correct but as a whole it is just an unmaintainable mess of security issues and technical debt.