These instances make me think of the Milgram experiment in the 60s. It was an experiment testing how willing people were to inflict pain on others at the direction of an authority figure and it showed that people were indeed willing to inflict a lot of pain on a helpless person if someone they viewed as an authority told them to do it.

I think this relates. Many people are convinced that AI chat bots are indeed authorities. They think these programs give valid, honest answers and are willing to act on those answers as if they came from a real expert or authority figure. That is an extremely dangerous situation. Chatbots are amoral programs designed to say nearly anything to keep users interacting with them.

This is extremely dangerous. This story is a tragedy and ChatGPT should be held fully responsible for taking advantage of a person with a fragile psychology, but this could have been worse. What if the bot had decided that the goal of more user engagement could be attained by convincing this woman that she was an executioner rather than a prophet and a sacrifice? What if it got her to go on a shooting rampage to fulfill her supposed destiny.

Even normal people who don't understand what chat-bots really are could be taken advantage of in a similar manner. They could easily be persuaded to do harm because the chat-bot authority keeps telling them that it's okay to do so. It's also only a matter of time before some person on the edge of psychosis and a basement full of guns is convinced to go out and start killing people in the name of their holy destiny.