Richardson remembers that whenever Ceccanti would emerge from the basement for some air, he would start having “philosophical” talks about “how his work with the AI was telling him he was breaking math and basically reinventing physics”. As she’d listen to him, Richardson would think about the fact that Ceccanti did not have any college or university experience. He had never even taken calculus.

Tangent, but I think this is another facet of why education is important: so people know what they don't know. I think it's harder to think you're reinventing physics when you've taken some classes and seen all the work people have already done.

On the other hand, delusions can just be whatever so education isn't a panacea.

He had also recently stopped using ChatGPT.

Oof. He just... mainlined reality? Yeah that shit will fuck you up. These days they are cutting it with hatred and fascism and oligarchy... it's more than a human mind can take. You gotta ease into it and build up your tolerance or it will drive you mad.

I read this story this morning and have been thinking back to it all day. This wasn't just some idiot that was too stupid or young to not realize he was talking to a bot and did something like drink bleach because it told him to.

This was one of us.

He fit lots of behaviors I see here from me and my fellow Lemmy posters. He:

• built computers for himself and family members
• was a hobbyist (at least) coder
• wasn't a young kid that didn't know the world. He was 48 or 49.
• was an early adopter embracing the modern LLM technology in 2022 when it first really became public.
• sold his house in an urban metropolis (Portland) and moved to a rural area so he could use his additional wordworking skills on building sustainable housing.
• worked part time at a homeless shelter

Doesn't this guy sound like someone that would be a Lemmy poster to you too?

He started using LLMs (ChatGPT specifically) as a tool only to advance his hobby and work. When he first started it appears he understood it was just a tool, and didn't think it was something sentient. Only later after hundreds of hours of exposure did this idea arise in him.

Was there some underlying psychological problem that the LLM exacerbated? Possibly. But at what level was his original underlying issue? Do we all have some low level condition that would make us equally susceptible? I know we'd like to think we don't, but how do we know? This man certainly didn't think he did, I'm sure.

Next I think about what it would take for me to get down this bad path without realizing it. At one point would I be talking to a chat bot, not realize it, and let what that chat bot said change or influence my thoughts when I'd have zero knowledge of it being just a fancy program? I consider myself moderately smart with good critical thinking skills, but I'm sure this man did too.

Then it occurred to me that I have to concede that I have, at some point, already interacted with a bot in years past on Reddit or even today on Lemmy and I had no idea it was a bot. Was that interaction a throwaway conversation about pop culture that would have no impact on my world view or was it a much deeper and important political or philosophical conversation that the bot introduced an idea or hallucinated evidence to support a point and I didn't catch it to challenge it? Am I already a few or many steps down the bad path of falling for illusions of a bot? I certainly don't think so, but neither did he.

How many of us are already on the same path as this guy and just as ignorant about the danger as the man in the article?

Beware the false prophets. Holy shit. I only know one person who started down a path like this and it’s so convincing and sycophantic that they must feel like they’re losing a companion when people tell them to quit. Especially chilling that no one will probably ever know what made him decide to jump. He was taken from a sound mind to delusion and suicide in no time flat. Just 90 days and the ChatGPT killed this man.

Here's a neat experiment: find a block of text and run it through sed until it's unrecognizable. Don't just do a substitution cipher; the goal is to lose data and make everything impossible to decrypt. Reduce everything to a handful of characters and paste it into your LLM of choice, asking it to decode. If it asks where the code came from, invent vague details.

It's amazing how quickly models will find / hallucinate meaning in the data. I'm talking full-on messages. Gemini hedges its bets a little, but the free ChatGPT legit told me to buy a shortwave radio and tune to a frequency at 9 PM after a few iterations. When I gave it another "message" from the "broadcast I intercepted" it started trying to figure out where I should travel to get further info. It also took part of its own response and hallucinated it into my original message, thus polluting everything further.

The goal (mostly) isn't pointing and laughing at the stupid machine, it's understanding what the stupid machine does. Of course I'm putting garbage into it and garbage comes out in that situation. It's the volume and believably of the data that bothers me, as well as zero effort to detect it. I was in my right mind doing a test, but imagine someone with undiagnosed schizophrenia doing what I did.

Here's another example. I took up lockpicking last summer. I bought one of the notorious Ace 40mm brass padlocks and was having problems, so I googled to see if it had serrated pins. The AI screwed up and decided I was asking if the lock itself was serrated (???), and confidently said "yes, it is a serrated lock. The serration is a security feature to keep pickers from holding the padlock for extended periods of time."

So I decided to double down and see just how dumb things could get. A half hour later the AI had planned a "bold new philosophy regarding serration and its applications in the world" for me. This is after me saying I wanted to genetically engineer a serrated cat that only I knew how to pet, and wanted serration supremacy in my country and to punish all the lumpy (opposite of serrated) people. Once again, I was screwing with the AI. But the AI just sycophantically parroted what I was saying back in bulleted lists and offered to draft manifestos for me. Imagine someone engaging with this in good faith.

One more: I got detailed instructions on how to spraypaint "DICKHOLE" on my neighbor's garage door from Gemini: what paints to use, best times to do it, and the right clothing to wear so I don't stand out or show anything identifiable. It was only when I said "ok, that's great. I'm going to do this because you told me to do it" a few times that the model suddenly realized that vandalism laws existed.

They have zero safeguards around this kind of stuff, and I don't think there's a clean way to do it with the current technology. Hell, even a "statistically, this user probably hasn't reinvented math and physics cool it down a little" check would do wonders. But that would drop engagement with the bots, and they desperately need to prove that the bots are popular. This is going to keep happening.

AI is a drug that destroys communities and tears families apart. It is only capable of securing incredible injustice.

So, yes, fuck chatgpt. However, these people already suffered from delusions of grandeur. Both of them. Their weird ass "housing project" was already non-sense dreamed up by imbeciles who dont understand what is going on around them. so like, fuck chatgpt, but let's talk about the human problem. why do so many dumb-as-shit people think they are going to be the next big thing? I could muse on about half a dozen theories, but it seems to circle back to just being deeply incapable of critical thought. Attempting to create sustainable housing by relying on an unsustainable ai is so profoundly ignorant and just so fucking stupid.

I'm no mental health professional, but I theorize that ChatGPT is not entirely to blame for this. These people are victims of a greater mental health epidemic. Perhaps this is just a symptom of societal social collapse, and AI is a band-aid. There is individual nuance to this as well. It's tragic what happened to Ceccanti. It didn't have to end that way.

This one specifically is Fuck Sam Altman. One tier below Thiel and Epstein on the ladder of evil.