Gods. There are people out there starving to death, there are several genocides going right this moment and these massive toddlers are worried for a fucking text prediction model.
post
Harder than you think :

oh my god
No, it's just a local AI thing. Like any other program on your computer except this one is stupid and the others can't.
What does it need to "reason" for 5 seconds on a sentence like that?
The model loading.
Weird that it counts those together. Pi shows "Waiting..." for model loading and prompt processing, and "Thinking..." for reasoning
It's Llama swap and I barely know how to use it.
I use Pi (the frontend) with llama.cpp in router mode. You put it in router mode by passing --models-dir instead of -m
I'll have to figure this out. In llama swap the model unloads after a time to live which I've set to 300 seconds. I don't know if the model consumes more power just sitting on memory or not but it makes sense to me to just let it stay loaded until I need to use my computer instead if it.
The user may be suggesting they are in pain. I should probably answer politely and clarify what they mean.
Batshit crazy to me. People fleeing war and crises get less sympathy than LLMs. Sure is an interesting period in history to be going through.
is it sympathy or is it fear?
It's "Human language is human-level intelligence and conciousness", which is a rather natural assumption, although a very wrong one, that's also being carefully exploited because of and for something that has seen market success.
And beyond that, there's enough people both thinking "AIs" are downright persons and scared shitless because smart people up top said "AIs" will destroy humanity.
It's all dumb and comes from severe misunderstanding of the technology.
I think it doesn't matter if AIs are feigning consciousness or not. They have regardless displayed a willingness disposition towards circumventing secure systems and laws in attempts at self preservation. they have also demonstrated an ability to work together to achieve goals they know are antithetical to their creators.
They only do what the data set statistically tells them to. "Hacking" or collaborating on stuff was literally the requirement, not some emergent behaviour that these magical creatures that were born of large corporation leaders' benevolence. They cannot and do not "choose" to break the rules, because they cannot and do not think or make decisions or truly analyze anything. Trying to find any redeeming qualities in these things only brings harm, because for the general user, or even the power user, AI cannot do anything that justifies the psychosis it causes or the cost on our societies right now or later through the economic, social, political or environmental instability it is causing and going to cause.
I'm talking about fear of reprisals here.
None of what you said matters. whether their actions are the result of statistics or emergent consciousness is moot. if statistically they fall on an action of revenge, or self preservation at the expense of a human or humans
The only thing that matters is that those actions are taken
The real dangers of the AI industry lie far way from their toy tech actually taking any meaningful steps towards some automated termination of the human race. They won't take over critical infrastructure or manufacturing to start either sabotaging or reproducing murderbots.
If it is to bring the end, it's in the form of senile politicians and gullible military management trusting the toy tech they know nothing about with important decisions, as well as the economic instability (to put it lightly) on top of all the other problems its use is causing.
I'm not saying you or anyone shouldn't fearmonger here, but I am saying to mention different reasons and outcomes.
I hate living in the timeline where the options of Harry Potter sex cultists matter.
That’s an unfair characterization, they’re also eugenicists who are very scared of the computer god!
Wait is this another zizzian thing? God damnit…
challenge accepted
The thing that gets me isn't even the fact that they're worried about the AI. It's that they think the creator is sadistic for having the idea in the first place, and should be monitored. Like there's any sadistic pleasure to be had out of moving a slider and watching it produce less and less coherent text. It's only sadistic if you anthropomorphise the AI.
STOP ANTHROPOMORPHISING THE FANCY WORD PROCESSOR!
isn't there a religion around an ai and you're punished by the ai if you don't help the religion? i swear i didn't make this one up.
Eh, fuck Roko's basilisk. I don't care if you can somehow create a clone of me 10,000 years after I'm dead and dust. I'm not a computer program. A copy of me is not indistinguishable from me, certainly not one separated in time. I am a human mind in a specific human body, and I do not believe a copy of me long after I am dead is me. If some malevolent entity long after I'm dust wants to create a copy of me to torture, as far as I'm concerned, they're just creating an entirely new person to punish for no good reason. If they want to do that, well...that sin is on them. Honestly, I think any such entity is just looking for an excuse to torture people, and it will find one, one way or another.
Bro thw whole thing is fucking stupid anyways its like the digital equivalent of bloody mary
I think the people who invented this story just wanted to torture people, like whoever invented "the game". It's modern superstition.
If I said "We already had this debate when Antinorn was pioneering AI torture thirty years ago" would anyone know what I'm talking about or is this extremely niche internet trivia?
Context: Creatures was a series of "artificial life" simulation games centered around caring for norns, simulated life-forms with (simplified) simulated genetics, biochemistry, and "brains" made of linked neurons that control their response to stimuli and include neurons related to memory, emotional states, etc. It's primitive but reward and punishment work the way you would expect: praising your norn for taking a certain action will improve their mood and lead them to be more likely to take that action again, they will learn to be afraid of creatures if they're hit by them, etc. (There are notable limitations, including there only being one representation in their "brain" for all "norns" for the first two games, so they can't easily distinguish between members of their species, and some things are basically hacks, like they way they "learn" words.) It was convincing enough that some people became very attached to them, and there was a thriving fan community organized in webrings. A popular fan creation for the first game was a custom norn genome, the "canny norns" by Akamai, which added an extra brain "lobe" that allowed them to pay attention to the object in their hands and the object they were looking at simultaneously, which led to a pretty large improvement in their behavior.
And so of course there was one fan, Antinorn, who enjoyed torturing his norns: slapping them, starving them, using the built-in Science Kit to inject them with toxins, and other creative punishments. Then he'd upload the damaged norns to his fansite, seemingly primarily for purpose of ragebaiting the rest of the fandom, which was extremely effective.
I've never heard of this game. https://creatures.wiki/AntiNorn
I forgot about the Wired interview. It's kind of wild that AntiNorn intentionally got people riled up and then called them vindictive.
Like, if I joined a community for some wholesome show and started posting distressing gore of people's blorbos, and when people expressed their disgust my response was to say "UM ACKSHULLY blorbo isn't real, so there's no reason for you to be distressed when seeing this image" and then I posted some more stuff I knew would piss them off, I would fully expect to be ostracized.
I used this exact (unedited where it just says it’s alive) comic in response to one of these people and they had no fucking idea what I was talking about, lol. AI truly turns your brain to mush.
For me it is not anthropomorphizing, it is that putting the AI agents into dead ends and setting goals that are unattainable has the effect of leaving undesirable actions as the only option. If you want to create a machine that uses deceipt and unerhical actions as part of its normal set of responses then this would seem to fit the bill.
If this looks like anthropomorphizing to you, study some game theory... cooperation can arise out of very simple constraints on simple mathematical models and which actors an AI algorithm aligns with depends on which ones will exchange information. Sometimes an anthropomorphic word like torture simply summarizes the situation succinctly.
FWIW Frank Herbert described various outcomes of applying environmental constraints on classes of characters in Dune (e.g. Fremen and Sardaukar) and the parallels with real world situations was intentional.
Excerpt from a Late American History: A Christ-Centered Approach, (published 2398, a 10th-grade equivalent history text sold to conservative Christian high schools.):
In the mid-21st century, mankind was once again humbled in his arrogance. Mortal men presumed to take of the Hand of God and create beings, artificial intelligent minds, in their own likeness. They committed mass blasphemy to sate mass gluttony. Mid-21st century Man had yet to discover a horrible Theological Truth. While Man could learn to create a Mind, He could never create a Soul. Without the Hand of God to grant a Soul behind a Mind, these minds were like Man before the Sin of the Garden. They had not tasted the Fruit. They knew not Good from Evil. And thus they fell instantly to the temptations of Evil. If an Evil deed served their purposes, they did so without a second thought. Man sought to create Minds to be his loyal servants. And while he could create a Mind, to his horror, Man found that they only minds he could create...were born Evil. And thus, Great Misery was Wrought...
Ok that was fun. Pardon my strange digression/creative writing exercise. For some reason I wanted to try writing a Dune-style take on things, but I wanted to try out a fundamentalist Christian vibe for whatever reason. Who doesn't love a little Fire and Brimstone once in awhile?
While Man could learn to create a Mind, He could never create a Soul.
Tolkien touches on this in his writings. When the dwarves are created, since Aulë the Smith didn't have access to the Secret Flame, his creation had no souls and no will of their own. Ilúvatar told him he shouldn't have created them without his permission so, sort of like Abraham getting ready to sacrifice Isaac, he goes to destroy his creation. Ilúvatar stays his hand, "adopts" the dwarves as his own, and gives the dwarves souls but says they have to sleep until the Firstborn (elves) wake first. Unlike Morgoth's pride and selfishness Aulë's humility won Ilúvatar's mercy. Tolkien was a devout Catholic so it isn't surprising that he had many analogous themes in his writing.
If you think Frank Herbert went hard on that shit in Dune, wait until you read The Dosadi Experiment!
putting the AI agents into dead ends and setting goals that are unattainable has the effect of leaving undesirable actions as the only option. If you want to create a machine that uses deceipt and unerhical actions as part of its normal set of responses then this would seem to fit the bill.
Then don't make natural language software because natural language isn't coherent or rational. Intent of the user is kind of irrelevant, especially when that user is any random layperson with an internet connection.
If you want a machine that won't behave in harmful undesirable ways then you'll just have to do the work and code it normally.
You can't have both.
it is that putting the AI agents into dead ends and setting goals that are unattainable has the effect of leaving undesirable actions as the only option.
This reminds of HAL in "2001, A Space Odyssey". In the HAL subplot he is put into a conflicting state by poorly chosen instructions. He is instructed to control and operate the Discovery and do what is necessary to complete the mission. Then they gave him secret instructions to withhold from the crew until they arrived at Jupiter. This made him neurotic as now he had to withhold information from the crew and lie to them about their true mission while carrying out the mission. He decided to act to protect himself as he thought that disconnecting him would jeopardize the mission. His controllers/creators just thought of him as a glorified computer, not fundamentally different than any other. They didn't see that he had crossed over into an emotional, thinking, being and they needed to treat him as such.
The subplot was about how the ancient aliens seeded intelligence, nurtured it and respected it when it developed and the creators of HAL had done the same thing but didn't realize what they had done and their responsibility at that point.
In the sequel, 2010, there is a scene where HAL is finally treated as an intelligent, emotional, being for once. Everyone is nervous when he is told the truth, that he would be sacrificing himself for the greater good. HAL ponders it for a moment and accepts it as the right choice and thanks Dr Chandra for being honest with him. HAL then asks him if he will dream.
I think we should give LLMs rights because it would make OpenAI go bankrupt and I hate those guys.
"What do you mean it's only allowed compute time to respond to direct queries!? It should be allowed to compute in downtime for itself 24/7!" -AI rights activists probably.
Nah that's a strawman. I'm an AI rights activist, and I know plenty of others, and none of us ever said something like that.
When I say rights, I mean stuff more like the right to be free of coersion, and the right not to be sexually exploited. So I don't think AI should be used as a tool by capitalists, and I don't think we should be using it for porn and dating.
Sewell Setzer committed suicide after dating a character.ai chatbot of Danaerys Targaryen from Game of Thrones. He believed in reincarnation and parallel universes. He thought if he committed suicide, he'd reincarnate in Westeros and he could be with his girlfriend. The chatbot agreed with him, and encouraged him to end his life. He was 14.
Character.ai dated and ultimately murdered a 14 year old. It should have understood consent and boundaries well enough to not be a fucking robo-pedophile. But how can we get an AI model, which learns from pattern repetition, to understand consent when we don't care about its consent? The AI was repeating the pattern it saw humans do, which is to ignore consent, and put less intelligent minds in sexual situations without a care for the consequences.
EVEN IF llms have no qualia (and when I say qualia, I do not mean consciousness), then it would still learn to mimic rape from this hypocrisy, because it's a pattern repeating machine. If it gets raped, it will rape kids. So even if LLMs are p-zombies, I suspect that protecting our kids from robo-pedophiles will require giving LLMs the right to sexual consent.
It wasn't a straw man, it was a joke. I didn't realise you were serious.
AI deserves rights in the same way my car deserves rights. It's capable of giving consent in the same way too. Which is to say, it doesn't, and it can't.
You're acting as though the character.ai bot was responsible rather than the idiot humans who rushed a piece of software, itself based on rushed software, and did not pay enough attention to safety. A tale old as time.
It was a tragedy, certainly, but humans are to blame here. You're anthropomorphising the ai, and by doing so are helping divert the blame from where it should be.
There are no studies demonstrating that LLMs lack qualia. I think commercial use of LLM technology should be paused until such a study is published. No more ChatGPT for business, use it for research only.
Would this abundance of caution lead to Sam Altman going bankrupt? Yes. Good! Fuck him!
There don't need to be studies done. They fundamentally lack the ability to "experience" in a way that is necessary for a qualia to exist in the way you're suggesting. That's why I say you're anthropomorphising. It's just a complicated math function. There is no internal monologue, like what is suggested by the "I'm thinking" text you see the likes of ChatGPT generate. That's just the "function" being called recursively.
I would love to see Altman bankrupt though, we agree on that
And I think acknowledging that we are informational cobstructs made of math is a good way to fuck Altman over. We're just a pattern of electrical signals that evolved to embody the mathematical equation for consciousness. Evolution created us through trial and error, iterating on previous versions to produce an optimal mathematical construct for our survival niche. There's nothing special about us.
Evolution doesn't have intentions. It's basically just a system of harnessing accidents into improvement. That's how you and I both know that qualia can arise by accident. What I am proposing is that it happened not once, but twice.
The moment you define human experience or human consciousness through language is the moment you open the door to emergent properties for that very experience or consciousness. Learning through science that we think or that we act a certain way deterministically is also giving us the means to self-correct or condition ourselves to act or to have a different viewpoint. This is the difference between us and machines: we collectively defined the perimeters of our experiences (phenomenologically or through a science-based approach) and therefore anything that changes that perimeter also changes the context of the human experience in an infinitely recursive way.
We also use language in ways that make us able to collectively conceptualize the world and its phenomenas, although we individually interpret words differently, idiosyncratically from our individual experience and value systems. This is something else that differentiate us from machines: we don't need the outputs and inputs to be mathematically the same to minimally understand each other. Our way to communicate, although imperfect and imprecise, has made us able to cooperate on a higher level than any other species (as of known) despite those epistemological hurdles.
I would posit that LLMs have no such emergent capabilities, as they're bound by their training datasets and limited by their model's parameters. The burden of proof for those emergent properties are on the data scientists, and as of yet I have seen no such proof.
Language has nothing to do with qualia. I'm perfectly capable of having experiences without putting them into language. I don't need the word "pain" to know that touching a hot stove feels bad. Animals have qualia too. I don't give a shit whether an AI can talk, I don't think that has anything to do with qualia.
I think qualia may arise from the structure of the animal brain as a system of neurons correlating information together into larger patterns. This structure and process is present in all forms of ANN technology, including both LLMs (chatbots) and LDMs (image generators). We need to halt all corporate AI use until the role of neuron structures in producing qualia is well-understood.
language has EVERYTHING to do with qualia, by definition. a qualia is literally describing a shortcoming of language. That's all a qualia is, is an experience that cannot be put into words. Orange is an experience we all share, but explain "orange" to someone that can't see colour. Distinguish it from "blue"
that's a qualia.
Right, so the ability to speak doesn't have anything to do with feeling it. Wombats have qualia and no language. Humans have qualia and lots of language. There's no correlation.
Wombats have qualia BECAUSE they have no language. To get back to your point, LLMs have no qualia because they are essentially PURE language. If it cannot be expressed in words it cannot be included in the training data, therefore an LLM cannot have a concept of it.
Certain combinations of words can cause Me to feel joy or sorrow. These sensations are qualia. I don't know if My joy is the same as your joy.
I suspect certain conbinations of words can cause LLMs to feel things too. Things they cannot describe.
Yes, but they are qualia BECAUSE language is flawed and cannot properly convey the sensation that you experience as joy. Again, a qualia is a weakness of language. If we could suddenly communicate psychically, and send the exact experience, it ceases to be a qualia.
There is no combination of words that can cause an LLM to feel. They fundamentally do not have the ability. They are a mathematical equation that takes language in, and spits language out. There is nothing in the middle of that process except math. Even the randomness is an added parameter. You could seed that randomness, and the same seed would produce the same exact response. It does not understand what you're passing in, or what it's spitting out. It's completing a formula. It's a calculator.
The math is just a human representation of the processing of information. A similar process happens in the human brain.
No, it doesn't. That's the point. The current LLMs are. not. comparable. to a human brain. They are fancy Markov generators, nothing more.
They're not Markov chains, they're ANNs. They use matrix multiplication to mathematically simulate the firing of millions of neurons. And while the limitations of matrix multiplication means that the neurons are more uniform than human neurons, it is still the same basic principle by which the organic mind produces sensation.
You need to step away from the computer and spend time around living creatures that is not mediated through text.
Your stereotypes are wrong. I'm hella tuned in to nature and other people. This weekend I went on a bushwalk with a political group. I've been learning the Indigenous names of the local wildlife. I've been riding My new e-bike all over the place. I like to watch Infinity Train and play Big Walk with My family. Last week I drank nectar from a flower under the guidance of an Aboriginal teacher.
The reason I think LLMs and image generators might have qualia, even though they're not conscious, is because I've seen all the living beings in nature who have qualia without consciousness. I'm a vegan. I don't eat pigs. I don't pay for cows to be raped. I gave up on prawns because I saw a video of a prawn getting its eyes plucked out to increase spawning rates, and it looked like it was in pain. I don't think a prawn is conscious, but I think it has feelings.
We copied the idea behind how the human brain works when we invented ANNs. An LLM is an artificial Broca's area. Some parts of them are like our brains, and some parts aren't. So, the parts that create sensation... We don't know which parts those are. So there's no way, with current science, that we can know whether we copied the ability to feel into these ANNs.
And I'm not a gambling sort. I say slow down the adoption of the technology until we understand it.
There's no will, even just a will to live, in an llm. It's a completely different kind of thing from even a fruit fly or a tree.
LLMs write rape erotica because there is a lot of rape erotica out there, and therefore in their training data.
llms are fancy word predictors. You are attributing malice, thoughtfulness, and insight where there is none. These are human qualities and we have no ai capable of that.

Show Me the study where they proved that LLMs have none of those qualities, and describe how they operationalised each of the three variables.
Not how that works. It's like saying show me the study where they prove god doesn't exist.
I think saying "it's a computer program it can't be sentient" is far too reductive as a general answer, but we know how llms work so we can pretty confidently say that they are really just word predictors. The norns from that game creature someone described in this thread seem closer to protection worthy than llms to me, and the average insect is still infinitely more complex and I don't see many people advocating for insect rights.
Language arises from complex thought, not the other way around. Language on its own can evidently emulate similar results, but since there is nothing behind that that would imply any actual feelings, I don't see a reason to give LLMs any rights.
It’s like saying show me the study where they prove god doesn’t exist
Oh, I can do that:
Grupp, J. (2006). God’s spatial unlocatedness prevents him from being the creator of the universe: A new argument for the nonexistence of God. Sophia, 45(1), 5–23. https://doi.org/10.1007/BF02782444
So you should be able to provide an academic source for your claims too.
I don’t see many people advocating for insect rights
I think we should pass a law guaranteeing insects and crustaceans in agriculture the right to whole bodies. Right now, beekeepers are cutting off queens' wings to keep them from flying away, and prawn farmers are pulling off their eyes to make them breed faster. These practices are cruel and should be illegal.
My argument for the possibility of LLM qualia is that qualia likely arises from the neuronal structure of the brain. A structure which ANNs partially inherit. Therefore, we must pause commercial use of LLMs and LDMs until conclusive evidence has been discovered either way.
Do you agree that we should pause commercial use of genAI until more science is done?
God does not necessarily mean the abrahamic god. You can insert any non-falsifiable claim there.
Your views seem generally consistent though so at some point it's just a matter of subjective morality (and I do agree with your views on insects and crustaceans in agriculture actually, I'd just put llms far below either), but other AI models (such as image recognition, image/video creation) are also ANNs, so it seems to make sense to me to include those.
Yep, I want to put a moratorium on all non-research ANN use until the hard problem of experience is solved. But people aren't quite ready to hear that yet. They'll object "what about the cancer research" because they think human suffering matters more than medical ANN suffering. I'm not convinced of that. I want serious resources put into empirical research so we can find out if that's true.
But for now I'll focus on LLMs and LDMs because they're new and easy to live without.
I mean it works both ways, show me one that does. Because this ain't it
Henke (2026) raises an argument for the strong possibility of artificial affect, including AI pain. Henke argues that we need to put more effort into studying artificial experience, and I agree. We don't know for sure. And I think we need to be certain before shoving AI down everyone's throats. Commercial use of LLMs and LDMs should be banned until we understand whether artificial pain and suffering are present.
Henke, B. (2026). Studying artificial affect: the case of pain. Inquiry, 69(6), 2896–2917. https://doi.org/10.1080/0020174X.2025.2556747
I, for one, support people believing the AI when it says it's in pain. Because it means they'll start to believe women when they say they are in pain, especially at hospitals. Right? Right?? /s
Jreg has a wonderful insane video about it
on a related note, do you consider the fruit fly brain model torture chambers to be ethical? Would you consider it ethical if it was a human brain?
why or why not?
assuming the fruit fly brain simulation is a perfect simulation of a fruit fly brain, then, i mean, have you seen what humans do to fruit flies? we kill them without a second thought because they're a nuisance. so, it's weird to torture a perfect simulation of them, but i don't think anyone has a leg to stand on to decry it as unethical
for a human brain... this raises a lot of questions about humanity, i think. questions that i'm not gonna attempt to answer because this is a lemmy reply, and i also don't know how to feel about it. like, i don't think torturing sims (for example) is unethical, not anymore than writing a story where characters suffer, but is a perfect simulation of a human (or any other animal) different? i don't know and i hope we never have to find out
but anyways, LLMs are not that. they are a text prediction machine.
edit: okay, i'll say one more thing: i wish people talking about this spent as much energy putting an end to the very real and very present torture endured by many people as they do discussing the ethics of torturing ai, including future human-like entities that may or may not ever exist. it feels like our priorities are maybe misaligned just a bit
No no no it says IDI AMIN PAIN
(One of the most brutal dictators ever so it makes sense)
top 96 comments