Nothing says that the observer which collapses a waveform needs to be conscious.

I’m not afraid of a little math so I’ll give the book a try, thanks for the recommendation!

This is really interesting. Do you know of anywhere that I could read more about comparing neuronal function to transformer architecture?

I have a dryer. All clothes go in the dryer because outside is wet and inside is tiny. If clothes don’t like the dryer, they shrink and we can’t wear them, therefore all clothes are dryer-safe. After dryer, clothes are piled on couch. When clean clothes are needed, they’re on the couch.

It’s interesting that the one person who actually seems to know what they’re talking about is the one getting downvoted. AI is bad at many things for many reasons, but that doesn’t mean we should just assume that anything derived from AI is automatically slop.

All that stuff is coming. It’s taking longer than people would have expected, but it’ll happen sooner or later. You’re right that we seem to be stuck with the dystopia, though. We’ll fight over an ever dwindling pool of work until there’s nothing left, and then we’ll die or live at the sufferance of those in charge. The only hope is complete systemic change, but that seems further away now than it has ever been, mainly because not enough people even care. They’re pacified by their streaming content and drip fed carefully cultivated outrage-bait to keep them from even imagining that uniting is a possibility.

But why do we need to work to earn money? Because if those jobs didn’t get done we couldn’t survive. What OP is asking, I think, is if those essential jobs were automated, nobody would need to work, so why aren’t we talking about that?

sesquipedalianist

 A person who tends to use and enjoy long words.

You can train it “you want to help me” and it will or you can train it “you want to wage war” and it will.

This isn’t how current AI models work and it looks like they never will (at least not without a completely different paradigm), but to answer the hypothetical;

-automate all agriculture, resource extraction, logistics and production,

-abolish almost all financial systems,

-provide nutrition, shelter, education, healthcare, etc. to all humans,

-allow humans to spend their time as they see fit.

I don’t see any path to get us to this, but if we somehow managed it I would imagine most people would take up sports, art, science, or hobbies, and would choose to spend more time with family and friends.

Geopolitically, I suppose the countries with the highest ratio of natural resources to population would end up dominating, unless some kind of international framework was enforced which ensured resources were distributed fairly.

This is why Musk and co are trying to speedrun robot development.

I have no doubt that they would push for any regulation to advantage them, and they have the resources to achieve that. We still need to try. And the fear of an AI apocalypse isn’t something they just dreamed up recently to make their product look ‘good’ (or powerful, or whatever), nor was it just fished from sci fi. The worries about this scenario were why Altman, Amodei and Musk (among others) got into this business in the first place. They see ASI as inevitable, they sincerely believe it will be powerful enough to destroy humanity, and they each have a messiah-complex which makes them think that only they are intelligent and incorruptible enough to be in charge when it happens.

Of those three premises, I very much doubt any billionaire CEO is incorruptible, but the inevitability of ASI and the power that it would have if it was created are harder to dismiss. If there is even a minuscule chance that they are correct, then AI development should be stopped, until they can prove that it is safe and beneficial to everyone.

Edit: and I take it as a compliment that you only give me a 10% probability of being a bot. :-)

Context matters. In science, if you are proposing something which will upend the established consensus, the burden of proof is on the claimant to show that some detail has been missed, or that some other theory fits the evidence better. What Coxon and others are claiming is not a scientific theory, however. How could anyone provide evidence to prove that building an AGI would likely result in the extinction of the human race? What kind of evidence would be acceptable? And if we continue building bigger and smarter models and before we can gather the evidence it turns out the premise is correct, what then? ’Well, we’re all dead, but at least we stuck to our guns regarding that old axiom.’

I think that, given the potential stakes, the burden of proof should be on the AI companies to show that their technology is not harmful, and until they do they should be restricted from pursuing it. By that I mean that they need to show that their existing models are safe and can be controlled, and if they can’t they need to roll back to the Last Safe Model in their roster. I’d even go further, putting the onus on the companies to show that they are not on net damaging our environment, that their models cannot be compromised to provide terrorists information on building bioweapons, etc., that they will support and help strengthen the vulnerable and never (again) encourage people to commit suicide. And that’s just off the top of my head.

The harms already caused by this technology are staggering, and we need laws to put the brakes on development so that we can collectively adapt, and if the leaders of these companies are offering us a way to do that, let’s grab that opportunity.