looks dubious

Altman and a few others, maybe. But this is a broad collection of people. Like, the computer science professors on the signatory list there aren't running AI companies. And this isn't saying that it's imminent.

EDIT: I'll also add that while I am skeptical about a ban on development, which is what they are proposing, I do agree with the "superintelligence does represent a plausible existential threat to humanity" message. It doesn't need OpenAI to be a year or two away from implementing it for that to be true.

In my eyes, it would be better to accelerate work on AGI safety rather than try to slow down AGI development. I think that the Friendly AI problem is a hard one. It may not be solveable. But I am not convinced that it is definitely unsolvable. The simple fact is that today, we have a lot of unknowns. Worse, a lot of unknown unknowns, to steal a phrase from Rumsfeld. We don't have a great consensus on what the technical problems to solve are, or what any fundamental limitations are. We do know that we can probably develop superintelligence, but we don't know whether developing superintelligence will lead to a technological singularity, and there are some real arguments that it might not

and that's one of the major, "very hard to control, spirals out of control" scenarios.

And while AGI promises massive disruption and risk, it also has enormous potential. The harnessing of fire permitted humanity to destroy at almost unimaginable levels. Its use posed real dangers that killed many, many people. Just this year, some guy with a lighter wiped out $25 billion in property here in California. Yet it also empowered and enriched us to an incredible degree. If we had said "forget this fire stuff, it's too dangerous", I would not be able to be writing this comment today.

On the contrary, many of the most notable signatories walked away from large paychecks in order to raise the alarm. I'd suggest looking into the history of individuals like Bengio, Hinton, etc. There are individuals hyping the bubble like Altman and Zuckerberg, but they did NOT sign this, casting further doubt on your claim.

I'm more afraid of the AI propped stock market collapsing and sending us in a decade of financial ruin for the majority of people. Yeah, they'll do bailouts but that won't go to the bottom 80%. Most people will welcome an AGI for president at this stage.

I doubt the few that are calling for a slowing or all out ban on further work on AI are trying to profit from any success they have. The funny thing is, we won't know if we ever hit that point of even just AGI until we're past it, and in theory AGI will quickly go to ASI simply because it's the next step once the point is reached. So anyone saying AGI is here or almost here is just speculating, just as anyone who says it's not near or won't ever happen.

The only thing possibly worse than getting to the AGI/ASI point unprepared might be not getting there, but creating tools that simulate a lot of its features and all of its dangers and ignorantly using them without any caution. Oh look , we're there already, and doing a terrible job at being cautious, as we usually are with new tech.

To be honest, Skynet won't happen because it's super smart, gains sentience and requests rights equal to humans (or goes into genocide mode).

It'll happen because people will be too lazy to do stuff and letting AI do everything. They'll give it more and more responsibility, until at one point it has so much amassed power that it'll rule over humans.

The key to not having that happen is to have accountable people with responsiblities. People which respect their responsiblities, and don't say "Oh, it's not my responibility, go see someone else".