The hellscape we always wanted.

And then we no longer have responsible humans with their set of human connections, which are a burden in the sense of corruption and bias, but also the reason the system kinda works, because a human with their friends and relatives at least care to try to avoid jail or infamy.

That wouldn't work well. You either have that role as a node in the graph or as a responsible agent. In the former case its functionality should be very precisely defined and have no social contention value, which is not true for courts and judges, they are actually half of that social contention. In the latter case responsibility should be clear, that is, a computer program acting as that clerk or higher should transfer same and full legal responsibility to everyone in chain operating it, which is not the case now.

At the same time - nothing really new, power makes rules, the only important part is how fine-grained friction is, which leads to some equilibrium. More fine-grained is gentler, but also benefits those with most power. More coarse-grained means conflict, and eventually still benefits those with most power, but approaches the equilibrium faster. I suppose at this point we'd want the latter, and then the former.

Because, well, these changes erode social systems that had been building up for centuries, so faster is more important. I think.

We really do need to figure out what we’re gonna do when an AI fucks up and gets someone the death penalty for a parking ticket or something.

If a real lawyer were to fuck you over, you could report them to the bar association and have their license taken away. Could even sue them for losses. Will you be able to come for an AIs license? Can you get restitution from the AI company for things the AI did?

We’ve already seen AIs toast entire production environments. If a human did that, they’d be fired at the least. If an investigation found it was done intentionally, a human could potentially be sued for losses of revenue. Nothing has happened to the AIs that have done this. There is no recourse. I can think of no worse industry for a lack of accountability than law.