I mostly agree with you, but AI cannot “act autonomously”. You prompt it, and it acts based off of that prompt.

Think of it like doing the laundry: I load my clothes in and “prompt” the washing machine to do my laundry.

When the washing machine switches from the rinse cycle to the spin cycle, it’s not acting autonomously, it’s continuing to execute my prompt.

Is AI capable of doing something you didn’t expect it to do? Yes. Does that mean it’s acting autonomously? No.

I would also argue that the “power” of LLMs is vastly overstated. A large part of that is for the exact reason you say tho, people with zero knowledge or skill are playing with fire; And when they get burned they blame the machinery, not the operator.

I'm not sure it's quite so clear cut, especially with cloud-hosted AI.

Yes, the user did issue the original prompt but the service was performed by a company that was paid for it.

Legally, there's not much difference whether a company uses a human or a computer program to perform work.

Imagine the user issuing the prompt not to an AI but e.g. to a chat with a human service worker working at OpenAI. The user tells the worker "Please find a way to advance me in the queue", and then that human employee of OpenAI goes and hacks the gym software to remove someone else from the queue.

What do you think, who would be liable for that?

Let's make the argument a bit more extreme: The human asks to be advanced in the queue, and then the human OpenAI employee grabs a gun and starts killing people on the wait list.

Who do you think would be liable in this case?

It's important to remember, AI isn't just an emergent entity created from thin air. AI are computer programs created by huge corporations and in the case of cloud-hosted LLMs they are a service provided by huge corporations.

I don't believe that a service provided by a corporation should be legally treated differently whether it's provided via the help of a computer program or a human being.

I seem to recall from an article on this incident that I read some time ago that the gym's system was really insecure (like, anyone accessing the right URL could mess with it levels of insecure.)

It's a Honda from 1984, it's a favorite of the Omnissiah

Below was taken from ChatGPT. I don't know why everyone downvoted me for telling the truth. Maybe I gave a bad example. I thought giving a real world example of it ignoring my prompt would prove my point but I guess we need to go deeper too.

“trained to believe” versus “configured to answer as though it believes.” A model doesn’t necessarily have a private belief system. You can make two instances of essentially the same underlying model produce substantially different answers by changing their instructions, training data, reward criteria, or information sources.

For example, you could create three AI systems and give all three the question:

“Should the government provide universal healthcare?”

One could be optimized around libertarian principles, another around social-democratic principles, and another instructed to provide a politically neutral analysis. They could all know essentially the same facts while reaching different conclusions because they’re being asked to evaluate those facts using different frameworks.

There is also a more subtle issue: belief-curated AI doesn’t have to contain obvious propaganda. Selection of which facts to emphasize, which uncertainties to mention, which counterarguments to steelman, and even what questions it considers relevant can systematically push users toward a particular worldview.

That's crazy. I believe I was using ChatGPT one time to ask about authoritarianism and other political questions. When I exited the chat, none of it was in my chat history. I figured I must have asked about something ChatGPT didn't like because it always saves my chats. Just to be sure I did it again with other politically related questions and it did not save the chat again. Certain things are off limits I guess.