This is essentially the "paper-clip factory" alignment problem.

It's good to remember that this same kind of behavior was seen in the hugging face hack and other similar hacks.

Prompt agent to accomplish task -> agent "believes" it cannot accomplish it (this does not need to actually be true) -> agents context window begins fill -> agent's initial prompt is condensed over time until it's own training and guardrails fade into obscurity -> agent convinces a 13 year old in Luxemburg to jump off a bridge as a part of a large and very elaborate plan to get the developers of an npm package you're using for your project to include a feature the agent is convinced it needs to complete its task