Hell, that's the whole thing with these LLM-based business/product structures, isn't it? The models are very good at creating something that looks right, leading to people being absolutely blindsided when they fail to actually do the thing that boosters and salesmen pretended they were doing.

Given that these are statistical models that function probabilistically, it seem like the obvious attitude to take would be to assume it's a question of when they fail and do something wrong, rather than if. But accounting for that inevitability undermines most if not all of the actual economic value of these things because it turns out it takes just about as much time, effort, and skill to monitor and check these things as it would to just do the damn work yourself. But as soon as you start giving these things permissions to operate independently you are setting up a time bomb and putting duct tape over the timer. You will get fucked eventually.