There are no guardrails that will work with AI, full stop. They keep talking about that in the hopes of avoiding regulation but the truth is that the models and the stolen data they're trained on will keep popping up nudes as long as that was in the training data. So you either ban GenAI at scale (great for humanity but oops the economy) or you ban the use of training on data that includes nudes.
But the issue is not a better version of a photoshop of "some celebrity", kids are doing this to each other because it's so easy to do. Do you want to explain to a thirteen year old how it's not a big deal if someone is passing around realistic nudes of them that were AI-generated? Will people always be dicks to each other? Of course, but that doesn't mean as a society we should allow it to be so readily available that kids who aren't even old enough to drive should have one-click access to it.
Personally I'd prefer if every data center disappeared tomorrow and AI companies were held liable for the theft they've orchestrated and the harm they've done. Short of that though, something needs to be done and it's likely going to have to be regulations on the training data because their "guardrails" are shit and always will be. This isn't inevitable and the only way to stop it is to start telling the AI companies, "No, you can't do that." If some clanker wanker in his parents basement wants to make an AI girlfriend, that's on him.