Reports of AI-related child sexual abuse material rise 8,400% in two years, data shows
(www.theguardian.com)
One concerning thing is that when these AI companies are pushed to crack down on this, they'll retrain and bias their models to only recognize traditional "pornstar" body types as 'adult'.
This is a problem when they then get used as a mass content flagging and moderation tool — it's not hard to imagine some adult woman taking nudes (as people do) and just because they don't have a huge bust, getting auto flagged as having illegal material. Not to mention body types that aren't even real, like the art made in furry communities.
It's not like there isn't precedent, recently Discord banned a ton of people because its scanning system started throwing false positives, and that was only corrected because it made such a stink.