Often times when these deep fake nudes are being generated, the most significant real world harm comes from what happens when they get circulated around. I know a teacher at a school where this was an issue, and the girl who was a victim of this was actually interviewed by the police because if it had been a genuine image, then she could've been charged with creating CSAM.
The image had been shared around the school, and the girl in question felt humiliated, even though it wasn't her real body — if everyone thinks it's you in the image, then it's hard to fight that rumour mill. As to why she cared about this, well even if you, as an individual, try really hard to not care, it turns out that a lot of people do care. A lot of people called her a slut for taking such provocative images of herself, even if that's not actually what happened.
This goes beyond the deep fake side of things. I know someone whose ex distributed nudes that she had sent to him (revenge porn, basically), and it led to her being fired from her job. The problem here is that it's not always the individual whose nudes (or faked nudes) are shared who has the biggest problem with that person being seen naked.
You're free to think about people naked as much as you like. Hell, if you wanted to generate deepfake nudes, that'd be unethical as hell in my view, but there's little that could be done to stop you. Do whatever you like in the privacy of your own mind, but if people are getting weirded out, then that suggests that it wasn't something that stayed contained within one person's mind.