Every person who knows you has a concept of you in their minds.
Yes, of course.
This is a part of your identity
I don't agree with calling that concept "identity". Others' "concept of you" is just that, their idea of you. That does not define you, in any way.
It’s why people are negatively affected when others misgender them.
Actually, I think this bolsters my point, not yours. The whole reason being misgendered is a negative experience is because that person's "concept of you" is wrong. They see you that way, but that is not the way you are. Your identity comes from you, and you alone.
In the end, it's obvious we have different definitions of "identity" and that's what our disagreement is rooted in. I define identity as the sum of what comprises one's sense of self.