I think you're being too strict with your definition of an identity because it is not just one thing. Identities are multi-faceted and fluid. I think that you ignore an important part of the picture when you ignore perceptions of you as part of your identity. They add to a conceptual cloud around you that is you and how you come across to others. I rather like to avoid oversimplification which I feel you are falling for, although I still do believe that one's own identity is most important of those and ought to be respected by others.

Have you seen the Clayton Biggsby sketch on the Chappelle show with the blind black white supremacist? He had no knowledge of being black, but I think most people would still argue that it formed a major part of his identity regardless of his own concept of himself.

To nuance your previous point, being misgendered is a negative experience because that person's "concept of you" does not agree with yours, becoming a point of conflict between you two and even inside yourself, not necessarily because they are wrong (although you are free to have that opinion). Sometimes people close to you will know you better than you know yourself.

What happens when a person has a brain injury causing retrograde amnesia, or dementia, or Alzheimer’s disease and forget the details of their lives? Are those forgotten aspects of their identity just gone? Or can they live on through their loved ones? What happens when we die and lose all possible sense of self? Is it like we never existed in the first place?