I wouldn't say it's wrong... But it might be inaccurate to call yourself lesbian. In that you don't fit what people expect that word to mean.

I can tell where the disconnect comes from: most non-bi people find the idea of sex with their non-preferred gender repulsive.

In that way your lived experience differs from that of most people. You have a clear preference, but it doesn't come with the sexual aversion for the other sex that most people experience.

Your experience with men sounds closer to how some asexual people describe sex, than what bi people describe. So neither "bi" nor "lesbian" correctly describe you. If someone was rude in how they said that to you, that was uncool of them.

I wouldn't worry too much. The important thing is that you understand yourself, which you seem to be well on your way on. If and when you need others to understand, you can add on some explanations.