You, the person I’m talking to are so vain to think I’m talking to you.

Believing something like that without evidence, even if it's true, is crazy. That's what that line is about. It doesn't matter that the song actually is about them. The fact that they can believe it among all the other possibilities is why they're vain.