It's the implication that your marginal convenience and comfort are more important than all the damage that car-centric infrastructure causes to the poor, those who can't drive, the environment, our cities, etc. That's what makes you self-centered.
EDIT: And by the way, humans are social creatures by nature - we don't all hate being in close proximity with random strangers. In fact, in the context of an urban area, I'd argue that many of the people who do feel that way are probably suffering from the psychological damage of growing up in a suburb.