In this case, it's easy to just say that you would like to listen to something else. You don't have to justify or explain. Its better if you don't. Music is a matter of taste. And, yes, if somone presses you for a reason why, saying you're overexposed is plenty valid.
However, assholes exist. And some people believe that their tastes ought to be universal. Sometimes, you'll just have to confront the fact that it isn't about the musical choice at all.