So... it does matter to them because they are looking to build connections with people and one of the ways we do that as humans is by talking together about things we both like.
So they get offended I think (I'm not sure) because they like the thing and they therefore attach it to their identity as a person, and because they feel snubbed that you didn't embrace their attempt to find common ground to build a relationship on.
I have a terrible habit of letting my mouth talk without thinking consciously about how people will react, and I've been burned by this in the past as a result. I tried to understand (when it was explained to me) but honestly, I'm mostly just going off what other people think here. If people are offended I generally view it as a them problem.