Im not so sure about that. Of course its fine to not be attracted to different traits, thats not in question. The problem is what we say.
If someone says they find a phrase impolite, or that they find a meal bland, there's an implication that the thing has not met that person's expectations.
People dont sail around expressing their opinion that they find something unattractive for that very reason.