It's a sensory thing. For me, some fabrics are fine but some drive me nuts. It also depends on the day; sometimes I'm feeling more or less tolerant.

(no official diagnosis in my case, just "Oh, some of this sounds familiar")