Not your fault when people are not tolerant of certain differences. This is why it's a "disability". Not because it is disabling on it's own (at least the most common forms) but because society makes it hard to interact. Just like a curb not having a ramp makes it hard for people in a wheelchair to go to socialize at the local social club/bar. It's not the disability itself, it's the lack of simple accommodation.
That said, this is why I've weeded out my friend group and love that I found a tolerant ND extrovert that brings together only people who are tolerant. It's hard finding new friends, but it was worth it for me at least.