Honestly, I do feel you OP.

While I can tolerate large gatherings and small talk, I really don't enjoy them at all. It can be overwhelming and exhausting at times, and IMO I'd rather be at home alone than do those things.

The only reason I like where I work is because I get on really well with the two other people in my office.

And that leads to the kind of social interaction I actually enjoy - socialising and doing things in small groups with people I know well.

I'm not suggesting this'll be your panacea, but perhaps you need to find that for yourself. A small group of trusted friends you can just let yourself relax with.