Yeahhh I can't think of much worse than those kinds of team building days or whatever they are.
Is it a big group?
Does your new boss know of your disability? I'd be honest about your recovery time after social events and maybe (if it's in your capacity) agree to one a year, not quarterly!
I just don't get those kinds of things, if it's not vital information and people are performing their jobs without attending these things, they really shouldn't be mandatory. If some people get something from it then by all means, run them, just don't make them compulsory and penalize people who a) don't get anything from it whether disabled or not and b) obviously where it actually has a negative affect, recovery time and so on due to disability.
You've been in the job for 5 years so you're obviously good at your job and I really hope your new boss can see that that's what matters!