Yeah, customer facing jobs suck. I worked a pair of that type of job for a few years, and thought I enjoyed that kind of customer facing work. Then I got into a more technical, more "back of house" job where I only need to interface with people who know what they're talking about (okay, maybe only 80% fit that, but def an improvement on the 20% from before). That definitely helped with some nerves related issues I didn't even realize I was having back then.

I don't know what key skills you have that could be translated to another line of work, but it sounds like trying to get away from the customer service desk is what you should angle for.