I do think it has made me healthier and happier

That is awesome, I'm glad you have found a lifestyle that works for you!

that there are tons of great reasons for everyone to do the same

I would disagree - I think the biggest issue globally is hyperinsulinemia, addressing that directly probably has the biggest health payoff for the planet.

now when I do fall to that temptation it just sits heavy in my guts and makes me feel miserable and I wonder why I did it to myself.

Sounds like your gallbladder isn't getting much of a workout on your diet, are you doing low fat as well as vegan?