You have it wrong, people don't despire nor are envious of your success.

People are just aware of this kind of "rags to riches (or just middle class)" transition based on hard work is the exception, not the rule. Countless people work as hard as you did, maybe more, and yet will see almost no improvement in their socioeconomic class.

More importantly this "success story" is so often used by capitalism and neoliberal supporters as "evidence" that "system works" and all you need to do is work hard. Which is, frankly put, just propaganda when we can all see that the main statistical factor for becoming rich is just how rich your parents already were.