I think you are severely overestimating the amount of money needed for a holiday, or your social cohort is quite far below the average income skewing your idea of what is rich, and how many people are "rich".

Australians are pretty well paid overall and get 20 days of paid holidays a year. A quick Google says that 55% of Australians have a passport. Okay, travel from Australia is expensive, but people definitely do it every few years.

Survey from 2023 states:
The majority of Australians surveyed (85%) have been on holiday in the last five years

More than 90% of 18-27 year olds surveyed have been on holiday in the last five years.

https://www.budgetdirect.com.au/travel-insurance/research/average-holiday-cost-statistics.html

Personally I find that linking "making money" with "climbing the social ladder" is at the core of many of our issues.

Going even further, ranking each other based on factors like wealth, education, occupation, influence, etc. is degrading and leads to inequality, corruption and emotional/mental health problems, to name a few. The idea that success is measured almost exclusively by economic gain and social rank distorts human values.

I don't think you need to feel bad because it's not necessarily a financial thing. People like to travel overseas for an exotic experience. I grew up in Canada and I've been to like, 4 or 5 major cities. I have never been to Banff, which every Japanese tourist has been to. Now I live in Japan and so many people, including myself, have never been to Mt. Fuji. But I've done tours of Europe and Asia. I'm going back in October for a one month trip of Europe with my family.