But all humans are destroying* the planet, whether Amazonians or Africans or billionaires.
Every one of those people require land for agriculture or hunting, all of them use natural resources and will until they are expended, all of them have children that exponentially increase the demand on everything we all need (or want).
Billionaires are much more harmful than many thousands and more, it is true... but examples are legion of regular old proles destroying the environment around them once their population reaches a certain level (this varies depending on the local environment).
The simpler/more primitive the society, the less stressors they put on the environment, but they remain stressors regardless.
*By destroy I mean to make it inhospitable for people as we know them, planet will be fine long after we are all gone.