You're missing the point entirely, assuming, incorrect, and actually insulting my mother. Good work.
The term "weed" is used to mean any plant growing somewhere it is unwanted. It has nothing to do with whether or not a given species is considered invasive. The reason weeds are normally removed is simply because they're growing there they aren't wanted. They made the mistake of coming to life in someone's flower garden and they don't have flowers, or not ones the gardener considers pretty enough or fitting for the section of the garden they appeared in.
Simply because you have the ability to reach out and rip the plant from the earth and it's completely powerless to stop you, does that give you the right to do so? To kill another living thing for no better reason than you've decided that you don't think it's pretty enough, or dislike the specific place it happened to grow?
If a more capable form of life saw humans as an invasive species as we seem very likely to kill off the rest of our garden if we remain unchecked and decided to kill us off would you support that?
If they had seeded our planet with the genetics for humans with a specific range of tan skin and upon returning considered anyone with skin lighter or darker than their ideal for their garden would it be fine for them to kill off anyone who didn't fit what they wanted in a planetary garden?