Humans are animals, too.

You're just one step away from de-humanizing other people and treating them like shit. (Or others doing the same to you.)

Recognize and reduce suffering where you can. Humans aren't that special. We're just lucky to have been the first on this planet to evolve to be smart enough to build societies and stuff.

There's plenty of science fiction stories that explore this. Oh look at these puny humans. They haven't even colonized other planets in their system. And they're still fighting amongst themselves. [Insert plenty of other good reasons that we suck, here.] Clearly, they're savages, and we can exploit them as we see fit. Oh, actually they taste really good, let's make a human farm, breed and kill millions every month.

I don't think you've thought this through. You should consider the possibility that you're just rationalizing something that has become quite normal, but actually is kind of horrific if you think of the harm that it's causing.

Some rich doof will think "workers are just workers, they don't deserve the same consideration educated people do". How would you argue against this logic? Because it's humans? But humans are also animals, how can we arbitrarily exclude ourselves from that group?

They are not putting them on the same level. Rather, you are trying to say animals have no value, not that they have less value than a human. You and every vegan agree that humans have more value than animals. Vegans simply disagree with you that animals have no value. Their lives must still be considered, even if they are not as important as a human being. They are not so worthless that your sensory enjoyment is worth more than their actual lives, even ignoring the direct harm it causes you and other humans.

Imagine if you only thought that human beings that you were personally related to or loved had intrinsic value and everyone else can be treated like, well, meat. That's very close to what you're suggesting here.