You kill 1000 plants, feed 50 animals. Then kill those 50 animals to feed about 10-15 people. If you remove the animals from this equation, you can feed 30-40 people using those 1000 plants. Basically, if people shift to plant based diet, not only you are reducing harm done to animals, but plants too.
Veganism means you are reduce harm and pain caused to every living being.
Ps - obviously those numbers and proportions aren't exact, but I am sure I gave you a better perspective.