Ad hominem fallacy is when the person is attacked as an invalidation of an argument.

You were defending the ethics of drinking cow's milk, so talking about the ethics of drinking cow's milk is perfectly on topic and not a fallacy. Otherwise, you'd be able to claim ad hominem on ANY counterargument!