Hrm... might still be worth it?
Or switch a community only after it increases to a certain size or at least passes an age threshold.
Or only allow votes from "trusted" instances, where moderation practices are actually enforced and serial abusers are detected and banned.