That only impacts communities on Piefed locally. So people unsubscribed from other instances, etc could still downvote and harm a posts visibility across the fediverse.
In any case, I personally haven't identified post downvoting on the whole to be that much of an issue that it is impossible to not notice repeat downvoters by just checking posts a few times a day for a week. Not many people do it.
Obviously if you are running a controversial community it might be different.