There will always be content you don't want to see on the fediverse. For all you know your "ragebait" is another person's valued content and they feel the same way about you down voting it without engagement as you feel about others voting in your community. The best option is to ignore it or block it at an individual level, not rely on a system which seemingly exists on the razor's edge of perception over what constitutes a "valid" vs "invalid" down vote.