Some may have legitimate reason to downvote. To those, I'd imagine asking isn't rude.

But though harder to do in the fediverse, I've seen plenty of cases of manipulation and silence spiral tactics to be suspicious of any massive amount of downvotes. If such is an inorganic movement, I'd expect it to be either bots or paid actors, and most further reactions being negative I'd imagine that'd be to try to beat the wrongthinker further into the ground.

I'd suggest not worrying much though. Ask if you wish, as debate can be good, but keep an eye for who may be legitimate or otherwise.