the protocols underlying the fediverse are largely.. hmmm.. more like guidelines than rules.
there are literally hundreds of different platforms/projects sending around activities with lemmy being only one of those interacting places. somtimes lemmy is just a relay for non-lemmy content. it can get very complicated.
and then add to this that server operators can modify their servers reactions to activities however they see fit. most users would use shit 'out-of-the-box', some heavily modify their source projects... for example, to ignore 'delete' requests to circumvent moderation.
if you think youre ever going to find the 'correct' vote count, youre expecting more accuracy from that metadata than it will ever provide from any source.