The instance admin can see votes. If you are noticing a pattern maybe ask them to confirm? There can be delays in federation so you might see them all come in at once even if they occurred quite a bit earlier.

I know your question was "why" but maybe confirming that is that is happening vs a different cause can be the first step.