Wait, is this a new community? I bet you it's a federation issue. If this is the first time that the lemvotes server has been pointed to that community. It may not have a copy of anything on hand to actually check against. Depending on how it works. Piefed and Lemmy both can take a while to populate a new community. And won't do so until the first person from its instance requests to view them. So you might not see much of anything at first but over the next few days especially if there are repeated attempts to access it should try to load in the backlog. But it's not instant. So it may just be a legitimate 404.

No what I was stating is that who the parent of a comment is can impact it as much as who made the comment. Go into a thread where anyone from Lemmy.ml has posted. If you try to look at their upvotes and downvotes it will give you a 404. If you look at the posts of anyone who replies to them it will also give you a 404. However if you see someone who replied to them post in another thread. It can report without a 404 simply because the person from lemmy.ml wasn't in the chain. If they enter anywhere into the chain you will get a 404. The same for any instance that has defederated

Oh wait I see what you're saying now interesting

That seems plausible. I wish I knew more about how it worked and could have given you better info