Downvoting means something different to everyone. It could just be that they felt you didn't add to the conversation well with a simple LOL and some emojis.
Some people downvote whenever they see an emoji.
I doubt anyone is downvoting because they "disagree" with you on this case, since the one user that actually downvoted the post didn't even downvote you.

I would just ignore these downvotes and move on.