Some people do use it to mean they don't agree with the thing that was said. Heck, I will down vote certain comments because they are a detriment to the conversation even if they aren't misinformation because they might be derailing or ignoring the context of the conversation.

I also down vote some comments for being unnecessarily rude even if I agree with them.

I just have a low bar for what deserves up and down votes, but there is always a reason. Heck, I just upvoted your comment to counter a down vote.

If something is extremely offensive/cruel/bigoted/etc, it probably needs to be reported and not just downvoted... Unless it's some community for hateposting, that kind of stuff is almost always straight up against the community rules.