Using they/them exclusively and intentionally is misgendering, especially if you know someone's pronouns. Using the singular they as part of natural speech isn't. There's a grey area in the middle as well.
I get they/them'd a lot by people trying to be inclusive. My goto response is usually along the lines of "I appreciate the sentiment but my pronouns are..."
If someone is being a hateful bigot just walk away