No obviously not. I wouldn't date black or Asian folks for instance, or fat people, or dumb people or overly outdoorsy folks or a right-winger or a religious person or anyone under 5'10" or a cis woman. I'll happily stand to defend all those folks rights and I'll stand by them hand in hand in solidarity, we just won't fuck.
This is ok. This is normal. When it comes to personal association, especially sexual, freedom of association - is a core tenet of any libertarian socially progressive ideology worth it's salt. And that's the kind of progressive I'm down for.