I mean the classic rebuttal is:
would saying "I only date guys" (/dudes) connotate gender neutral? Same for "That's a guy."
It's "neutral" because it falls prey to the English language issue of having male-oriented terms as the default for people (whereas objects- ships, planes, etc tend to be female)