Sure, there are a minority of people who have legitimate physical disabilities that make it difficult for them to get laid, or impossible to have sex at all. I was under the impression that we were discussing the general case, though. I'm not going to prefix every comment I make with a statement about the exceptions when I'm speaking casually.
It makes conversation awkward and difficult to follow, because you have to dig through the throat-clearing and ass-covering to figure out what the person is trying to say. If you want to discuss those exceptions, feel free to bring them up, but if you feel that I'm ableist for refusing to pad out all my comments with performative acknowledgements to satisfy your asinine sense of morality, then I don't know what to tell you. Well, actually, I guess I do: "No".