In canon, no.

IRL, RTD didn't want to deal with viewers laughing at David wearing "women's" clothes, even though Jodie's outfit was fairly unisex IMO.

I expect the size difference would have been more of a practical issue.