Although this stance makes me think I should never use remote desktop at all
Yeah, generally speaking, remote access logically puts the remote system (or whatever resources are being remotely accessed) in the same "security domain" as the endpoint being used to do the remote access. So, system administrators and other security-conscious people indeed tend not to SSH or remote desktop in to important systems from other people's computers :)