Fortunately, this fucking windows partition I only keep for VR with my shitty Oculus Rift CV1 reminds me how fucked up the alternative is. I can't fucking wait to get a Steam Frame and ditch it.
I never bothered to remove the Nvidia drivers. The kernel won't load the kernel module if it doesn't detect the card, and the gl/vk loaders correctly load the right implementations for the new card, so i never saw a need to remove them other than an extra download and dkms step when I do an update.