Fortunately, this fucking windows partition I only keep for VR with my shitty Oculus Rift CV1 reminds me how fucked up the alternative is. I can't fucking wait to get a Steam Frame and ditch it.
You used Linux like Windows and got bad results, OP treated Windows like Linux and got bad results. The problem is neither OS but how familiar you are with it and their peculiarities.
That being said, GPU drivers are not a rough edge on Linux, only Nvidia drivers are. And even then it's usually a single click/command to install the proprietary drivers if you need them, otherwise the open source ones work like a charm. This used to be more of a problem a few years back, when both manufacturers used proprietary drivers, but AMD ones are open now and therefore integrated into the mainstream kernel, so they just work.