Its mostly because of what nvidia does - the newer stuff has open source drivers, but this is a more recent thing (past year or so), but a good chunk of their libraries are still closed, their firmware is still closed, and anything not supported by the newer open source kernel modules has to use the proprietary drivers.
This is, unfortunately, outside of the control of the Linux kernel. And nvidia can be made to work just fine (I have a bunch of quadro cards for example) without dropping to the console, but that first setup to make it that way takes some knowledge. That, to me, is the big problem.
Thats also why you'll see a lot of intel gpu's and amd gpu's in the linux world, while nvidia made that switch I mentioned to open on the new stuff, that was in 2024. AMD started their effort in 2015, its much more mature, and intel had open drivers for a bit over 10 years before that.