The ideological issue (which you probably don’t care about) is that it pretty much requires proprietary (non-FOSS) drivers which run in kernel space and so in theory have complete access to all data on your computer (but then so does Intel ME). This is the main reason I personally will never use NVidia cards.

The only meltdown I've had with Linux occurred on a minor rev-level update to Debian that plugged some hole in the kernel the NVidia proprietary driver was crawling through. I had used Debian and an NVidia proprietary driver for years on an ancient motherboard. Then suddenly that "solution" disappeared. I had to replace the whole machine. Yeah, it was time. No, I wasn't ready. I don't know whether I should have been more pissed at Debian or NVidia, but I'm still on Debian. After the kernel update, X11 reverted to a default driver, and no install, uninstall, reinstall combination of the proprietary drivers seemed efficacious. I'm sorry I don't remember the exact software rev-levels and drivers involved. All notes I took at the time, if any, were lost in the subsequent crash and recovery from incompetently trying to roll back the kernel update.