I’ve never had that be an issue in practice. The NV DC drivers cover this need quite well IME: https://wiki.debian.org/NvidiaGraphicsDrivers

It’s also the best route I’ve found for setting up CUDA, so two birds, one stone.