Ah sorry to hear. A year ago I had to disable kernel modesetting to make the nvidia drivers work for me, but I did not lose hardware acceleration, so I thought modesetting was optional for the drivers.
Your experience though sounds like the driver fails to load entirely, and the kernel switches to the standard VESA drivers, which are not really usable. Maybe it's the VESA drivers not being compatible with modesetting, or they set a weird resolution/refresh rate not supported by the actual panel?
Best of luck!