Unlike in Windows, in Linux the graphics UI concerns are outside the kernel

is in the sense that the Linux Kernel isn't in any way form or shape optimized for any kind of graphics features, unlike in Windows. The software design concerns about graphical interfaces are in user space and, as you say, the "kernel talks to hardware and user space driver talks to kernel".

I don't see how what I wrote is inconsistent with what you wrote: the kernel only delivers access to the graphics and leaves to user space the details of what's done with it, most notably (by comparison with Windows) all the software design concerns from having a graphical user interface - I just described in high-level Software Architecture terms the "why" for the "how" that you described in Software Design terms.

Granted, with GPUs the complexity is so much more than with traditional systems (such as networking or data stores) that a lot of the performance improvement happens is in the graphics drivers, which is not quite kernel but kernel-adjacent, so it's a little less perfectly split between kernel-space and user-space than what I wrote made it seem.