The nice thing is you can give a container full hardware access if you wanted too. So if perf was a must, just steal the whole GPU for the container.

Though my ideal would be sidecar container to base desktop container. Just share what you need bus, and device wise.