Spitballing:
Depending on how demanding those games are, perhaps they could still be run in VMs, just without physical GPU at all? Then you can passthrough USB controllers or attach USB devices just for the input and audio.
I guess you probably want third "management monitor" for this where you run virt-manager.
I'm guessing a "thin client" setup might not make sense here, but otherwise if you have some laptops around to use as "terminals viewers" that could also work I guess. I guess you would just do Steam Link remote play if this is just for Steam gaming though.