You cannot eliminate X11/wayland overhead. You need a display manager of some sort. I suspect most games/proton will require X11 or at least xwayland and a wayland compositor. You probably do want to use a window manager of some sort as well or you do lose out on a lot of controls like window placement and sizing. Some games might do weird things if they dont directly launch in full screen mode. And steam itself would probably want to be run in big picture mode to make it go full screen. If you want something designed for gaming then you might try gamescope which is what the steamdeck uses as its window manager in the game mode.
There are probably other areas with a higher impact that you can optimize more before really worrying about a lack of window manager though.