An option to consider: LibreElec does this with Kodi, and Kodi itself can be used for this. LibreElec can be set up for a controller from boot.
The downside with Kodi is it's a bit clunky to do anything other than play media, but it can be used to launch games reasonably well or even launch a dedicated gaming interface like RetroArch for a good interface for games.
I'm sure you could achieve what LibreElec achieves using Armbian and DietPi if you're willing to spend some time configuring things.