A lot of the software targets Ubuntu compatibility so that would be the easy distro choice. But, you should be able to get this stuff running on any/most distros as long as you're comfortable building/compiling the software as needed. Or using containers like Docker, etc. I doubt you'd have any issue doing this with Mint if that's your preference.

I'm just tinkering with this stuff so other people here will have way more experience. But as a point of reference I have no trouble running llama.cpp / sd.cpp on current Debian Trixie (with/without an updated Linux Kernel). This is with an AMD R9700 (32 GB VRAM) in an older Intel based system. Everyone says that running with Vulkan is faster but I've found it's good to test your own setup - My overall inference speed is actually faster using ROCm. ROCm gives me less tokens per second but for whatever reason the RAM-VRAM operations are much faster when using ROCm which leads to the overall time from prompt to result being shorter. There's likely more tweaking I could try to get more speed but I'm not heavily using AI at the moment so am happy with how it works now.

Before that had the same setup with a old Nvidia gaming GPU (6 GB VRAM), it sort of worked but was slow as hell and I could only get it to run with Vulkan. That GPU supported an older CUDA version (7.5 I think) and I never figured out how to get that working with llama.cpp + Nvidia software + Debian.