My experience thus far...

I am in the process of rolling this out myself; I have just one HA Voice Preview, HA running in Docker (which is itself running as a VM in proxmox) + a whole stack of additional stuff (z-wave, matter, wyoming-whisper, piper, openwakeword...). The wyoming stuff is hooked up to a different stack of local llm stuff (ollama, which is using a gpu for inference, through pci passthrough), and taps into it, and piper for tts. Then in HA voice settings, I have two assistants (thinking vs non-thinking, for speed) that I setup, then back to the HA Voice Preview I can set up what assistant system to use. It's a whole thing, and there are different ways to go about it that don't require a gpu or llm, but I had the llm first so I was already halfway there, figured I might as well just use it for voice in HA too.

The HA UI section: