You can. There are simple options, that only recognise predefined sentences, that even work on a Raspberry Pi, and at the other end of the spectrum you can host an LLM locally and chat with that if you have the right hardware (Coral isn't powerful enough for that, you want a GPU with lots of VRAM). Obviously setting this up is more complicated, but there are a lot of options to do it your way.

IIRC it's supposed to be getting better rapidly, as it's an active focus of development for Home Assistant.

That said, I thought this seemed like a good guide on how to set it up as of 8 months ago. (I'm not necessarily a fan of that guy's bombastic over-enthusiasm, but the info seems good.)