if you have the time and patience to select the correct smart devices and set them up properly, you’re fine

Yup. If your smart devices use Thread or Zigbee (specifically not wifi—though wifi can be ok too if you use an entirely separate network for smart things) to communicate to a Home Assistant server in your home, the security and privacy risks and minimal.