I do this as well, but another approach I was thinking about implementing (i havent tried it yet) was to also block all IP addresses not belonging to mobile networks or residential ISPs in my country.
That way, in theory, only a mobile network IP or somone on residential wifi would pass through my firewall to Home Assisstant, and this would filter out IPs belonging to datacentres which may be hosting hostile VPS's, Tor exit nodes, proxies, VPN exit points, etc, etc.