There's a world of possibilities between "public facing server" and "turn the internet off". Like using a VPN or mTLS. Or only connecting when you are at home.
However it sounds like you aren't worried about the increased threat landscape, or at least aren't willing to compromise on your current level of convenience. That is a fair answer to OP's question.