I set my caddy reverse proxy to only accept request from localhost.

It is a bit more involved but you can actually get a proper cert for localhost stuff, with your domain pointing to an internal ip addr and not risk exposing your public ip and having to open a port.