I can’t be much of a help with Caddy however, for Traefik you can use the OIDC Middleware to forward requests to your authentication service.
Plus I worry I set something up wrong and expose my network
The only port that would need opening is :443, leave port :80 closed so that people cannot connect to your services insecurely. Slap fail2ban or geoblock on it and call it a day. Also, DDNS allowlist for that deny-first approach.