cross-posted from: https://programming.dev/post/39212874

I recently migrated my services from rootful docker to rootless podman quadlets. It went smoothly, since nothing I use actually needs to be rootful. Well, except for caddy. It needs to be able to attach to privileged ports 80 and 443.

My current way to bypass it is using HAProxy running as root and forwarding connections using proxy protocol. (Tried to use firewalld but that makes the client IP opaque to caddy.) But that adds an extra layer, which means extra latency. It's perfectly usable, but I'd like to get rid of it, if possible.

I'm willing to run caddy in rootful podman if needed. But from what I understand, that means I can't have it in the same rootless network as my other containers. I really don't wanna open most of my containers' ports, so that's not an option.

So, I'm asking whether any of these three things are possible.

  1. Use firewalld to forward ports to caddy without obscuring the client's IP.
  2. Make rootful caddy share a network with other rootless containers.
  3. Assign privileged ports to caddy somehow, in rootless mode. (I know there's a way to make all these ports unprivileged, but is it possible to only assign these 2 ports as unprivileged?)

Or maybe there's a fourth way that I'm missing. I feel like this is a common enough setup, that there must be a way to do it. Any pointers are appreciated, thanks.

all 13 comments

sorted by: hot top controversial new old
[–] 7 points 1 year ago (1 child)
  • [–] 6 points 1 year ago (2 children)

    I'm not running your configuration so can't tell you with the assurance that I have it working but Forwarding ports with firewalld appears to address port forwarding to rootless podman using firewalld. If that doesn't work for you you might need to clarify what your firewalld configuration is that obscures the client IP. I wouldn't expect a simple port mapping to affect IP address.

  • source
  • hideshow 2 child comments
  • [–] 5 points 1 year ago (2 children)

    Rootless podman caddy doesn't need those priviliged ports, if you have your server behind a firewall device. You can map your ports on the firewall/router 80:8080 and then on the caddy container 8080:80. This way there is no need for priviliged ports and the traffic seems to go on ports 80 (and 443 the same way).

  • source
  • hideshow 2 child comments
  • [–] [S] 2 points 1 year ago (1 child)

    I mentioned in the post that it seems to make the client IP opaque to caddy.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 1 year ago

    I’ve never used your exact setup, but I have had issues with a web server behind a WAF not getting the client IP (all user traffic was shown as the WAF IP). In my case, the WAF was appending the client IP in a header, and I just had to tell web app to use that header as the client IP instead of the actual IP. Again, not sure if this helps since I have never used podman or caddy (this setup was with Wordpress and an Azure Application Gateway) but the same principles might apply.

  • source
  • parent
  • [–] 4 points 1 year ago* (2 children)

    You can use rootless caddy via systemd socket activation, here's a basic setup:

    1. rootless-caddy.service
    [Unit]
    Description=rootless-caddy
    
    Requires=rootless-caddy.socket
    After=rootless-caddy.socket
    
    [Service]
    # a non root user here
    User=El_Quentinator
    ExecStart=podman run --name caddy --rm -v [...] docker.io/caddy:alpine
    
    [Install]
    WantedBy=default.target
    
    1. rootless-caddy.socket
    [Socket]
    BindIPv6Only=both
    
    ### sockets for the HTTP reverse proxy
    # fd/3
    ListenStream=[::]:443
    
    # fdgram/4
    ListenDatagram=[::]:443
    
    [Install]
    WantedBy=sockets.target
    
    1. Caddyfile
    {$SITE_ADDRESS} {
            # tcp/443
            bind fd/3 {
                    protocols h1 h2
            }
            # udp/443
            bind fdgram/4 {
                    protocols h3
            }
            [...]
    }
    

    And that's it really.

    You can find a few more examples over here: https://github.com/eriksjolund/podman-caddy-socket-activation

    Systemd socket activation has a few more interesting advantages on top of unlocking binding priviliged ports:

    • allowing to not bind any port on the container, which itself allows to use --network none while still being able to connect to it via systemd socket, pretty neat to expose some web app while completely cutting its own external access.
    • getting rootful networking performance
    • starting up your service only when connecting to it (and potentially shutting it back down after some time)

    Drawbacks is that the file descriptor binding is a bit awkward and not always supported (caddy/nginx/haproxy do support it though). And that podman pods / kube do not support it (or at least not yet).

  • source
  • hideshow 2 child comments
  • [–] [S] 1 point 1 year ago (1 child)

    It seems that I'd still need to modify net.ipv4.ip_unprivileged_port_start=80 in sysctl, which I don't want to do. If I do it, the socket isn't even strictly necessary.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 1 year ago* (last edited 1 year ago)

    TBH I haven't played with passing caddy's podman network to other containers, mine is a simple reverse proxy to other standalone containers but not directly connected via podman run --network (or quadlet network). In my scenario I can at least confirm that net.ipv4.ip_unprivileged_port_start doesn't need to be modified, the only annoyance is that I cannot use a systemd user service, even though the end process doesn't run as root.

    EDIT: Actually looking at the examples a bit more closely I think the primary difference with my setup is that the systemd socket is started with systemd --user which thus requires the sysctl change, whereas I'm not using a systemd user service, relying instead on User=some-non-root-user to use rootless podman, but requiring root privileges to manage the systemd service.

  • source
  • parent
  • [–] 3 points 1 year ago*

    Hi,

    The client IP problem is a longstanding issue in podman's virtual bridge networks.

    As a workaround I'd run HAProxy rootless, using the pasta networking mode as that one allows seeing native client IP. With pasta's -T flag (see docs) I'd forward traffic to another caddy container binding to 127.0.0.1:8080 or something similar.

    This would coincide with your firewalld/HAProxy port-forwarding setup, but it has more rootlessness to it. It's still not perfect and you'd still need to tweak sysctls, but I hope it may be useful

  • source
  • [–] 2 points 1 year ago

    Why does it have to be a priviledged port?

  • source