It works! I discovered way too late that lemmy was rate limiting traffic from mlmym, because it wasnt part of my nginx config. It was lemmy itself that was doing rate limiting since all requests came from the same docker ip.
But forwarding the real ip headers fixed that, so very good. Huge thanks!