You can absolutely use it without a reverse proxy. A proxy is just another fancy HTTP client that contacts the server on the original client's behalf and forwards the response back to it, usually wrapped in HTTPS. A man in the middle that you trust.
All you have to do is expose the desired port(s) to all addresses:
# ...
- ports:
- 8080:8080
...and obviously to set the URL environment variables to localhost or whatever address the server uses.