In the end, when I was setting it up, I realized that the only apps I use to connect and listen are using the rest API and I never got that one to work when I was setting it up. I had to bypass rest API endpoints like other poster here and create internal users in navidrome to keep some kind of security.
I would love someone posting up to date guide how to do it properly.
In the end problem was with the Subsonic Api and and the fact that I did not know how to implement subsonic authentication scheme on my proxy (caddy).