Another thing to review is your riding position and bike fit.
You'll notice people on road bikes will have their bodies very horizontal which allows them to put more power in to the pedal strokes. At the other end of the spectrum you have a kind of Dutch riding position where the body is straight up. This will be inherently slower just because it's more relaxed.

On an XC bike in the city you are probably somewhere in the middle of those positions, i would guess.
Then it is good to make sure you have your saddle at the right height so you are able to use all your leverage. If it is too low then you'll be pushing more with your knees and you'll need to spend more effort to move the bike. You want your leg to be almost but not fully straight when you have the pedal at its lowest position.
I agree with the other comment that if you're already doing those kinds of distances, if you keep riding then your "bike fitness" will come together quickly.