It really depends on the type of dirt, and how wet. Most of my summer the ground is hardpack or maintained crushed gravel. So I swap to a hybrid city tire, I don't need lots of tread. That's why I have the winter set with nubs and carbide studs for other weather. Is there a bike shop or two in the local area of the trails you ride? They can recommend a good tire.

But don't worry about speed/time, just go have fun :)

If most of your riding is on paved surfaces, you want "city bike tires." For your bike, they'll be roughly 26x1.5"-2.0" and they'll be relatively smooth instead of knobby. Better ones will have features like extra puncture resistance and retroreflective sidewalls.

Random example from an image search:

The "some nubs on the outer sides but a smooth patch down the center" type of tire the commenter above recommended would look something like this...

...but honestly, I'd probably go for a full-blown city tire instead unless you're regularly riding on loose surfaces, not just "occasionally."

I run Michelin Country Rock as a hybrid tire. Anything that's not knobbly is going to be a huge upgrade on pavement.

You also don't need to run the same tires on both wheels. Having a slicker tire in the back can make sense as you put more weight on it, and slipping with the back wheel is a nuisance, while slipping with the front is often a disaster. So you could start off just swapping the rear. That way if it doesn't work out, you also spent only half the money :)

Lastly I'd want to share that I really disliked those wheels that are only knobbly along the edges. Leaning over when taking a curve at speed, and suddenly transitioning from smooth to vibrating was very unsettling to me.