It really depends on the type of dirt, and how wet. Most of my summer the ground is hardpack or maintained crushed gravel. So I swap to a hybrid city tire, I don't need lots of tread. That's why I have the winter set with nubs and carbide studs for other weather. Is there a bike shop or two in the local area of the trails you ride? They can recommend a good tire.
But don't worry about speed/time, just go have fun :)