I don't really bike much when it's wet, and never in mud, but the main downside of a slicker (but not slick) tire I've noticed is on steep gravel trails. I have a knobbly in front, so descending is ok, but ascending is definitely harder.
Mind that tread isn't everything, tire size, air pressure, rubber hardness etc. all play a part. A wider softer tire might not struggle as much even without tread. (Speaking of which, I should try letting out some air next time.)