I don't trust small bakers...

It is a grammar thing. You can have a lot less of a non-count noun, like sand. But you have to have fewer of countable nouns, like loaves of bread, or bakers of bread

You can use "less" when it's a non-discrete plurality, such as water or sand (ignoring the technical fact that these can now be observed as discrete components below the macroscopic level) or money (the made-up kind, not necessarily the physical representations thereof). It's vastly more messy to have 1.78 bakers, and their families get really upset about it, so it's safer just to use "fewer."