If your shoes are covered in mud (or full of gravel???) then you take them off. But if you've just gone outside to go to the shop and your shoes are still relatively clean, you wipe them on the doormat and go in, the amount of dirt you actually bring in is miniscule.

Then there are rules about not putting shoes on sofas or beds, so if you want to put your feet up you take your shoes off then.