Because I don't go touching every surface. Most of the times I only touch the flush button with a finger tip.

Hands get much dirtier anywhere else, sitting on a park bench for example, but doesn't have the yuck factor.