Fluoride has no known negative effects on adults in the amounts found in most drinking water.
Heavy metals and estrogens (if your area is particularly polluted? You can usually find reports of exactly how clean your local water is) are easily filtered out with a cheap undersink carbon block filter (it will also remove chlorine, as a nice side-effect).