Cancer is also not the only harmful thing radiation does - you can also get infertility, birth defects, etc at lower doses.

This is a good point. eg, there was even mention of decreases in all cause mortality associated with 'low' dose radiation, but there's quantity of life and quality of life. "Oh yeah, you'll live 6 months longer than anyone, but we have to take a random body part each couple of years"