If you've ever called someone an idiot or moron, that's just as wrong (arguably more wrong) as calling someone a retard. Look up the "euphemism treadmill".

All of these terms were used to describe the same people and fell out of favor when people started using them as pejorative terms.