If by "removed" you mean the R word, yes. Saying that someone's cognitive development has been retarded is no different than saying that by throwing an anchor out into the ocean, a ship's speed is retarded.
Context is important.
Those usages are completely different than the epithet of the R word when used to mean "stupid". It is being used pejoritively.
If you use valid uses of the word to defend pejoritive uses, then that is your choice to do so. In my book, it reflects poorly on you.