Well, we didn't really outlaw it. We made it only legal under certain conditions.

"Neither slavery nor involuntary servitude, except as a punishment for crime whereof the party shall have been duly convicted, shall exist within the United States, or any place subject to their jurisdiction." - emphasis mine

Your point still stands though