Historically, slavery was often viewed in a similar way to poverty - "It would be great if there was a world without it, but the world, as it works, always has rich/free and poor/slave"

Thus, the thinking was not "Slavery itself is moral or immoral" but "There are ways of treating slaves that are moral and immoral."