You know what, you're right, it was Paul who said that specifically.
However, the Old Testament is explicitly pro slavery, to the point where it goes into detail about how to properly beat the shit out of them.
Jesus then said that he didn't come to abolish the law, but to fulfill it.
Jesus was surrounded by slavery, and yet never once said "hey guys, maybe don't own people"
So I stand by my statement