Most slave owners in the US South weren't doing all that well. They typically had large debts and widely variable cashflow (not that they deserved sympathy for that). The ones who really made big money on slavery were the slave traders (and their investors) and the banks.