Even those who didn't would sometimes 'rent' slaves out from the big plantation owners when they needed extra labor.

"Look, see how useful our neofeudal system is? A few more generations and you might be a lord like us... if you help keep these slaves down"