Ironically enough, even most of the plantation "owners" didn't really own their plantations or slaves. They were typically massively in debt, usually to English bankers who were financing the loathsome operation to provide cheap raw material for their cotton factories. This is why the British government -- which had led the way in banning slavery in the British empire and using its navy to interdict the transatlantic slave trade -- was somehow on the side of the Confederacy during the Civil War.