If it really sucked so much to own a home, the upper class wouldn't be buying as many of the fucking things as they could.

Owning a home is 100% upside 0% downside. You have your name on it and someone else pays for it. You use their money to pay other people to maintain it, skim the rest and fuck off to the beach. If you fail at this you wouldn't last an hour at a real job.

If all the money just goes to the interest on the mortgage then no, you aren't exploiting them, the bank is exploiting both of you. If the person is paying for your equity then you are benefiting off of that person's misfortune of not being able to own a house.

Many slave owners were relatively poor or heavily in debt, Washington wasn't solvent until after his presidency, Jefferson too. They would probably say they have to work their slaves to pay off their debts, doesn't make it right.

I feel like following this train of thought results in either nobody owns anything to keep it fair

Sort of, both anarchists and communists support the abolition of private, not personal, property, ie stuff you own not to use, but to make money off of. So you can own a house to live in, you can't own a house to rent out.

or everyone is entitled to a home for free, both of which are not realistic.

Not necessarily, the third option is public / social housing. The government owns housing and operates it at cost instead of seeking a profit. So all the money used to pay for housing is going to produce and maintain housing instead of into the pockets of landlords. It's not exploitative assuming the government is democratic, just as taxes aren't exploitative if you get a say in what happens to them.