There are reasons people have second homes. There are few reasons people have a second home that costs more than $5 million dollars. If they make enough money that it would be the reasonable way to have a second home, they should be paying more taxes on that wealth.