Also people tend to move from lower cost to higher cost houses over the years. If house prices go up a lot, the price gap between the houses is larger.

Basically the only people benefiting from house prices rising are those downsizing.