Thing is, giant mansions existed in the 19th century too. They would've already been affecting the average home size in the 20th century.

The growth AFAIK is largely driven by the suburban mcmansions, which aren't necessarily a 1% thing.