Just the materials? That's unlikely - some guy has to manually fasten together every cubic centimeter of wall and floor, and they need to be paid. IIRC, labour actually comes out to more than the materials. For a remodel, labour definitely costs a lot more.
Modular homes can theoretically do better by shifting that labour to a factory nearby, but it's still not going to be nothing.
Still, cheaper is better, just like for every other good, even if, as in every economic change, there's some losers.