It seems to be a generational thing from what I read in Wikipedia, older people from UK and Netherlands for example didn't remove shoes but younger generations do. The US is a big mix of cultures so I'm guessing people that came here from all over some didn't remove shoes but people from other places do, and as we go more people are removing shoes because it makes sense.

Also most people I know have switched to hard floors over carpet as well. For example when we got our home we removed all the carpet but the bedroom