It hurts THEIR environment. How can they have their ideal environment of a suburban home in a cul-de-sac if the neighbor isn't white? (And maybe also because they can't afford to buy a house with all the billionaires hoarding everything, but mostly that guy in particular not being white)