Firstly, racism is always a thing. Xenophobia is how humans kept infectious diseases at bay for most of human existence. But then we did the agriculture thing and decided to try out larger societies. Throughout history, though societies have always had to contend with half the population hating the other half enough to get violent.
You've got that backwards. It's the smaller insular groups that tend to get wiped out by communicable diseases. Larger intermixing populations means better immune systems.