I am not generalizing people based on genetic traits, race, etc. I'm talking about the culture of a broad society. It is a documented systemic issue and sociological observation. Like saying that Victorian era people in the UK were sexually repressed. There is nothing intrinsic to Victorians that made them that way, and it certainly didn't apply to everyone. But that was absolutely a significant part of the culture at the time.
X people are A. Y people can't B. Z people are worse than C people. Where X, Y, and Z are races or racial traits, these are racist statements because they attribute intrinsic qualities to a racial group, or comparatively qualify there human value. Stating a fact of sociology can often be used to erroneously support such a racist statement. It is justifiable to feel uneasy about it for that reason. But the sociological fact isn't racist in itself.
Being xenophobic in a culture where the differences between people is very small and nuanced is very common and part of human nature. Tribalism is an evolutionary adaptation that gave our ancestors an advantage when resource scarcity was a bigger issue. But it's not needed anymore like it once was. And it takes cognitive effort to overcome that impulse. The more diverse that Japan becomes over time, the less xenophobia they will become. It is not a problem with Japanese people as a race. It is a consequence of their general isolation from regular, direct contact with other cultures and people.