ah, didn't realize you meant "racism" as meaning anti-white. To be honest I do think the relevant aspect of what makes something racist is the direction of the flow of power.

When a marginalized group is persecuted or oppressed by a dominant group, it's the oppression and power difference that is relevant even more than the narrow sense in which group categories are used to make decisions. This is why affirmative action is not racist, for example.