I know what you mean, but this is also essentially the same rhetoric that actual racists use to refute good policies like DEI. Diversity is a good thing, and I'm not sure you can achieve it by letting everyone self-police their behavior towards someone of a different race. Racism simply exists in our culture whether or not people are aware of it, so we need policies that counteract it.

Maybe a more persuasive way of saying the same thing: if you agree that some ethnicities or social demographics are historically disadvantaged and need more help than others, why would you not try to focus policy to prioritize their needs? It's similar to making policy that reduces child suffering at the expense of some freedoms for adults.

That said, I also believe it's good to try making policy that doesn't explicitly target specific races if possible. For example, strengthening workers' rights in general improves the lives of an entire economic class, and coincidentally it would disproportionately help Black and Mexican people (per capita).