If someone is really sick of everyone calling them a racist, maybe the problem is actually them being a racist. I can count on one hand the number of times I’ve been called a racist in my nearly four decades of life, and every time it was by a conservative claiming I was being racist against white people. (I am a white man.) I am extremely opinionated and very open about my opinions. So the problem seems not to be that everyone with a loud opinion is labeled a racist.
And yes, you were literally talking about racists not feeling able to express their opinions without consequences. I don’t view that as a problem. There should be consequences for expressing a racist opinion.
It is rather telling though that nowhere in my comment did I claim that you are racist, but that’s how you interpreted it.