Correct.

The inverse is just as true; Nazis doing something doesn't mean that it IS racist.

Hitler was a vegetarian. That doesn't make a vegetarian diet racist.
Hitler didn't smoke. That doesn't make smoking antifa.