I think they actually are told specifically to not be racist, beyond just their training data. Like, you can pretty easily lead an LLM to tell you that it's a reasonable idea to install your own swimming pool. But it will fight you if you try to get it to say something bad about a racial minority. This is almost certainly due to specific rules that the LLM companies put in place, similar to those they have to stop the LLM from telling people how to make acid or making porn.