The problems don't come from explicit racist language or something overt like that. It can be as simple as a trained model settling into correlations that we don't expect, and reinforcing biases in that way.

Like you give it positively scoring prior hires, who all happen to be in men's sports leagues during college. The model picks that up and scores new resumes with men's leagues participation higher.

I think they actually are told specifically to not be racist, beyond just their training data. Like, you can pretty easily lead an LLM to tell you that it's a reasonable idea to install your own swimming pool. But it will fight you if you try to get it to say something bad about a racial minority. This is almost certainly due to specific rules that the LLM companies put in place, similar to those they have to stop the LLM from telling people how to make acid or making porn.