What do you think a conspiracy is, if not a large group of people creating "facts" out of thin air to support something so that thing seems justified, and collating those "facts" into a narrative that makes the thing as believable as possible to wide audiences?
The Latin meaning of "conspiracy" is "combined breath", or metaphorically "combined intent". It is the shared intent of racists to justify racism, and so they conspire to create a narrative that justifies racism.
What is the difference between historians collectively deciding to hide the "truth" of ancient aliens and phrenologists collectively deciding to hide the truth that phrenology has no predictive power and was retrofitted to describe black people as psychologically adapted for slavery?