One of humanity’s most remarkable cultural achievements is the division of explanatory labor. When my daughter came down with a fever, I solved the problem by consulting a pediatrician, who shared a likely diagnosis of the flu. Unlike the doctor, I wasn’t able to tell my daughter much about the mechanism by which the flu leads to fever, or exactly why it was so important for her to keep drinking fluids. But the explanation I acquired—roughly “fever because flu”—was what was needed to treat her.
Society runs on this efficient type of information sharing. Just as coordinating specialized skills can streamline the fabrication of products—be it pins or cars or newspapers—coordinating specialized knowledge can facilitate the answering of questions. You can drive a car even if you don’t have the expertise to manufacture one yourself, and I benefited from the pediatrician’s explanation even though his understanding far exceeded my own.
We can thank this divide-and-conquer method for many advancements in medicine and science, engineering and commerce; in fact, it has helped drive the accumulation of human knowledge across communities and over time. But it also leaves us susceptible to illusions of understanding: cases in which we mistake the understanding of others for our own.
Yes, broadly speaking we decide if we don't know better, but it sounds right, then it is probably right.
AI excels at sounding right no matter what.
But not new, if you take a look at most business successes, you'll definitely find confidence, but often not competence. "Confidence men" have been a thing for a reason, and business leaders are usually con-men.
AI is good at being a con-man.