I ask LLMs medical questions almost daily and it gets it wrong most of the time. Do I have data or a study to back this claim up, no, but if you check the sources (Perplexity) you can just see the wrong interpretation of studies or disreputable web content sources (county fair science projects, Reddit, quack websites)

It’s sometimes useful for general knowledge if you really don’t care that it might be wrong, but I’m not sure when this would be the case with medical advice

The confidence with which LLMs misstate facts and my inability to know which of my healthcare workers is blindly trusting of these tools makes it dangerous