Historically, by drinking and recording how it made them feel, over the course of generations.
Currently we have the ability to perform mass spectrometry and chemical assays to determine what the active chemical compounds produced by plants are. For example: https://pmc.ncbi.nlm.nih.gov/articles/PMC8998931/
Some people swear by their effects, while others don't notice them at all. Everybody processes the compounds differently, and there's certainly a psychosomatic component at play as well.