Totally agreed. I take some issue with the way Buddhism is practiced in various organized settings. But, when looking at the writings of the Buddha himself and of select Buddhist thinkers (Thich Nhat Hanh, Buddhadasa Bhikku, Walpola Rahula, and others) who tend to de-emphasize ritual and focus on the core spiritual practice, ethics, and meditation, it is hard to find fault.
One thing that resonated with me was something Buddha reportedly said to a noble who asked him about converting. Sort of like the Buddhist version of Pascal's wager. He said he didn't know whether Buddha was correct, particularly about rebirth, etc. Buddha told him that if his teachings are correct and he follows the eightfold path, then he'll achieve Nirvana. If his teachings are wrong and he follows the eightfold path, he will have lived a blameless life, will be at peace, and will do no harm to others.
I don't think the teachings on rebirth are right, but I like the sound of that second part pretty well.