Edit: I'm wrong, and I learned something new!
Spices were associated with rotting food, because they were often used to cover up the flavor of overly game-y meat — at least that was common in the U.S. .
I don't think you can blame people for eventually realizing that if a piece of meat has a suspicious amount of spice on it, it might just make you sick. You also can't blame someone for being averse to the flavor afterwards — spice kicks twice, and if that second kick is also associated with one of the most violent shits you've ever taken... well you get the picture.
That being said, I do love spice. Not a terribly huge fan of the second kick, but such is life.