I have a similar problem with caffeine sensitivity. I can have one or two coffees a week without much incident, and quite a bit more tea than that. However if I don't break the habit and give my body time to rest, the negative side effects of caffeine start compounding. I get heart palpitations which feel like anxiety and can induce anxiety if you aren’t aware of where they are coming from. I get long piercing headaches. I get pretty bad diarrhea. However I can still have a coffee now and again so long as it doesn't become habitual. It seems to me the deciding factor really is caffeine content. Decaf isn’t very good for you and still has a non negligible amount of caffeine in it. Herbal tea can be great but it doesn’t give you that buzz.