I haven't done such a deep dive, but I have noticed specialty coffee can get real expensive. There's a local roaster who sells some really good beans, but it's the sort of thing I only buy occasionally as a treat, as opposed to the tea I drink pretty much daily.
And yes, the cafe isn't really a fair comparison, since you're paying for the staff and the building and such, but for me, when I'm at work I have a choice between grabbing a drink from one of the cafes on the block, or putting on the kettle. It's about the same amount of time and effort either way, so I think it makes sense to compare them.