As GreyShuck said, many (most) teabags use plastic. Not only are these not compostable, they also release upwards of 10 to 20 billion particles of microplastic while steeping in your cup, so best not to use them at all if avoidable (you can cut the tops off the top the bags and dump the contents into a stainless steel tea steeping container to not waste what you already have).
Off the top of my head, I know that Yogi brand teabags do not use plastic in their bags, so those should be compostable.
👏 1