I understand this sucks. A lot. And you are probably right that the birth mother is influencing your child. But you can’t let it get to you and you can’t let you anger towards it (and the birth mother) show.
Unfortunately, it seems your child is stuck between his two parents. He will most likely lash out, now and in the future. And it will hurt.
I don’t have a lot of advice, just some support to give. Don’t let it get to you. Your kid loves you and you love him. Acceptance (yours towards whatever he say and will say) is fundamental even in the face of anger and rejection. You are also right that with only two hours a week there is little to do. I wish you all the best.