You don't sound thin skinned.
Having a thick skin doesn't mean being unbothered by people trying to walk over you, and thereby letting them.
I get accused of being thinskinned sometimes because of starting a confrontation over a problem or behaviour I've noticed, but that makes no sense. Being sensitive to issues is not a weakness, and being numb to them is certainly not a strength.
I can push for change precisely because I'm unbothered by the stress of working against the status quo.
But like others said, you don't always need to convince. If you say you won't cover a task because it's not your responsibility, then there is nothing to discuss. If they expect a task to be done even when you said it won't, that is not your problem.
It's theirs.