Let's say you believe taxation is unjust. Suppose someone proposes a law where only non-whites pay taxes, and you support it. In that situation I will call you a racist. It doesn't matter that you believe tax is generally unjust. Supporting the whites-only tax relief is an act of racism.
Analogously, if you believe trans people should be forced to go the toilet of the wrong gender, you are a transphobe. It doesn't matter if you believe gendered toilets are dumb or whatever. The act of forcing trans people to use the wrong toilet is transphobic.