It's all imagined. First of all, even as a cisgender man, if I had intent to perv on women in the bathroom, the acts of invasion of privacy and assault are already illegal. If I peep into your stall, touch you, harass you (sexually or otherwise), those are already crimes. And they would be crimes if I were a cisgender woman, too, or a trans man or woman, or anything else on the rainbow. Being physically in the restroom does not inherently constitute an invasion of privacy because it is still a communal area. Nor does it inherently endanger you even if I am a man. As such, should not constitute a crime in an of itself. That is particularly true when it necessarily involves legally defining a line between gendered spaces, and that line will necessarily involve invasive investigations to even begin to figure out where on the line each person entering that space "belongs".

That rolls into my second point, the notion that you will just "know" someone's sex from seeing a fully clothed person alone is a joke. There are more than a few recent examples of cis women being accused of being trans women and there are many trans men that you would be requiring to use the women's restroom that you would be absolutely certain are cis men by sight alone. Are you going to require a bouncer check ID at the door? Check out their genitals? Do a blood test? Or are we just going to police people on vibes and just absolutely hound people that do not meet your minimum requirement of "man" or "woman" (which also means the trans people that more closely resemble their cis counterparts will pass without issue, and supposedly rape everyone).

And third, there is no evidence that trans women are a danger to cis women at all, nor that any cis men are pretending to be trans women in order to gain access to bathrooms to commit crimes because... once again... they are still crimes regardless of your perceived gender. It does not help a pervert or rapist by half-assing a disguise like a cartoon character, wearing a wig and a skirt, calling yourself a "gal", and assaulting women in the restroom. He could just as easily do that without the stupid subterfuge and it would be equally as illegal.

You can always find "a case" that they'll use to say "SEE I WARNED YOU". The point is - it is not an issue, it's a scare tactic because "the others" are scary to them. Same reason with racists and immigrants from other countries - I have heard that 4000 doctors saving lives deserve to be kicked out because there was one person from a foreign country that killed someone. Bigots don't operate on logic, facts or statistics. They find / manufacture facts that support their "view" and ignore everything else

Is there any anti-trans "argument" that doesnt make stuff up?

There are cases.

Trans people are people and can be abusive or assault others just like everybody else. Trans folk have to deal with more mental health issues than the average person. I’m just guessing, but it might make them more likely to be victims of violence or to abuse others.

These are just a few from the first page of DDG results.

https://www.fox5dc.com/news/former-student-convicted-sexually-assaulting-two-girls-released-loudoun-county-judge

https://www.kxii.com/content/news/Transgender-woman-allegedly-sexually-assaults-teen-in-walmart-505820451.html

https://nypost.com/2023/06/01/oklahoma-parent-files-suit-after-daughter-was-beaten-by-trans-student/

https://www.foxnews.com/us/transgender-wyoming-woman-convicted-of-sexually-assaulting-10-year-old-girl-in-bathroom

https://www.usatoday.com/story/opinion/voices/2021/10/29/trans-bathroom-policy-sexual-assault/8568005002/

Of course this is only anecdotal.

Bathroom restrictions don’t protect against abusers of any gender. People should feel safe using a bathroom regardless of their gender or sexuality.