No is a complete sentence, you can just say no and not go. They’ll likely not feel embarrassed, justifying whatever they said as reasonable and assume you making “a big deal” out of it is because you’re autistic. They’d likely be very understanding if you told them, but calmly that it wasn’t for you because you were neurodivergent but as soon as you make a big grand stand out of it they’ll go on the defensive and treat you like an attention seeking weirdo and probably find it cringe. People don’t care about the reality others experience, if they feel defensive they’ll always be working against you