No, they do. They do. If you know you're in an inappropriate setting and still do the joke, joke's on you. It would be exactly the same if it was the other way around.

Don't joke with things people care about if you're not willing to come off as an asshole. You're not doing a Netflix special. You don't have 'jesters' privilege' if no one sees you as a jester, but as a coworker.