No the joke is that first think they got wrong, then realize it's a factorial so they got it right (the a-ha moment!).
Then we realize that they might not have gotten it because the wrong answer is 4 and they may not know of factorials.
The ambiguity is very clever. Some other commenters call it a pun.