There's an illegal gambling game that has been extremely popular for decades called the animal game (jogo do bicho), in which each number represents an animal. 24 is the deer (veado). There's also a derogatory word for gay men, viado. The two words sound the same. So 24 is "the deer's number" but it sounds like "the fag's number".

Viado comes from desviado, which means someone who was driven off the proper path. It's just a matter of homophony (and homophobia).