As far as I remember the rules, unarmed strike damage is 1 + Str modifier (i.e., a 1d1 damage die). And anyone untrained in unarmed strikes (not monk, not having the Tavern Brawler feat or similar) couldn't add their prof bonus to the attack roll. This makes it significantly weaker than a proper dual wielding build or something like PAM, where the attacker typically gets a proper damage die and prof bonus. Which is why it doesn't seem like a big deal to allow it.
Unarmed strikes can be done for flavor with kicks, elbows, etc. But mechanically I'd allow it as a proper bonus action if the character were wielding a single weapon without a shield. Anyone can describe anything however they want for flavor, I'm just talking about balancing the action economy.