Yeah.... People just don't know how easy it is to break a bone in your hands. Skulls are made up so much harder and more dense bones than your metacarpals.
There's a reason even people with years of conditioning as boxers or mma fighters still wear gloves and protective hand wraps, and it isn't to protect their opponents'faces.