To be fair they do go mad

Incest was rare, even among the historical pre-Westeros Targaryens. Definitely also rare in Valyria. There were a shit ton of noble houses that ruled Valyria, and this ensured a complex web of political scheming, while also diluting the incest problem. Now, another thing that also caught my attention when reading Fire and Blood is that Targaryens are, health wise, flimsy as fuck. They die like flies of things others would survive. They are also prone to neurodivergency and many were labeled as weak of disposition and often not very athletic. It is what makes the odd tale of a Targaryen with strength and prowess for combat actually stand out. So in many more ways than just facial structure, incest seems to be affecting them somehow.