Yep, every time I've heard this from a man (and I am one) it's been the biggest red flag, and the woman was clearly the more responsible and dutiful one when all the facts were considered. I've learned to take these kinds of statements from men with a Mt Everest-sized grain of salt. I suppose it could go the other way too, but I've only ever seen men saying these things.
These same men will go through hell to gaslight and distort the reality and opinions of those around them to paint their former partners as crazy villains, and the courts as unjust. In reality, family court has seen it all and will take literally everything they can into consideration for an impartial decision - it's literally their job.