I find his style cringey and out of date. Plus he wasn't just a dick. He was championed as this great writer of women and a feminist in Hollywood while treating his stars like shit.
It's also hard for me to not see how he directed for the male gaze. If there are boobs, they're low cut with some jiggle. (Ex: In The Avengers, there's a woman that Cap saves. She gushes to a reporter afterwards. The scene that establishes her as a character who met Cap earlier? That was cut, but he kept the boob shots.)
Adam Baldwin may be worse but we are talking a paycheck for voicework compared to, say, showrunner status for Whedon.