Counterpoint: X11 wasn't designed with today's security needs in mind, and developers were building based on the assumptions that those security holes would remain. We don't actually want everything that X11 had, we only want the good bits.
Or to put it another way, the switch from X11 to Wayland = https://xkcd.com/1172/