-nolisten is an actual option passed to the X server—your distro may do so by default—to work around a known security issue in some versions. I admit I'd have to look up the details, as it's been a couple of years since that issue was reported. Recent X versions almost certainly have a patch.