I'd argue, that it sometimes adds complexity to an already fragile system.

You don't have to argue that, I think thats inarguably true. But more complexity doesn't inherently mean worse.

Automatic braking and collision avoidance systems in cars add complexity, but they also objectively make cars safer. Same with controls on the steering wheel, they add complexity because you now often have two places for things to be controlled and increasingly have to rely on drive by wire systems, but HOTAS interfaces (Hands On Throttle And Stick) help to keep you focused on the road and make the overall system of driving safer. While semantic modelling and control systems absolutely can make things less safe, if done well they can also actually let a robot or machine act in more human ways (like detecting that they're injuring someone and stopping for instance).

Direct control over systems without unreliable interfaces, semantic translation layer, computer vision dependancy etc serves the same tasks without additional risks and computational overheads.

But in this case, Waymo is still having to do that. They're still running their sensor data through incredibly complex machine learning models that are somewhat black boxes and producing semantic understandings of the world around it, and then act on those models of the world. The primary difference with Waymo and Tesla isn't about complexity or direct control of systems, but that Tesla is relying on camera data which is significantly worse than the human eye / brain, whereas Waymo and everyone else is supplementing their limited camera data with sensors like Lidar and Sonar that can see in ways and situations humans can't and that lets them compensate.

That and that Waymo is actually a serious engineering company that takes responsibility seriously, takes far fewer risks, and is far more thorough about failure analysis, redundancy, etc.