Pedantry incoming: "Right of way" implies drivers have to yield to pedestrians, which isn't true — they just have to not hit them. You're still absolutely right about everything else, "right of way" is just the wrong expression. You're thinking of "duty of care".