Pedestrians always have right of way in my state, regardless of legality. They can indeed step out into a busy road and it is entirely the drivers responsibility to stop. Ensuring safety in a shared space is always heavily biased towards the person who is operating a deadly weapon, as it should be.

If you can't be sure you wont kill a person walking, even one making irresponsible choices, you need to slow down, drive more defensively, or not drive at all.