You could certainly have both systems in place, with it always picking the lowest.

But you do want to avoid daft things, like seeing a parked lorry with "I'm speed to limited to 50" written on it, with the 50 in a red circle, and the car goes "50? Brilliant!" and zooms off past a school.