Maybe a default branch in the handler could do, though

This is exactly what it's for. #[non_exhaustive] forces you to add a default branch when matching on the error. It's already used a lot, so without a way to do that, it becomes less useful in library code.