If you are paired against an opponent and choose not to play against them then you get an uncontested match loss and will tank your ELO pretty quickly. This is also anecdotal but I've never met a player who would refuse an opponent of roughly equal ELO