The list of opponents Goku agrees need to be eliminated is much shorter than the list of ones he wants to fight again when they are stronger. Despite most of them after the start of DBZ being existential threats.
Though he goes back and forth on Frieza lol.