I was under the impression that there is in fact such a thing as a complete mathematical system (if you take "mathematical system" in the broader sense of "internally consistent system"), but such a system would be pretty limited and therefore rather useless.

The answer would still not be 0 as 0 is clearly still well defined within that system. NaN, undefined, etc. would be acceptable answers though. Otherwise you define:

for x > y, y - x = 0

Which defines that x = y

Resulting in the conditional x > y no longer being true

Also x/0 isn't NaN. It's just poorly defined and so in computing will often return "NaN" because what the answer is depends on the numbering system used and accidentally switching/conflating numbering systems is a very easy way to create a mathmatical fallacy like the one above.