You can do arithmetic with infinite. Defining an operator on the set of hyperreal numbers is pretty simple. It's actually really cool, because it'll teach you about scales of infinities and infinitesimals.
And, while you could define a set to includes your cat, that doesn't make your cat a number. I can make a set of six apples, and none of those are numbers. That is categorically not how numbers work.
I mean, you can square root a negative in complex numbers, but you can't in real numbers. That doesn't mean that the square root of negative one isn't a number.