Physics tends to treat numbers as inputs, outputs, and sometimes constants which are also inputs, but more like calibration to make the math describe our world. Like, c could have any value and the math would be math, but in our universe there's a particular value that produces useful outputs and all others don't. So we use that one.

As for whether or not it's worth studying, I think that depends on you. Basically no physicist in the last 50 years does any real calculating in their head. We have tools for that, and they do a fine job. So if you can't memorize your multiplication tables, who cares. Not important.

But the reasoning part is important. The problem solving part of figuring out how to put pieces together, or how to model something, is important, and also is a skill and mindset. Also, what I didn't cover is algebra, which is a set of rules for transforming an input into an output, transforming one tool into another, while keeping all the relationships intact. Still not numbers, but again, it is a mindset. Like a language, you have to learn it and become adept at it, and it takes practice to get used to it. And that is very important.

So! The good news is there's nothing stopping you! You can lookup a list of, like, highschool physics equations and constants, some highschool problems, and see if you can get some answers! Use a calculator, use Excel, whatever! Play around with stuff, get a feel for how it handles. The numbers aren't the interesting part, the equations are, and how they're used. For you, the numbers will just be something you punch in at the end to see if you got your reasoning right.

So yeah, give it a shot, there are no rules, and try not to get discouraged! 😛

Physics is a way to generally describe the relationship between things, mathematics is to predict specifically how would those relationships affect the things involved.

You can learn the general theory but application needs mathematics.

One way to do it would be instead of learning numbers use numbers as names of the objects. And other characteristics as adjectives to describe them.