Physics is a way to generally describe the relationship between things, mathematics is to predict specifically how would those relationships affect the things involved.

You can learn the general theory but application needs mathematics.

One way to do it would be instead of learning numbers use numbers as names of the objects. And other characteristics as adjectives to describe them.