I think this is true for lots of people. I also think there's a bunch of us that have never had that feeling of it being a memorisation task.

In fact, the reason I liked maths and science was because it wasn't memorisation. Unlike languages (for example) you could always work out the bit you forgot, and didn't need to depend on some made-up aide-memoire that only applied 75% of the time and remember what 25% it didn't apply to.

All I can think is that some early teacher failed you, and didn't lay out how the foundations worked.