by "extrapolate" I mean to learn a pattern in the training data and then apply that pattern (in a useful way) to a problem outside the domain of the training data.

I think the ability to generalize is the very definition of "machine learning", without it, models would just be really slow databases. We observed these phenomena quite frequently, like the original Groking paper, where models remembers modular arithmetic and generalize it to produce an algroithm in its weight to solve all modular arithmetic.

It is kind of hard to say whether the model are "aware" of its weight and "understand" the algorithm it developed. We certainly don't have a good understanding of how our brain works.

"clearity" and "confusion" are really hard to assess for anything other than ourselves. It is certainly valid to say that if AI can solve math problem it must understand it. In other way, Dijkstra famously argued that the question of whether machine can "think" and "understand" is completely pointless.

But the end goal of code, math and many other subjects IMO is to "communicate", so that these knowledge can be maintained and transfered. Of course, unless we want to complete forefit our understanding to machine, which I feel can be a bit dangerous especially for foundational fields like mathematics, but we will see.