It's rather surmountable, considering the context window is measured in the MILLIONS (soon to be billions) of tokens, and each memory(note) takes in the neighborhood of fifty. I'm pretty sure that has long since eclipsed the capacity of your own memory...
Additionally, there's zero reason a model could not have dynamic weights, to learn in real time. It's just not enabled on consumer-facing models, to provide a consistent, reliable experience. It's not a limitation of the technology.