Except that it’s been demonstrated multiple times that original training data can be extracted from a language model, so it is completely valid to talk about the model as a database, because the training data is stored within it.

It's been demonstrated that some more prominent pieces of training data can be reproduced, the majority of it cannot. This shows that those particular pieces of data are represented in some form within the model, it does not show that the way it works is equivalent to database lookups. If I can write down the lyrics of a song from memory, it shows that those lyrics are encoded as data in some form in my brain, but that doesn't mean it's valid to talk about my brain as a literal database, especially not in the sense that the limitations in the capabilities of a database can be ascribed to me (or its strengths, I cannot remember the exact lyrics of most songs I've heard, even if I can remember some).

Hill Climbing Algorithm & Artificial Intelligence - Computerphile

This video literally starts out by describing evolution as a similar optimization algorithm. If you know the basic mechanism of evolution, does that mean you can use that to then say with certainty and specificity what biological life in its vast diversity of techniques is not capable of? The "underlying operating principles" of evolution don't "understand" chemistry or deception, but they still produce organisms capable of photosynthesis and camouflage. It's an algorithm that produces other algorithms, which is what puts those resulting algorithms in a different category of comprehensibility than fixed algorithms that were explicitly written by someone. We are very far from having a comprehensive understanding of biological systems, despite knowing how evolution works.

This is not a tool that can comprehend what it is doing, it can’t be self-aware. It can only process large amounts of input data and attempt to maximize for particular dimensions. This seems vague to humans because the amount of variables being handled at any given time is far more than a human mind can focus on, but that doesn’t make the optimization routine intelligent or conscious. It’s just doing a lot of number crunching really fast, optimizing for specific aspects as directed by its developers.

This is like saying evolution is only a simple mechanism taking in the world as data, which, yeah, obviously, but that property doesn't carry forward to what it produces. The bigger problem here though is, again, concepts like comprehension, consciousness, and intelligence are not well defined in computational terms, and it is unclear what statements involving them mean in any practical sense. These sorts of claims are non-falsifiable and don't make testable predictions about the boundaries of AI capability.