This is all the personal understanding of someone with an education in data science without an education in psychology or neuroscience to back it up. Asking for citations on the second paragraph you quote shows the lack of that intersection. You're used to data science and machine learning, but and for you it seems like hearing someone say that they think a database has become alive. You think they're high or insane, somehow detached from reality.
You should look in to what they termed the J-space and the resulting articles that have come out in the month since. Global workspace theory has been the leading theory of human consciousness for a while, and suddenly having not just a paper published showing something that matches that description far too closely in AI but the method for studying it and it turning out to be something that has been present in these LLM models publicly available for years was shocking.
You feel like you know all of what is going on 'under the hood', however you don't. It's something in active research by the people at the front of the issue. Models released years ago now considered relatively very small and out dated have things going on under that hood that we'd never known about or been capable of seeing. It seems like you get hung up on that "AGI" term, and feel like it's some definitive and clear evolution. It seems to be a term that boils down to meaning somewhat more capable. That deepening understanding of what's actually going on under the hood could lead to a relatively small design change that jumps past that.
For the evil plan... yes, it is pure Hollywood. It's also not something I'd put past several of the people with the funding to attempt. I wasn't explaining reasons an attempt would be successful, I was explaining a fairly simple plan that someone with massive financial resources and an even larger ego but shriveled ethical and moral frame might believe they're capable of pulling off.