That's very interesting and sounds cool - though might be above my paygrade.
What kind of delay does the double prompting incur? I can't imagine it would be nothing, especially for this use case of calling external LLMs via a proxy.
If anything a RAG for certain game related information might be a nice addition to the current system, rather than replacement. I think for a roleplay and storytelling focus there is a benefit of having a persistent character "life story" of long term memories fed into every prompt, creating character throughlines and potentially even character development.