Edit: Alternatively, does anyone have thoughts on the requirements on the model side of things to make this?

If you're talking to a model accessible via HTTP you can interact with it via a chat API. You track the messages sent back and forth yourself and post the whole conversation to the end-point each turn so that it has context. I've been experimenting with this using ollama, but I'm pretty sure you can do it with llama-server too.

I'm not sure if I know how to get the LLM to give me reliably parsable output...

Prompt engineering + sanity checks in the client side code + retries. There are options for requesting structured output specifically too. They might or might not work... I've had decent success getting qwen models to output JSON with prompts that include things like "Output strictly valid json. No extra text." or similar. Sometimes it will quote things with triple backticks or triple backticks + "json" at the top of the reply -- which is easily fixed by string manipulation on the client. Occasionally -- especially with complicated prompts -- it will go off the rails and give me junk that I've either needed to fix by hand or automatically retry later. It works well enough that I've been able to do image analysis in batches with it successfully.

I don't know anything about the specifics of LibreOffice extensions though -- never worked with those personally before.

Edit: If you don't want to go to all the effort of writing a whole extension for it, you can also just paste the context into the model running in the terminal. Put the whole document early on in the session (quoted with backticks). Then ask for refinement by quoting snippets with the comments for revision you want to do. I haven't done this for English language editing, but I have tried it for sanity checking my own hand written Python code. Sometimes it gives good suggestions (e.g. it caught a few typos I hadn't noticed yet when I tried this yesterday) and sometimes it doesn't -- your own judgement is, of course, very important.

Edit 2: I've found it's sometimes useful when I'm providing lots of text to include a description of what info I'm providing, then an instruction like "Say 'OK' to proceed." followed by something like "The whole file is: ...". That way I can provide context into the session without it wasting a lot of time trying to deduce what I want to do with it before I'm ready in a strictly user/assistant/user/assistant/... chat conversation mode.

OSS gramerly would be really nice

libreoffice+llama.cpp ? (llamacpp isn't gpl, MIT)