I've done this successfully twice with local LLMs. One of the cases was a book of data that I needed to be able to be able to pull examples from quickly, and the other was a long printed prose document that I didn't have a digital copy of but wanted to be able to reference on my kindle. The book of data was much trickier to transcribe overall (multi-column layout + text that was more prone to hallucination/typo output + entries split across pages in a way that was awkward to accumulate), but both had some challenges.

The general workflow was:

• actually scan the documents
• pass each scanned image through the LLM with several prompts, logging the results
• do a merge pass (per page) on the LLM's results including he final formatting I needed
• concatenate/merge the per-page results into a single document
• final processing (e.g. Markdown -> HTML -> ebook with calibre for the prose document)

I did a combination of automated and manual verification + some manual cleanup. In both cases the results likely are not perfect transcriptions, but were good enough for my needs.

I wrote a few scripts to run scanimage on Linux with the appropriate parameters to save into automatically numbered sequential images as I turned pages -- separating evens and odds for one of them since the book did not fit nicely on my scanner. That simplified the process of scanning a lot, but it was still quite tedious. I automatically displayed the image after each scan and had a flag to rescan the last page -- which I needed to do occasionally.

It took some experimentation to get the prompts right and I don't have a complete, easy-to-use process to share; what I've got right now needs tweaking per book/document.

Some of the prompts were things like this though:

Transcribe the text on page {{page_number}}. Do not include headers (e.g. author, book title, page number, etc).

and

Which words (if any) are italicized on page {{page_number}}? Output the word/snippet -- plus context. This may include individual words, phrases, or entire sentences. Look carefully.

Having it convert to Markdown was helpful to get styling although it wasn't always consistent about it. I had better results using simple prompts and multiple passes than trying to get it to do things all at once in a single detailed prompt.

I set a thinking budget (1000~2000 tokens depending on task) and used a low temperature. I got about ~100 pages an hour on the simpler prose document (per pass) with Qwen 3.6 at Q6_K quant on an R9700 GPU. I suspect you could probably do similar with Gemma4 on a lower VRAM GPU if needed.