The quoted numbers are the interesting part. 380 GB just for weights plus several GB of context per user means this only pays off at Revolut's scale. What stood out to me is that the article credits the biggest gains to NVFP4 quantization, splitting prefill and decode, and context tuning rather than adding GPUs. Those tricks carry over to much smaller setups too.
Rule of thumb that holds up well: model file size + room for context + a bit for the process.
- 7-8B at 4-bit: ~4-5 GB file, so 8 GB total is comfortable
- 13B at 4-bit: plan ~12 GB
- 70B: not a CPU thing, don't bother
The one people forget is context. Cranking num_ctx (or OLLAMA_CONTEXT_LENGTH) up because the model "supports 128k" eats memory fast, and the usual symptom is a model that loads fine and then dies halfway through a long answer.
On CPU, expect roughly 5-15 tok/s for a 4-bit 7-8B on decent cores. Memory bandwidth matters more than core count after a point.
ARM is a non-issue these days for the usual self-hosted stuff; most popular images ship arm64 builds (check the tags on Docker Hub before you commit to one). For a box at your mum's, I'd honestly consider staying on macOS + Docker or OrbStack rather than Asahi, purely because remote fixes are easier when the OS is the boring one. Add Tailscale so you can reach it without opening ports. 8 GB is plenty for things like a photo backup, an ad blocker, or a small search engine like SearXNG; it's tight only if you try to run local LLMs on it.
What's stuck for me after trying too many things:
- Ollama (or llama.cpp server if you want control over flags) as the model server, bound to localhost only
- Open WebUI as the front-end; it also handles OpenAI-compatible APIs, so local and hosted models live in the same history
- LiteLLM in between once you have more than one backend: one endpoint, per-key budgets, and logs
- SearXNG as the search tool for anything agent-ish; keep concurrency low or upstream engines start blocking it
Biggest lesson: RAM decides what you can run, and document upload/embeddings in Open WebUI eat more memory than chat does. On small boxes, point embeddings at an API or give it 4 GB.