--moe-cache-mib doesn't seem optimized for tight VRAM situations or if your PCIe link gets saturated by it, but it improves performance on some setups.

you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 1 month ago

FWIW, vLLM already had a lot of this. This makes it so you can run decent models on smaller devices like phones or Raspberry Pis.

Getting pretty good results with llama.cpp and Bonsai on laptops and older phones.

  • source