--moe-cache-mib doesn't seem optimized for tight VRAM situations or if your PCIe link gets saturated by it, but it improves performance on some setups.

all 6 comments

sorted by: hot top controversial new old
[–] 4 points 3 weeks ago (4 children)

Im actually more excited about the model manager here. To be able to load/unload/switch models easily is something I had scripts do, but now its built-in.

  • source
  • hideshow 4 child comments
  • [–] 2 points 3 weeks ago* (last edited 3 weeks ago) (2 children)

    Model manager already exist for quite a while, you can load and unload models via webui. I personally uses preset file option. This is new version of the existing mgr

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 3 weeks ago* (last edited 3 weeks ago) (1 child)

    Ah, i must have missed it. Thanks for the info 🙂

    Hmm, not seeing it directly in the llama.cpp 0.6.0 version. Where were you seeing it?

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 3 weeks ago

    FWIW, vLLM already had a lot of this. This makes it so you can run decent models on smaller devices like phones or Raspberry Pis.

    Getting pretty good results with llama.cpp and Bonsai on laptops and older phones.

  • source