--moe-cache-mib doesn't seem optimized for tight VRAM situations or if your PCIe link gets saturated by it, but it improves performance on some setups.

you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 1 month ago (4 children)

Im actually more excited about the model manager here. To be able to load/unload/switch models easily is something I had scripts do, but now its built-in.

  • source
  • hideshow 4 child comments
  • [–] 2 points 1 month ago* (last edited 1 month ago) (2 children)

    Model manager already exist for quite a while, you can load and unload models via webui. I personally uses preset file option. This is new version of the existing mgr

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 1 month ago* (last edited 1 month ago) (1 child)

    Ah, i must have missed it. Thanks for the info 🙂

    Hmm, not seeing it directly in the llama.cpp 0.6.0 version. Where were you seeing it?

  • source
  • parent
  • hideshow 1 child comment