--moe-cache-mib doesn't seem optimized for tight VRAM situations or if your PCIe link gets saturated by it, but it improves performance on some setups.

you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 3 weeks ago

Should be added sometime early this year. I think it was mentioned in HF blog/article when i was looking for way to use multiple models without restarting llama

  • source
  • parent