Maybe a better alternative to Tiel Coder for people with 32GB RAM and 8GB VRAM

you are viewing a single comment's thread
view the rest of the comments
[–] [S] 3 points 2 weeks ago

It's a MoE model so it's much more flexible than a dense model, see my guide for llama.cpp https://lemmus.org/post/24235317

Now there's also --moe-cache-mib that could beat --n-cpu-moe depending on your exact hardware. I'll need to write an updated guide.

And now there's Strata and PolyStrata which get even better performance than that.

  • source
  • parent