Deleted (aussie.zone)
๐Ÿ‘ 1
you are viewing a single comment's thread
view the rest of the comments
[โ€“] 1 point 5 months ago*

MoEs can be very fast with hybrid inference. I run Xiaomi Mimo 2.5 (a 310B model, 116GB weights) on my single 3090 + 7800 CPU, and it outputs faster than I can read it.

It's also easier to fit long context, if you need that.

It's best to use the ik_llama.cpp fork for that, though. It gives a huge boost to hybrid MoE speeds.

  • source
  • parent