Deleted (aussie.zone)
๐Ÿ‘ 1
you are viewing a single comment's thread
view the rest of the comments
[+] [S] 6 points 5 months ago (1 child)
  • [โ€“] 1 point 5 months ago*

    MoEs can be very fast with hybrid inference. I run Xiaomi Mimo 2.5 (a 310B model, 116GB weights) on my single 3090 + 7800 CPU, and it outputs faster than I can read it.

    It's also easier to fit long context, if you need that.

    It's best to use the ik_llama.cpp fork for that, though. It gives a huge boost to hybrid MoE speeds.

  • source
  • parent