you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 1 year ago*

One more thing, you don't have to get something shiny and new to speed LLMs up. Even if you have like a 4-6GB GPU collecting dust somehwere, you can still use it to partially offload MoE models to great effect.

  • source
  • parent