Deleted (aussie.zone)
๐Ÿ‘ 1
you are viewing a single comment's thread
view the rest of the comments
[โ€“] 5 points 5 months ago (5 children)

Does anyone have a recommendation for a local model that can run well on a 5070 12GB? It pretty much would only get used for help with homelabbing and simple scripts.

  • source
  • hideshow 5 child comments
  • [+] [S] 6 points 5 months ago (1 child)
  • [โ€“] 1 point 5 months ago*

    MoEs can be very fast with hybrid inference. I run Xiaomi Mimo 2.5 (a 310B model, 116GB weights) on my single 3090 + 7800 CPU, and it outputs faster than I can read it.

    It's also easier to fit long context, if you need that.

    It's best to use the ik_llama.cpp fork for that, though. It gives a huge boost to hybrid MoE speeds.

  • source
  • parent
  • [โ€“] 3 points 5 months ago* (2 children)

    Depends on how much CPU RAM you have, and how fast it is.

    As others said, Qwen 35B at the very least. But you can get better models with more CPU RAM.

  • source
  • parent
  • hideshow 2 child comments
  • load more comments (-1 replies)