you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 11 months ago* (last edited 11 months ago) (5 children)

Most aren't really running Deepseek locally. What ollama advertises (and basically lies about) is the now-obselete Qwen 2.5 distillations.

...I mean, some are, but it's exclusively lunatics with EPYC homelab servers, heh. And they are not using ollama.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 2 points 11 months ago (4 children)

    Thx for clarifying.

    I once tried a community version from huggingface (distilled), which worked quite well even on modest hardware. But that was a while ago. Unfortunately, I haven't had much time to look into this stuff lately, but I wanted to check that again at some point.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 2 points 11 months ago* (2 children)

    Also, I’m a quant cooker myself. Say the word, and I can upload an IK quant more specifically tailored for whatever your hardware/aim is.

  • source
  • parent
  • hideshow 2 child comments