137
Deleted (aussie.zone)
submitted 4 months ago by to c/selfhosted@lemmy.world
158 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 14 points 4 months ago (5 children)

Yes, I got a Strix Halo machine before the RAM price hike and use it to run all my ML stuff on it.

Currently using llama-swap with llama.cpp/ComfyUI and opencode/Open WebUI as frontend.

I'm running Qwen3.6-27b, Voxtral Mini 4b, Piper and Qwen Image. Also, some embedding and reranking models.

I use them for:

  • Tagging and classification of my documents in Paperless
  • Home Assistant (voice assistant)
  • Translations (both text and image)
  • Transcriptions
  • Some light coding and debugging
  • Avatar/Backdrop generation for DnD sessions
  • permalink
  • fedilink
  • source
  • hideshow 5 child comments
    •
  • [+] [S] 2 points 4 months ago (5 children)
    [deleted]
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 3 points 4 months ago (4 children)

    About 200 t/s prompt processing and 10-20 t/s with MTP.

    Greatly depends on the task, predictable things like code generates at 18-20 t/s. Creative writing more like 10-17 t/s.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [+] [S] 2 points 4 months ago (4 children)
    [deleted]
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 3 points 4 months ago (3 children)

    Given the 27b is a dense model, I think the numbers are quite ok. Curious about the quant tho.

    The cool thing about the strix is its large unified memory, but it lacks memory bandwith for compute intensive workloads. Something like Qwen3.5-122b MoE with only like 12b active parameters might run at twice the speed if it fits the configuration.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 4 months ago (1 child)

    Curious about the quant tho.

    Q8 from unsloth.

    Something like Qwen3.5-122b

    My go to model for knowledge. Definitely much faster at Q5 but it lacks the tool calling quality of the Qwen3.6 models. Really hoping we see a Qwen3.6-122b soon...

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 4 months ago

    In case you missed the Ornith 1.0 release (Qwen and Gemma RL finetunes for agentic / coding workloads), they look interesting to bridge the gap until we see larger 3.6 models or a 3.7 release. I didn't test them yet but according to benchmarks, the 35b MoE seems to be more or less on par with Qwen3.6 27b dense, while ofc a lot faster.

  • permalink
  • fedilink
  • source
  • parent
    • load more comments (-1 replies)
    • load more comments (-1 replies)
    • load more comments (-1 replies)