you are viewing a single comment's thread
view the rest of the comments
[–] 19 points 1 week ago* (last edited 1 week ago) (2 children)

He's getting nervous because modern tech is making the LLMs easier to run at home. MoEs with intelligent routing can be amazing on well equipped gaming machines. On my AMD 7900XT with 64GB RAM (all purchased before the RAMpocalypse) using Strata I can run Qwen 3.8 Flash Next a 90GB 125b parameter model with 256k context size and get between 30 and 40 tokens per second. Strata and probably similarly FreeToken have been absolute game changers.

  • source
  • hideshow 2 child comments
  • [–] 2 points 1 week ago (1 child)

    i have a similar build except with a 9070xt, running 64k context window

    how well is 256k window working for you?

    also which model did you pick? i picked swift 1.5 iq3 xxs

  • source
  • parent
  • hideshow 1 child comment