268
If open weight models are the future, U.S. AI companies are going to have a hard time (www.fastcompany.com)
submitted 3 months ago by to c/technology@lemmy.world
86 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 3 months ago* (3 children)

+1 for Hermes.

If you have a newer Nvidia GPU, you can run Qwen 27B via exllamav3 and get good quality/speed in 16GB. And it's worth the trouble, as 27B is an amazing model.

If it's AMD, yeah, a3B is a good bet, depending on how much spare CPU RAM you have.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 1 point 3 months ago (2 children)

    Thanks for the tip! I do have a 5080 so I'll have to look up exllamav3

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 3 months ago* (1 child)

    You want this one:

    https://huggingface.co/turboderp/Qwen3.6-27B-exl3_3.30bpw/tree/main

    Or maybe the 3.5bpw one if you don’t mind less context, or 3bpw if you need more:

    https://huggingface.co/turboderp/Qwen3.6-27B-exl3

    For faster inference at the cost of a little more VRAM usage:

    https://huggingface.co/turboderp/Qwen3.6-27B-DFlash-exl3

    And you run those in:

    https://github.com/theroyallab/tabbyAPI

    And FYI, if you have 64GB of RAM or more, you might consider hybrid inference instead.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •