▲ 268 ▼ If open weight models are the future, U.S. AI companies are going to have a hard time (www.fastcompany.com) submitted 3 months ago by sanitation@lemmy.today to c/technology@lemmy.world 86 comments fedilink hide all child comments
[–] DJKJuicy@sh.itjust.works 2 points 3 months ago (5 children) Awesome! I'll give it a try. permalink fedilink source parent hideshow 5 child comments replies: [–] Asafum@lemmy.world 4 points 3 months ago (4 children) I can also vouch for lmstudio. If you can get Hermes running on Linux I would suggest trying that as well. It connects to lm studio and you use Hermes to communicate with the model. Iook into it as there's a lot to it, I've really been enjoying using it so far it even learns how I like to create tasks and I've stopped having to ask it to delegate certain tasks, it just knows to do it and to break down the tasks so my fairly context starved local model can handle it. As for a model, the Qwen 3.6 family of models do really well. I'd suggest the Qwen 3.6 35B a3b probably Q4 depending on your hardware. It's large, but because it's a mixture of experts model only 3b of experts are kept on vram at any one time so it stays fast. Qwen 3.6 27b is the smarter "dense" model, but trying to stay with Q4 for quality it becomes too large for 16GB vram and for me runs at like 2 tokens per second lol permalink fedilink source parent hideshow 4 child comments replies: [–] brucethemoose@lemmy.world 2 points 3 months ago* (3 children) +1 for Hermes. If you have a newer Nvidia GPU, you can run Qwen 27B via exllamav3 and get good quality/speed in 16GB. And it's worth the trouble, as 27B is an amazing model. If it's AMD, yeah, a3B is a good bet, depending on how much spare CPU RAM you have. permalink fedilink source parent hideshow 3 child comments replies: [–] Asafum@lemmy.world 1 point 3 months ago (2 children) Thanks for the tip! I do have a 5080 so I'll have to look up exllamav3 permalink fedilink source parent hideshow 2 child comments replies: [–] brucethemoose@lemmy.world 2 points 3 months ago* (1 child) You want this one: https://huggingface.co/turboderp/Qwen3.6-27B-exl3_3.30bpw/tree/main Or maybe the 3.5bpw one if you don’t mind less context, or 3bpw if you need more: https://huggingface.co/turboderp/Qwen3.6-27B-exl3 For faster inference at the cost of a little more VRAM usage: https://huggingface.co/turboderp/Qwen3.6-27B-DFlash-exl3 And you run those in: https://github.com/theroyallab/tabbyAPI And FYI, if you have 64GB of RAM or more, you might consider hybrid inference instead. permalink fedilink source parent hideshow 1 child comment replies: [–] Asafum@lemmy.world 2 points 3 months ago Thank you for all the links! Much appreciated :) permalink fedilink source parent
[–] Asafum@lemmy.world 4 points 3 months ago (4 children) I can also vouch for lmstudio. If you can get Hermes running on Linux I would suggest trying that as well. It connects to lm studio and you use Hermes to communicate with the model. Iook into it as there's a lot to it, I've really been enjoying using it so far it even learns how I like to create tasks and I've stopped having to ask it to delegate certain tasks, it just knows to do it and to break down the tasks so my fairly context starved local model can handle it. As for a model, the Qwen 3.6 family of models do really well. I'd suggest the Qwen 3.6 35B a3b probably Q4 depending on your hardware. It's large, but because it's a mixture of experts model only 3b of experts are kept on vram at any one time so it stays fast. Qwen 3.6 27b is the smarter "dense" model, but trying to stay with Q4 for quality it becomes too large for 16GB vram and for me runs at like 2 tokens per second lol permalink fedilink source parent hideshow 4 child comments replies: [–] brucethemoose@lemmy.world 2 points 3 months ago* (3 children) +1 for Hermes. If you have a newer Nvidia GPU, you can run Qwen 27B via exllamav3 and get good quality/speed in 16GB. And it's worth the trouble, as 27B is an amazing model. If it's AMD, yeah, a3B is a good bet, depending on how much spare CPU RAM you have. permalink fedilink source parent hideshow 3 child comments replies: [–] Asafum@lemmy.world 1 point 3 months ago (2 children) Thanks for the tip! I do have a 5080 so I'll have to look up exllamav3 permalink fedilink source parent hideshow 2 child comments replies: [–] brucethemoose@lemmy.world 2 points 3 months ago* (1 child) You want this one: https://huggingface.co/turboderp/Qwen3.6-27B-exl3_3.30bpw/tree/main Or maybe the 3.5bpw one if you don’t mind less context, or 3bpw if you need more: https://huggingface.co/turboderp/Qwen3.6-27B-exl3 For faster inference at the cost of a little more VRAM usage: https://huggingface.co/turboderp/Qwen3.6-27B-DFlash-exl3 And you run those in: https://github.com/theroyallab/tabbyAPI And FYI, if you have 64GB of RAM or more, you might consider hybrid inference instead. permalink fedilink source parent hideshow 1 child comment replies: [–] Asafum@lemmy.world 2 points 3 months ago Thank you for all the links! Much appreciated :) permalink fedilink source parent
[–] brucethemoose@lemmy.world 2 points 3 months ago* (3 children) +1 for Hermes. If you have a newer Nvidia GPU, you can run Qwen 27B via exllamav3 and get good quality/speed in 16GB. And it's worth the trouble, as 27B is an amazing model. If it's AMD, yeah, a3B is a good bet, depending on how much spare CPU RAM you have. permalink fedilink source parent hideshow 3 child comments replies: [–] Asafum@lemmy.world 1 point 3 months ago (2 children) Thanks for the tip! I do have a 5080 so I'll have to look up exllamav3 permalink fedilink source parent hideshow 2 child comments replies: [–] brucethemoose@lemmy.world 2 points 3 months ago* (1 child) You want this one: https://huggingface.co/turboderp/Qwen3.6-27B-exl3_3.30bpw/tree/main Or maybe the 3.5bpw one if you don’t mind less context, or 3bpw if you need more: https://huggingface.co/turboderp/Qwen3.6-27B-exl3 For faster inference at the cost of a little more VRAM usage: https://huggingface.co/turboderp/Qwen3.6-27B-DFlash-exl3 And you run those in: https://github.com/theroyallab/tabbyAPI And FYI, if you have 64GB of RAM or more, you might consider hybrid inference instead. permalink fedilink source parent hideshow 1 child comment replies: [–] Asafum@lemmy.world 2 points 3 months ago Thank you for all the links! Much appreciated :) permalink fedilink source parent
[–] Asafum@lemmy.world 1 point 3 months ago (2 children) Thanks for the tip! I do have a 5080 so I'll have to look up exllamav3 permalink fedilink source parent hideshow 2 child comments replies: [–] brucethemoose@lemmy.world 2 points 3 months ago* (1 child) You want this one: https://huggingface.co/turboderp/Qwen3.6-27B-exl3_3.30bpw/tree/main Or maybe the 3.5bpw one if you don’t mind less context, or 3bpw if you need more: https://huggingface.co/turboderp/Qwen3.6-27B-exl3 For faster inference at the cost of a little more VRAM usage: https://huggingface.co/turboderp/Qwen3.6-27B-DFlash-exl3 And you run those in: https://github.com/theroyallab/tabbyAPI And FYI, if you have 64GB of RAM or more, you might consider hybrid inference instead. permalink fedilink source parent hideshow 1 child comment replies: [–] Asafum@lemmy.world 2 points 3 months ago Thank you for all the links! Much appreciated :) permalink fedilink source parent
[–] brucethemoose@lemmy.world 2 points 3 months ago* (1 child) You want this one: https://huggingface.co/turboderp/Qwen3.6-27B-exl3_3.30bpw/tree/main Or maybe the 3.5bpw one if you don’t mind less context, or 3bpw if you need more: https://huggingface.co/turboderp/Qwen3.6-27B-exl3 For faster inference at the cost of a little more VRAM usage: https://huggingface.co/turboderp/Qwen3.6-27B-DFlash-exl3 And you run those in: https://github.com/theroyallab/tabbyAPI And FYI, if you have 64GB of RAM or more, you might consider hybrid inference instead. permalink fedilink source parent hideshow 1 child comment replies: [–] Asafum@lemmy.world 2 points 3 months ago Thank you for all the links! Much appreciated :) permalink fedilink source parent
[–] Asafum@lemmy.world 2 points 3 months ago Thank you for all the links! Much appreciated :) permalink fedilink source parent