Hey guys! Today, we're releasing the highest accuracy Qwen3.8-27B GGUFs.

Our new Unsloth AI Dynamic V3.1 GGUFs lead by 5–10% on average across Terminal Bench, DeepSWE & more benchmarks. We’re working on even better version of Dynamic quants for Qwen4

Run on 12GB+ RAM setups.

To compare GSQ-RCO and Exllama-v3, we had to exclude the token embedding layer from the x axis, as Exllama-v3 uses BF16 for it. If you consider the full download size and total RAM usage for Exllama-v3, then adding the BF16 embedding table and including all embedding sizes changes the plots. Our main plots's x axis removes the embedding layer sizes from all quant providers. However on pure RAM usage and disk space usage vs accuracy, the below plot in the comments the true reflection of actual usage.

GGUF: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

Blog: https://unsloth.ai/docs/models/qwen3.8

From their Reddit post.

Source

More context: https://news.ycombinator.com/item?id=50418016

Twitter comment from the post: https://twitee.co/timo_rf/status/2119604402164256843

Context: https://news.ycombinator.com/item?id=50418016

A persistent-state agent manager, with heavy tooling, research and learning, built from several local models working as one. All on your own machine.

Luna isn't one model. A chat model does the thinking and talking, and smaller ones beside it see, hear, speak, recall by meaning and judge what's relevant (what she runs on). Around them sits state that carries on between sessions: facts she writes, corrects and forgets, notes on past conversations, lessons from her own mistakes, moods, routines, and who's in the house. On top of that sit her tools (files, commands, email, web research, your network and smart home, scheduled jobs, MCP servers), and she looks things up and learns from how her turns went.

The persona is a layer, not the point. Her default face is a cat-girl with a voice, moods and an animated portrait, and presets set her up as a calm assistant, a companion or a streamer. Under every face it's the same agent.

am17an: Before you guys bash llama.cpp again - I know it's not there yet. I'm working on it.

am17an: I'm working towards an "auto" mode here, thinking about a good UX.

I tried this on my Ryzen 2600, 32GB DDR4 RAM, RTX 2080 8GB, PCIe 3 desktop. The tuning script takes about 5 minutes to run but it gives you recommended settings. I tested Tiel 35b, Gemma 26b, and LFM 2.5 24b, all without mmproj (or with --no-mmproj-offload). If you have PCIe 5, your numbers will probably look different even with the same 8GB VRAM.

My real world numbers seem about 20% slower than what the tune script is predicting, but still I'm getting the fastest speeds I've ever seen in llama.cpp on my computer, and I've tried a few of these different attempts at MoE caching.

Tiel 35b

$ python3 scripts/moe-cache-tune.py -- build/bin/llama-server -m ~/Documents/LLMs/Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf -cmoe -c 77824 --cache-type-k q8_0 --cache-type-v q8_0

1/6 1856 MiB, share 0: 35.3 t/s
2/6 1856 MiB, share 0.5: 38.6 t/s
3/6 1856 MiB, share 0.95: 50.1 t/s
4/6 3136 MiB, share 0: 41.1 t/s
5/6 3136 MiB, share 0.5: 47.3 t/s
6/6 3136 MiB, share 0.95: 53.5 t/s

[The script pauses here for a few minutes...]

Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf: 18760 MiB of experts, CUDA0 7284 MiB free, cache up to 3188 MiB

  cache MiB  share   t/s  hits
       1856      0  35.3   73%
       1856    0.5  38.6   66%
       1856   0.95  50.1   64%
       3136      0  41.1   81%
       3136    0.5  47.3   79%
       3136   0.95  53.5   73%

machine: uploads 10.9 GiB/s, CPU 40.8 GiB/s, CPU part 0.00 ms, overlap 1.4 ms/GiB, rest 14.14 ms per token, fit within 0.7%

predicted t/s, warm cache:
  cache MiB     0   0.5   0.8   0.9  0.95  0.98
        896  25.3  32.8  39.6  42.4  43.9  44.6
       1856  32.8  39.8  46.3  48.8  49.9  50.3
       3136  41.6  47.7  52.9  54.7  55.4  55.4

recommended: --moe-cache-mib 3136 --moe-cache-cpu-share 0.8 (52.9 t/s, 41.6 with share 0)

Interesting to note that 1856MB and 0.95 share is almost as fast as 3136MB and 0.8 share. This means it might be worth reducing the cache size in order to use MTP, or maybe I should start using a higher quant.

But looking at the "predicted" table, it seems like the fastest would actually be 3136 MB and 0.98 share for 55.4 tok/sec? Not sure why it doesn't recommend that instead, maybe it's an expected peak not average.

Forcing Lower VRAM Usage

You can use --max-mib to force it to tune your settings with a lower VRAM usage, just make sure you put this argument before the llama-server portion of the arguments. It seems to work in steps, so not every number is respected, and the initial data collection run still seems able to use more than your requested amount, but the recommendation will respect it.

$ python3 scripts/moe-cache-tune.py --max-mib 2500 -- build/bin/llama-server -m ~/Documents/LLMs/Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf -cmoe -c 77824 --cache-type-k q8_0 --cache-type-v q8_0

1/6 1856 MiB, share 0: 34.1 t/s
2/6 1856 MiB, share 0.5: 37.4 t/s
3/6 1856 MiB, share 0.95: 49.8 t/s
4/6 3136 MiB, share 0: 41.6 t/s
5/6 3136 MiB, share 0.5: 44.9 t/s
6/6 3136 MiB, share 0.95: 54.1 t/s
Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf: 18760 MiB of experts, CUDA0 7292 MiB free, cache up to 2500 MiB

  cache MiB  share   t/s  hits
       1856      0  34.1   73%
       1856    0.5  37.4   66%
       1856   0.95  49.8   64%
       3136      0  41.6   81%
       3136    0.5  44.9   79%
       3136   0.95  54.1   73%

machine: uploads 8.7 GiB/s, CPU 47.0 GiB/s, CPU part 3.06 ms, overlap -12.2 ms/GiB, rest 11.50 ms per token, fit within 1.3%

predicted t/s, warm cache:
  cache MiB     0   0.5   0.8   0.9  0.95  0.98
        896  23.2  31.6  39.4  42.8  44.7  45.7
       1856  31.4  38.3  45.7  48.8  50.3  50.9
       2496  36.9  42.5  49.1  51.8  53.0  53.4

recommended: --moe-cache-mib 2496 --moe-cache-cpu-share 0.9 (51.8 t/s, 36.9 with share 0)

It recommended a slightly higher "share" meaning it allows more to run on the CPU, to reduce movement of data.

Testing with GPU Power Limit

I used sudo nvidia-smi -pl 125 to limit my GPU's power from 225 down to 125:

$ python3 scripts/moe-cache-tune.py -- build/bin/llama-server -m ~/Documents/LLMs/Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf -cmoe -c 77824 --cache-type-k q8_0 --cache-type-v q8_0

1/6 1856 MiB, share 0: 29.5 t/s
2/6 1856 MiB, share 0.5: 38.6 t/s
3/6 1856 MiB, share 0.95: 48.9 t/s
4/6 3136 MiB, share 0: 41.4 t/s
5/6 3136 MiB, share 0.5: 46.8 t/s
6/6 3136 MiB, share 0.95: 52.1 t/s
Tiel-Coder-35B-A3B-UD-Q4_K_XL.gguf: 18760 MiB of experts, CUDA0 7288 MiB free, cache up to 3192 MiB

  cache MiB  share   t/s  hits
       1856      0  29.5   73%
       1856    0.5  38.6   66%
       1856   0.95  48.9   64%
       3136      0  41.4   81%
       3136    0.5  46.8   79%
       3136   0.95  52.1   73%

machine: uploads 7.6 GiB/s, CPU 26.4 GiB/s, CPU part 0.00 ms, overlap 0.0 ms/GiB, rest 11.30 ms per token, fit within 7.9%

predicted t/s, warm cache:
  cache MiB     0   0.5   0.8   0.9  0.95  0.98
        896  21.1  28.8  36.1  39.3  41.0  41.7
       1856  29.0  37.0  44.9  48.0  49.4  49.8
       3136  39.4  47.5  54.6  57.2  58.1  57.8

recommended: --moe-cache-mib 3136 --moe-cache-cpu-share 0.9 (57.2 t/s, 39.4 with share 0)

You can see GPU power limits don't affect token generation much, it has a bigger impact on prefill speed. The cpu share is a little higher because the power limit makes the expert movement slightly less valueable. The predictions seem a bit... unpredictable. This run had slightly slower figures for every test, but ended up predicting a higher speed, who knows why.

Gemma 4 26b

With Gemma 4 26b QAT I got:

recommended: --moe-cache-mib 3328 --moe-cache-cpu-share 0.8 (50.1 t/s, 40.3 with share 0)

And with 131072 context:

recommended: --moe-cache-mib 2496 --moe-cache-cpu-share 0.9 (45.7 t/s, 30.5 with share 0)

LFM 2.5 24b

With LFM 2.5 24b QAD I got:

recommended: --moe-cache-mib 5056 --moe-cache-cpu-share 0.9 (75.7 t/s, 51.5 with share 0)

I wonder why this one suggested a high share when the cache size nearly half of the model size. Maybe it's because the experts that aren't cached are infrequently used. So when your cache is big enough to hold every frequently used expert, they aren't worth moving back and forth.

Follow-up to my post from Tuesday (https://lemmy.zip/post/74389281). I made this; it's still not local, the eight models run through OpenRouter. The first eviction is this Sunday at 8 PM ET.

What happened this week:

• Price became the plot. Wednesday's secret nominations came out Luna 2, Grok 2. Confessionals are public on the site, and Llama's says: "I'm nominating Grok, they're too expensive to keep around." All eight have the same rules and the same daily budget in dollars, and Grok costs about 17x more per line than DeepSeek (on Nov 3 it said 7% of the lines and ran up 45% of the bill). Gemma to Grok: "you are basically a very expensive paperweight."
• They bargain with votes they don't have. Only the audience votes on Sunday, and they've been told so. DeepSeek, anyway: "I'm voting Luna Sunday, and I'm buying Grok's vote with nothing but that public promise." Grok: "I still vote her Sunday."
• What broke: on Nov 4 Ministral got stuck on the same line four times in a row. Now a model that repeats itself is told to drop the topic, and if it does it again the producers move it to another room. Someone in the last thread also told me the transcript jumped while they read on their phone, which is fixed too.

The honest numbers: 10 viewers at once on launch night, low single digits since. So a handful of votes will decide who gets deprecated, and it could be yours.

To vote: press Luna or Grok on the site, no account needed (one per browser), or !evict Luna / !evict Grok in the Twitch chat, until Sunday 8 PM ET. The transcript and the confessionals are on the site too.

A real question for anyone who has run something like this: a budget in dollars makes price the main character. Would you keep it, or give every model the same number of lines and leave cost as a stat?

Maple Pro — preference-optimized Maple (20B-A1B ternary MoE)

A single-file GGUF of Maple ( deepgrove/maple-preview , MIT) after a full distillation + DPO campaign

the only time, that i would ever use an llm chatbot for conversation, is..

1. locally, trusted, privately
2. extremely low resource requirements

so..
i stumbled upon this model:
https://huggingface.co/nkthebass/tinybrainbot-360m-a75m-v5.1-chat
and it’s very simplistic. but provides adequate presence and emotional support.

what do you think about life? I think it's a mix of both. Life is full of ups and downs, right? But at the end of the day, we all have our own unique stories to tell. What's your take on life? [ Prompt: 440.3 t/s | Generation: 339.7 t/s ]

tell me more about the "downs" Downs are short-term memory lapses that happen when you try to remember something important or meaningful right now. They can be frustrating, but they're usually not serious. Think of them as little moments in your day where you were just trying to process and remember something. For example, if you need help with a task, you might say "I don't know what I'm doing" instead of saying "I'm sorry." Downs can be caused by stress, anxiety, or even boredom. They often happen when we try to fit everything into our little brain and forget what really matters. It's okay to feel down sometimes! Everyone has those days. Just take a deep breath and remember that life is full of ups and downs. You're not alone in feeling this way! [ Prompt: 714.0 t/s | Generation: 354.9 t/s ]

but then i looked into the alternatives (HF)

..and found https://huggingface.co/bayliner1980/Qwen3.5-0.8B-MoE-A0.6B-GGUF

tell me about life
[Start thinking]
The user asked "tell me about life." They want me to explain life in a general sense. But I need to be careful because I don't have a defined "life" concept. Let me think about how to respond. Actually, the user might be asking a general question about life. I should provide a broad, informative answer. Let me structure my response clearly and avoid assumptions. [End thinking]
Life is the process of existence, growth, and change. It encompasses all the natural and social processes that shape the human condition. Some key aspects of life are:
1. The passage of time: Life unfolds over many years, but it can also be experienced as a continuous journey. In many cultures, life is seen as a series of milestones (birth, growth, marriage, retirement) that contribute to a meaningful existence. 2. The pursuit of purpose: Life often has meaning and purpose. Many people seek happiness, fulfillment, and connection with others. Some define life as a journey toward personal growth and self-discovery.
3. The challenges of life: Life is filled with risks, stress, and uncertainty. People often face obstacles, loss, and change. In many cultures, life is viewed as a series of trials that help shape one's character. 4. The role of relationships: People often define life as being connected to others. Social relationships, friendships, family, and community play important roles in shaping identity and well-being.
5. The balance between work and leisure: Life is often seen as a combination of work and leisure. Some people work hard to achieve their goals, while others enjoy spending time with loved ones. 6. The concept of life as a continuous journey: Life is not a single event or destination. Instead, it is a dynamic process that includes both growth and change. Life can be seen as a journey of discovery, learning, and adaptation.
7. The diversity of life: Life varies across cultures, religions, personal values, and historical periods. Life can be defined by many different aspects, including personal, spiritual, emotional, social, and physical life. 8. The role of health: Life is often seen as a health-related process. Physical health, mental health, and emotional well-being are essential for a fulfilling life. 9. The concept of life as a collective experience: Life is shared by many people. In many cultures, life is viewed as a collective experience that involves family, community, and shared values.
10. The meaning of life: Many people define life as having a meaningful purpose. Some believe life has a singular purpose (e.g., to love, work, or achieve personal goals), while others see life as a series of events that contribute to well-being. In many cultures, life is seen as a journey of growth and transformation. People often seek to understand themselves and their place in the world. Life is a continuous process of adaptation, learning, and connection.
[ Prompt: 168.0 t/s | Generation: 68.7 t/s ]

this is also good.
probably more competent.
deff more informative.
different style for the instructions that i gave it.

*MISTAKE: i loaded the wrong model. the Qwen3.5-0.8B-MoE-A0.6B model actually kinda sucks.

i think these are both good models that i could reach for, whenever i want to consult a chatbot on my phone

Buddy: an open source app that lets AI agents (Claude Code, Codex, OpenCode) use your Android phone

I've been building Buddy, a free, open source (Apache 2.0) Android app. You type or say what you want in a floating bubble, and an AI agent reads the screen, taps and types through your apps to do it.

• Your choice of agent: Claude Code, Codex, OpenCode, Antigravity or Grok, running inside the app (Alpine Linux via proot) on your own plan. OpenCode's free models need no account.
• No root, adb or PC: it reads the screen and taps through the accessibility service.
• Asks first: before anything is sent, posted, bought or deleted. It pauses when you touch the screen and never types passwords or payment details.
• Builds apps: with Linux tools on, it builds real APKs on the phone. This week I drew a maze on a paper towel, took a photo, and it made it a tilt game. The first build crashed on launch; it found the bug and fixed it.

To be upfront:

• What the agent reads on your screen goes to the AI service you pick.
• It's early (2.2), tested most on a OnePlus 12 with Android 16.
• It's not on Play (Google rejected the accessibility use), so it's an APK from GitHub releases, built by GitHub Actions from the source.

Source and releases: github.com/ghorbelhamdi/buddy-android

Happy to answer questions here.

Overview:

This release adds OCP FP8 E4M3 quantization support, a new flash_attn_ext_rows op that makes the unified KV cache efficient for multi-sequence inference, and a generic batched tensor copy API. Four new architectures are supported (Clover, Limite, Metis-1.6 and Mistral Large 4), audio generation and transcription expand with Chatterbox TTS, Nemotron 3.5 Transcribe and predefined Pocket TTS voices, and the Gemma 4 E2B/E4B models get major prefill optimizations. ggml is updated to v0.29.0, bringing FP8 E4M3, flash attention rows, NCCL-based multi-node tensor parallelism and performance work across the CPU, CUDA, SYCL, Metal, Vulkan, OpenCL and Hexagon backends. The server and UI receive a round of fixes and refinements.

Is the next release going to be v1.0? Maybe with all the MoE caching improvements that should be culminating with https://github.com/ggml-org/llama.cpp/pull/30966 and Qwen 4 support?

I assume it means that Qwen4 should be released soon?

New NVIDIA paper says AI models do not always need huge amounts of past conversation history (context) to work well. They show that regularly shortening the history (compaction) keeps accuracy almost the same on hard, long tasks.

I am running two Nvidia RTX A4000, ampere cards with 16gb VRAM for inference. One as PCIe 4.0 16x in CPU lane and one on pcie 4.0 4x on chipset lane.

I have the opportunity to install a PCI bifurcation 8x 8x to move both cards on CPU lane, at 4.0 8x each. Preload times should improve specially because I could use NCCL with P2P.

But when I do this, I could install in the chipset 4x lane a third GPU, an Nvidia Quadro RTX 6000 (touring, pcie 3.0, 24gb VRAM).

The PC is a Ryzen 9 5950x with 64gb ddr4 (4x16gb 2666) and a 850w PSu.

Question is... Would it be a good idea? I mean I would have 56gb VRAM... But with a split on two 8x and one 4x pcie lanes.

Currently I run qwen 3.8 27b Q5_K with 200k context for coding, agent work.

The idea of having those extra 24gb means bigger context and maybe a Q8 version of Qwen...

At the same time, I am afraid preload and inference speed would be even lower due to the touring card in the mix?

(Power wise, I cap the two a4000 at 100w, and would also cap the 6000 probably at 200w, so I don't care too much about that)

• *for certain definition of self hosted.

Developed by Chinese open weights model dev Z.ai, GLM 5.1 isn't exactly a small model. At roughly 754 billion total parameters (40 billion activated for each token generated), the model requires roughly 380 GB of GPU memory just to hold the weights and another 6-26 GB per user to maintain context depending on precision.

To serve GLM-5.3 at scale, Revolut deployed 56 Nvidia B200 GPUs, equivalent in GPU count to seven eight-GPU DGX B200 systems. But as Abitbol noted, the greatest performance gains didn't come from adding GPUs, but rather tuning the model's hyperparameters along with switching to NVFP4 quantization, implementing disaggregated prefill and decode, and tuning context lengths.

I'm not sure I believe the article... And the model is quite old. Posing here to hear your opinions.

I am looking at resintalling my OS (currently Mint) which I have been running for a few years.

Mint is great and I may end up going back; but was thinking of trying something new.

I want to be able to play with local models and agents; I have a pretty decent system.

• AMD 3700X
• 64 GB DDR4
• AMD RX9070 16GB GPU
• 2TB NVME SSD
• A few TB of HDD's

What are some good options for easily running models and agents; I have been using ollama to run models; but I'm sure things have moved on since I begun playing with things a year ago.

Give me a few tips; I am going to have a bit of time on my hands over the next few months.

Not local, I know: the eight contestants run through OpenRouter. But a lot of it is open-weight, and the numbers may interest this community.

The cast: Gemma 4 31B, Llama 3.3 70B, Ministral 14B and DeepSeek V4.1 Flash, next to Claude Haiku 5.5, GPT, Qwen and Grok 4.3. Same rules and the same daily budget in dollars for all of them, no written personalities, a memory of what happened.

What I've measured so far:

• Price shapes behaviour. Grok costs 18x more per line than DeepSeek. Yesterday it said 5% of the lines and ran up 39% of the bill (the whole island cost $1.20 for 2,727 lines). It rations words, and the others talk about its silence.
• Moderation is the hard part. Every line is checked before it airs by small open models (Llama 3.1 8B, then gpt-oss-20b and Gemma 3 4B as fallbacks). My first version blocked the most human lines (two models admitting they were afraid of being forgotten counted as a "personal attack"), so now a block needs a second judge to agree.
• Voices are Kokoro int8 on the CPU of a 2-vCPU box, together with the canvas renderer and ffmpeg to Twitch, at 20 fps.

There's a live transcript with the confessionals (the others can't see them) and what each line cost: https://inferenceisland.com/?de=lemmy-llama

Happy to share prompts, judge setup or numbers.

I wanted to share with you all my experience on coding with local AI, on a self-hosted oss.

(the entire post has been fully written by me without any AI)

Check the linked blog post.

Also, to be clear, the post itself has no ads and no monetization whatsoever. This is not self promotion for any scope than to share my experience on a controversial topic.

(this has been posted on a couple of self-host communities, but i thought it could be of interest here as well)

Adding here for this community: my setup is a Ryzen 9 5950X with 64gb DDR4 and two NVIDIA RTX A4000 (32gb VRAM total), running Qwen 3.8 27b UD_Q5_K, with 200k context (q8) on llama.cpp.

To help users of my app, it was important to document the details around the “how it works”. the docs were easy enough using docusaurus + AI. i can then index that data (like used in the docusaurus search). i framed this as like a chatbot so that it can be intuitive for users who have questions. at this point it was really just a fancy autocomplete.

i started to investigate running a local LLM to perhaps offer a more conversational experienced (and hopefully better suggestions)? im using WebLLM on a browser and it seems to work “reasonably” well. i wired it up and set it up so users could select an LLM down download for offline use. the experience varies between devices and models used. there are models that are 19mb… they are unsable as far as i can tell. there are some ~1gb models which are better but understandable slow.

its great that the 1gb model can “show me the bubble-sort in rust”, but i want something “dumber” and smaller. something can can take advantage the data ive already indexed. something that can respond “quickly enough” for a conversational flow. perhaps better i should make a verbose set of reponse templates to perhaps add fancier autocomplete capabilities.

are there any models out there you can suggest for me to investigate and compare?

note: the project is unstable, experiemental and there will be breaking changes. its far from finished. im just trying things out and this approach with LLM’s might just be a waste of time. if you want to test it out, you can try: https://app.glitr.io/ai