I think it's a bit early to be locking down your local LLM setup. The history of computing suggests there is some cost reduction to come for the hardware. In the mean time there is time to find the sweet spot for performance of the small and medium models. There are plenty of cloud hosts which can run the open models and allow you to experiment while the models mature. Hopefully your not burning 6k worth of tokens anytime soon.

I've been running Qwen3.6 35B very easily. but that's because I've got a ASUS Z13, which is one of the newish laptops that have the AMD Ryzen AI MAX+ 395 in it. you can get that chip with 128gb of unified memory, 96gb of which I have dedicated to be VRAM. I can also run Qwen3 Coder Next 80B. I'm not sure how many tokens per second I'm getting with Coder, but it's fast.

honestly I think this unified memory might be the future of mobile chips, because the things I can do with it are pretty crazy. it's not just useful for AI either, it's in a few gaming laptops because it also works really well when gaming. but the things you can do with LLMs or diffusion models is amazing. I donate compute to AI Horde, and I'm finishing image generation jobs for people in like 4 seconds.

I think this is where a lot of LLMs will land, in local usage. At first people will try to use big general models for their rather domain-specific tasks until they realize smaller specialized models can do the same thing, but cheaper, with no subscription or per-token costs and with full ownership of the model, not just renting a model.

Imagine having a model that is proprietary to your company, cost nothing but the cost of electricity to run, no renting computing capacity etc. You own the model and can customise it at will and have no restrictions.

Currently super large models are driven by a hope for AGI capabilities, but we are not there yet for years and LLMs will never get us there. It requires different architectures.

I run ministral 8B on my laptop and as long as I don't ask it too complex tasks it can do things like translating, explaining simple things or help me understand functions and basic code while I am learning. If it can web search, RAG search and index you don't need that big of a model for it to work as a decent assistant.

If you want to replace people and not just enhance their work, then at current prices i don't think you get your bang for the bucks. A model owned and run by someone else outside the company is a just a very expensive consultant... They'll leave with all their competence if you stop paying them. An in-house AI will never leave and only get better with time and you don't have to pay anything other than electricity and the initial capital cost for the server.

I give it a year or two before companies realise this, but first they need to realise that the money they spend on subscriptions and tokens aren't investments, but rather costs. They don't get the money back. With a local model, it is an investment that keeps on giving after initial capital investment and cost much less for repetitive work.

Mistral is betting on this and I think it will pay off. Unless I am wrong about AGI.

I don't think that's any new insight 😂. That's how the AI game works. There's always been two classes: Big corpo. And the GPU poor. Of course the big AI companies get to shape AI. Economy of scale also works in their favour. They've bought most of the skill. And they have all the money. They simply buy a 4x EPYC +3TB RAM connected to 16 Nvidia AI cards. And then a few hundred nodes more. You don't even buy one. It's just a very unequal environment if you want to compare the two.

You know its funny because I kinda hate when people bring reddit stuff here but I love when people actually communicate rather than just dropping links or images. So overall I like this post because your not just pushing reddit in my face your just talking about your experience there. I kinda hope that local llm kinda morph into operating system agents that are experts in the operating system where its a bit like the next run level. so like run level 3 being online terminal and 5 being graphical and this would ideally become more like the computers in star trek. I figure its programmed to answer operating system questions initially and it can be given read permission and like to run programs for you. maybe permission to browse the web and get results. app type extensions or such. Of course I could not trust it unless its gpl and community based and completely under my control to configure.