I think the issue is also that you need some serious hardware to get good inference speed when your devs are working, but then most of the time this hardware will be under utilized.

That being said you can get good performance from indie inference farms, at a fraction of the cost of the big US labs. I think it's a great compromise and in a few months the open models will be near parity with opus 4.6 which is really all you need for most tasks.

Bigs definitely do, and anyone with confidential data should be.

Because the people selling the AI wants to make sure their customers don't know about this. It's all about causing a dependency so they get subscription income forever.

Because in the feeding frenzy, every company with a product/marketing budget is trying to make the customers pay by the token and companies are doing jack to help "mere mortal" companies get going with this stuff on premise.

You are right that the technical hurdles are not insane to get this going, but most companies don't know where to begin and there's no huge marketing blitz telling the business leaders this is realistically on the table and here's the company you can call to make it happen for you.

Even if you overcame that and proposed really how to get going, you will still probably hit the aversion to capex that has persisted since Amazon told the industry that capex is toxic and you really want all your money to be spent on opex. Big companies like Amazon will take on that scary CapEx for you and you're expenses will be nice and just OpEx. Coincidentally, the companies that spend the most on CapEx manage to pull in more revenue and profit than you will ever dream to, but still, remember CapEx is toxic.

It's not an answer you'd get from Claude — it's real, organic content:

• 👶written by a genuine human
• 💡delivering original ideas and language
• 🚀going above and beyond to answer
• ✨synergizing cross-platform initiatives

(🤪 this is a joke)

Yes, the big proprietary cloud models have an edge, but it is narrow and the open-weight models are constantly closing the gap. There is no moat when it comes to AI models and no company has yet discovered some secret special sauce to improve their model significantly over others.

Running the latest and greatest open-weight GLM, Kimi, or Qwen model is basically equivalent to running the previous latest and greatest version of Claude. So if you were happy with Claude then, you'll basically be happy with an open-weight model now.

Match current Claude is not, but Claude 6-12 months ago should be possible using Open model

Mostly down to frameworks (the bits around the LLM like RAG, memory, prompts, agents etc.) now. The ability to just throw more tokens at the problem is also super important. And you can because you're just paying for electricity (and CapEx for the hardware), not tokens from companies that are doing pre-IPO monetization (i.e. tokens gonna go up, way up). They've been losing money hand over fist to gain market share and pump the idea, that was never going to last.

Hence asking questions

No that's not how this works. Inference is cheap and efficient. AI companies are bankrupting themselves with training costs that they need to recoup back by selling inference. Open-weight models have already been trained.

Also, going big in terms of model size shows diminishing marginal returns on accuracy, not efficiency of scale. Smaller models are way more efficient and consistently catch up to the largest models, which is why today's SOTA 27 billion parameter model competes with yesterday's SOTA 500+ billion parameter model.

There's a big difference between training a model, running a model, and running a model at scale.

A small, self hosted setup will have lower accuracy and queries per second, and it will have a cost, but the cost will be no more than playing a videogame. You'll still have something surprisingly accurate and responsive for some tasks, like being a wiki interface or something.

Remember that some of these models can run on a standard smartphone, and all the hoopla when people found that chrome was downloading models onto people's devices.

a 128GB framework desktop could do that job. it's increased a bit in price since i last looked at it but €4500 isn't that much for a company.