The industry keeps benchmarking against other labs instead of against user needs: If a 3.5GB model answers 95% of everyday questions well enough, the remaining few percent has to justify hundreds of gigabytes of weights, huge energy bills and constant cloud costs.

It’ll be “local AI” when they let me point to my own self hosted inference servers, rather than simply OpenAI, Anthropic, or Gemini as providers. Soon to include Apple provider, I guess.

I’ve used the Apple Intelligence ecosystem. The models are slightly acceptable in extremely small context windows, but they completely shit the bed for any kind of practical ad-hoc use. Even if you try to dumb it down to like 6 words. It’s trash. It couldn’t even find a picture of my finger with the keyword “finger.” It couldn’t explain basic details of my phone… it was like interacting with a shittier version of ChatGPTs first release — much shittier.

It told me I have an iPhone 18. I have a 17 Max Pro, the 18 hasn’t been released yet. I don’t plan to ever buy the 18, given the hardware regression with their charging port. But… as I said, it’s not even released yet.

That’s fine with me, actually. If that’s all my phone can handle then so be it. But, when I eventually and obviously will want something more practical, my only options shouldn’t be to pay frontier cloud models if I want deep integration with my phone. The only alternative shouldn’t be to subscribe to a higher tier iCloud+.

I can put a vpn on my phone to access vLLM or ollama locally. Why won’t they let me use that?

I’m pretty skeptical local models can hold a candle to the cloud-based ones, particularly the ones Apple trains.

Cook’s biggest product might be expectation management. He rarely promises tomorrow’s miracle, which buys Apple room to ship when it suits them instead of when Wall Street gets impatient.

Absolutely. Looking forward to seeing the next generation of Macs.

The decentralised operation of LLMs would also be significantly simpler and cheaper for the use of decentralised renewable energy sources.

At least gpus will finally cease their breakneck price climb, I say, for the 800th time over the last two decades.

It would seem so. On the other hand, it is puzzling that they did not also allocate the necessary resources to the development of LLMs. 🤷

Still nobody build a unified memory model with HBM. I would buy Apple hardware, then. If I can put Linux on it.

Perhaps I can buy a used Instict with generics science acceleration, once the smoke clears. And add a couple solar kWp to my capacity.

Stock prices aren’t proof of being right, but they do show investors can change their minds a lot faster than the narratives do.

Just a note that PCC is different from the ChatGPT integration. You might be using PCC resources without noticing, because the iPad won’t tell you in advance. Only way to be sure is to disconnect from the Internet and/or check your Apple Intelligence Report (in System Settings > Privacy) after the fact

I’m waiting for the next generation of Mac Mini and Mac Studio.

Apple’s marketing deserves skepticism, but the technical argument is separate. Local inference does not eliminate giant training clusters, it mainly cuts inference costs, latency and improves privacy. Apple still uses cloud models when needed.