551
Sam Altman: "Local inference is a terrible allocation of scarce resources" (twitee.co)
submitted 1 week ago by to c/localllama@sh.itjust.works
171 comments fedilink hide all child comments
 

Sam Altman:

A lot of people want local inference, and somehow they assume that the problem is solved now because we have open-source models, we're going to be able to run them on our computers, and so we're not going to need all the stuff in the data center.

But from a global allocation perspective, actually it's quite terrible to have everybody, like, have a bunch of the world's limited supply of DRAM sitting in their own computer and being used five percent of the time.

Like you want maximum utilisation in the cloud if you're trying to deal with this very supply-constrained environment,

And somehow many people's intuition is the opposite of that

TL;DR Guy selling product: "Here's why owning your own product instead of renting it from me is a terrible idea"

you are viewing a single comment's thread
view the rest of the comments
[–] -1 points 1 week ago* (last edited 1 week ago) (2 children)

like when cellphone companies tried to claim megabytes were a precious finite resource

Limiting the usage of a shared channel is a valid approach, it's remarkable how so many people just absolutely refuse to understand this. You can use the entire bandwidth of the shared channel for a while, when your webpage is loading. If you use the entire bandwidth nonstop, you fuck over everyone else on the channel. So the company has the choice between giving you fast load times but limiting your usage otherwise, or giving you 32 kbps unfettered.

If you want an entire channel of guaranteed bandwidth with unlimited usage, you always could and can get it — under business plans, which are orders of magnitude costlier, because they don't share the channels.

In a similar fashion, if your healthcare system is entirely free, it's getting clogged with retired old people who are lonely and have nothing better to do. If a doctor visit costs even a couple bucks (depending on the region), the doctors are suddenly much more free to deal with real cases.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 1 week ago (1 child)

    Right… In this instance I would argue openAI is the shared channel. There are three memory manufacturers now illegally rigging the market as a push back against the corporate AI. Something like a memory oligopoly VS ai monopsony?

    If we all have inference, we avoid clogging shared channel.

    And as for megabytes, the value of the megabyte isn’t a finite resource and the value of it is determined by what data is served in the megabyte and the cell companies didn’t care what megabytes you’re getting charged for (example full bitrate advertisements on YouTube). I still think that’s a valid argument to be had.

    By the value based on scarcity of bandwidth logic, the phone companies would be able to charge peak megabyte hours to specific websites (again like they used to for telephone calls with plans that included free calls at night) when in fact they can handle much more bandwidth now. In reality when it comes to that traffic, it’s often the providers at the ends of the line who have to suffer the constraints of demand. Maybe they should charge per minute of video accessed or it could be a solved problem in a hyper-competitive, over-saturated marketplace.

    I understand what you are saying. I just don’t believe it applies in this specific domain any longer. This is potentially a more valid take an underserved market.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 6 days ago

    the phone companies would be able to charge peak megabyte hours to specific websites

    ISPs in the US were aiming to do that, since video traffic dominates the web pretty heavily now. But that could be easily abused, so they were slapped away with the net neutrality regulations, which iirc Trump rolled back.

    I just don’t believe it applies in this specific domain any longer.

    You referred to the megabytes thing in the past tense, yes, but the reasoning applied back then. In fact, gigabit speeds are still not prevalent worldwide (nor just in the US, from what I understand about their shitty ISPs). I can easily saturate the 100 mbps allotted to my cable-connection plan in the neighbourhood that doesn't provide much choice, and I'm pretty sure mobile connection isn't better than that in most cases.

  • permalink
  • fedilink
  • source
  • parent