551
Sam Altman: "Local inference is a terrible allocation of scarce resources" (twitee.co)
submitted 1 week ago by to c/localllama@sh.itjust.works
171 comments fedilink hide all child comments
 

Sam Altman:

A lot of people want local inference, and somehow they assume that the problem is solved now because we have open-source models, we're going to be able to run them on our computers, and so we're not going to need all the stuff in the data center.

But from a global allocation perspective, actually it's quite terrible to have everybody, like, have a bunch of the world's limited supply of DRAM sitting in their own computer and being used five percent of the time.

Like you want maximum utilisation in the cloud if you're trying to deal with this very supply-constrained environment,

And somehow many people's intuition is the opposite of that

TL;DR Guy selling product: "Here's why owning your own product instead of renting it from me is a terrible idea"

you are viewing a single comment's thread
view the rest of the comments
[–] 69 points 1 week ago (5 children)

Ram is so cheap to make that until the AI demand companies basically had to create new levels of density/speed and sell for less just to keep the business afloat. This is like when cellphone companies tried to claim megabytes were a precious finite resource. We live in a post scarcity society and these billion assholes are like “well that just won’t do”.

It reminds me of that kid we all know who just learned economics from school house rock and then goes “now if I can simply find every evee Pokémon card in the world and burn them all except for the one I have, I’ll be rich!”

It’s just so frustrating to see this sheer stupid amount of greed all of the time. Thanks for coming to my TEDex talk.

  • permalink
  • fedilink
  • source
  • hideshow 5 child comments
    •
  • [–] -1 points 1 week ago* (last edited 1 week ago) (2 children)

    like when cellphone companies tried to claim megabytes were a precious finite resource

    Limiting the usage of a shared channel is a valid approach, it's remarkable how so many people just absolutely refuse to understand this. You can use the entire bandwidth of the shared channel for a while, when your webpage is loading. If you use the entire bandwidth nonstop, you fuck over everyone else on the channel. So the company has the choice between giving you fast load times but limiting your usage otherwise, or giving you 32 kbps unfettered.

    If you want an entire channel of guaranteed bandwidth with unlimited usage, you always could and can get it — under business plans, which are orders of magnitude costlier, because they don't share the channels.

    In a similar fashion, if your healthcare system is entirely free, it's getting clogged with retired old people who are lonely and have nothing better to do. If a doctor visit costs even a couple bucks (depending on the region), the doctors are suddenly much more free to deal with real cases.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 1 week ago (1 child)

    Right… In this instance I would argue openAI is the shared channel. There are three memory manufacturers now illegally rigging the market as a push back against the corporate AI. Something like a memory oligopoly VS ai monopsony?

    If we all have inference, we avoid clogging shared channel.

    And as for megabytes, the value of the megabyte isn’t a finite resource and the value of it is determined by what data is served in the megabyte and the cell companies didn’t care what megabytes you’re getting charged for (example full bitrate advertisements on YouTube). I still think that’s a valid argument to be had.

    By the value based on scarcity of bandwidth logic, the phone companies would be able to charge peak megabyte hours to specific websites (again like they used to for telephone calls with plans that included free calls at night) when in fact they can handle much more bandwidth now. In reality when it comes to that traffic, it’s often the providers at the ends of the line who have to suffer the constraints of demand. Maybe they should charge per minute of video accessed or it could be a solved problem in a hyper-competitive, over-saturated marketplace.

    I understand what you are saying. I just don’t believe it applies in this specific domain any longer. This is potentially a more valid take an underserved market.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 1 week ago

    the phone companies would be able to charge peak megabyte hours to specific websites

    ISPs in the US were aiming to do that, since video traffic dominates the web pretty heavily now. But that could be easily abused, so they were slapped away with the net neutrality regulations, which iirc Trump rolled back.

    I just don’t believe it applies in this specific domain any longer.

    You referred to the megabytes thing in the past tense, yes, but the reasoning applied back then. In fact, gigabit speeds are still not prevalent worldwide (nor just in the US, from what I understand about their shitty ISPs). I can easily saturate the 100 mbps allotted to my cable-connection plan in the neighbourhood that doesn't provide much choice, and I'm pretty sure mobile connection isn't better than that in most cases.

  • permalink
  • fedilink
  • source
  • parent