99
I gave Qwen 3.8 27B a reverse-engineering job I assumed needed a frontier model, and it finished in 30 minutes (www.xda-developers.com)
submitted 2 months ago by to c/technology@lemmy.world
32 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 48 points 2 months ago (10 children)

I agree, but this will help pop the bubble because local models we can run on our own hardware will crash the demand for data centers

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 10 child comments
    •
  • [–] 3 points 2 months ago (4 children)

    I don't think you're thinking this through. You can have 1k people running 1 server or 1k people each with their own hardware running models. Which one of those do you think will increase hardware demand?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 1 point 2 months ago (3 children)

    Local models tend to be more efficient since people will be more likely to run compressed and MoE models.

    Also, it's basically 1 GPU >= 1 request for the most part in data centres, each request is its own LLM. Each time you make a new request after a set timeout, model weights get loaded in VRAM, context gets initialized, the query gets parsed, and it spits out tokens. These frontier models can be 300 GB in size or more, which all needs to be kept in VRAM for best performance, usually distributed across multiple GPUs, and each loaded model can only answer one query at a time.

    Compare this to someone like me, trying to cram Qwen 3.6 MoE on the 16 GB RX 6800XT I already have in my own computer, not using up drinking water or prime real estate to cool my PC, powered using hydroelectricity.

    It's so much more eco-friendly and economical, I wish frontier models would just die tbh, or at least only be used for distillation. LLMs offer diminishing returns past a certain point, and you can get 90% of the frontier model with a MoE local model.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 4 points 2 months ago (1 child)

    Where do you think those models are trained?

    And most personal computing has tended towards only having a smartphone, and a laptop for a real PC. You can't run much in the way of an LLM on those (at least not with any real speed).

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [+] -8 points 2 months ago* (last edited 2 months ago) (2 children)

    Not that I have anything against local use---I do my stuff locally---but people running local models will very probably greatly increase demand for memory relative to cloud-based AI compute (and it's why I think that in the near term, most AI compute stuff is gonna be in the cloud, because we don't have the memory to do everything locally).

    If you run in the cloud, the hardware is shared. When one person isn't using it, another can be. That can lead to high rates of capacity utilization, approaching 100%.

    If I have local AI compute hardware, if I'm not using it, it's idle. If 1% of the time, I have it crunching something for me, then I'm only getting 1% capacity utilization of that hardware. That means that to provide the same level of compute capability to everyone for local use, I need 100 times as much hardware.

    AI cloud companies have purchased more memory than the rest of the world is buying.

    If we did everything locally, we'd need a hundred times more memory than what the cloud AI companies are picking up.

    We couldn't do that without far more memory production capacity. Even if someone started on a buildout of that scale today, a new memory factory takes 4--5 years to get into production.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •