37
What are your LocalLLaMA "hot takes"? (lemdro.id)
submitted 1 year ago by to c/localllama@sh.itjust.works
31 comments fedilink hide all child comments
 

Or something that goes against the general opinions of the community? Vibes are the only benchmark that counts after all.

I tend to agree with the flow on most things but my thoughts that I'd consider going against the grain:

  • QwQ was think-slop and was never that good
  • Qwen3-32B is still SOTA for 32GB and under. I cannot get anything to reliably beat it despite shiny benchmarks
  • Deepseek is still open-weight SotA. I've really tried Kimi, GLM, and Qwen3's larger variants but asking Deepseek still feels like asking the adult in the room. Caveat is GLM codes better
  • (proprietary bonus): Grok 4 handles news data better than GPT-5 or Gemini 2.5 and will always win if you ask it about something that happened that day.
you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 1 year ago* (3 children)

Agreed.

Those models could be easily trained with renewables alone but you know, capitalism.

It's really sad to read the articles how they're planning to bulldoze Texas and do fracking and all these massively invasive things and then we also run a lot of the compute on coal and want more nuclear plants as well. That doesn't really sound that progressive and sophisticated to me.

The thing is, those models are already out there and the people training them do not gain anything when people download their open weights/open source models for free for local use.

You're right. Though the argument doesn't translate into anything absolute. I can't buy salami in the supermarket and justify it by saying the cow is dead anyways and someone already sliced it up. It's down to demand and that's really complex. Does Mark Zuckerberg really gift an open-weights model to me out of pure altruism? Is it ethical if I get some profit out of some waste, or by-product of some AI war/competition? It is certainly correct that we here don't invest money in that form. However that's not the entire story either, we still buy the graphics cards from Nvidia and we also set free some CO2 when doing inference, even if we didn't pay for the training process. And they spend some extra compute to prepare those public models, so it's not no extra footprint, but it's comparatively small.

I'm not perfect, though. I'll still eat salami from time to time. And I'll also use my computer for things I like. Sometimes it serves a purpose and then it's justified. Sometimes I'll also do it for fun. And that in itself isn't something that makes it wrong.

I’m a huge fan of RAG because it cites where it got the information from

Yeah, that's really great and very welcome. Though I think it still needs some improvement on picking sources. If I use some research mode from one of the big AI services, it'll randomly google things, but some weird blog post or a wrong reddit comment will show up on the same level as a reputable source. So it's not really fit for those use-cases. It's awesome to sift through documentation, though. Or a company's knowledgebase. And I think those are the real use-cases for RAG.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 1 year ago (1 child)

    I can’t buy salami in the supermarket and justify it by saying the cow is dead anyways

    That’s not comparable. You can’t compare software or even research with a physical object like that. You need a dead cow for salami, if demand increases they have to kill more cows. For these models the training already happened, how many people use it does not matter. It could influence whether or how much they train new models, but there is no direct relation. You can use that forever in it’s current state without any further training being necessary. I’d rather compare that with nazi experiments on human beings. Their human guinea pigs already suffered/died no matter whether you use the research derived from that or not. Doing new and proper training/research to get to a point where improper ones already got is somewhat pointless in this case, you just spend more resources.

    Though it makes sense to train new models on public domain and cc0 materials if you want end results that protect you better from getting sued because of copyright violations. There are platforms who banned AI generated graphics because of that.

    we still buy the graphics cards from Nvidia and we also set free some CO2 when doing inference

    But you don’t have to. I can run small models on my NITRO+ RX 580 with 8 GB VRAM, which I bought 7 years ago. It’s maybe not the best experience, but it certainly "works". Last time our house used external electricity was 34h ago.

    Regarding RAG, I just hope it improves machine readability, which is also useful for non-AI applications. It just increases the pressure.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 1 year ago* (last edited 1 year ago)

    That’s not comparable. You can’t compare software or even research with a physical object like that. You need a dead cow for salami, if demand increases they have to kill more cows. For these models the training already happened, how many people use it does not matter.

    I really like to disagree here. Sure today's cow is already dead and turned into sausage. But the pack of salami I buy this week is going to make the supermarket order another pack next week so what I'm really doing is have someone kill the next cow, or at least a tiny bit because I'm having just some slices and it's the bigger picture and how I'm part of a large group of people creating the overall demand.

    And I think it's at least quesionable if and how this translates. It's still part of generating demand for AI. Sure, it's kind of a byproduct but Meta directly invests additional research, alignment and preparation for these byproducts. And we got an entire ecosystem around it with Huggingface, CivitAI etc which cater to us, sometimes a sunstantial amount of their bussiness is the broader AI community and not just researchers. They provide us with datacenters for storage, bandwith and sometimes compute. So it's certainly not nothing which gets added due to us. And despite it being immaterial, it has a proper effect on the world. It's going to direct technology and society in some direction. Have real-world consequences when used. The pollution during the process of creating this non-physical product is real. And Meta seems to pay attention. At least that's what I got from everything that happened with LLaMA 1 to today. I think if and how we use it is going to affect what they do with the next iteration. Similar to the salami pack analogy. Of course it's a crude image. And we don't really know what would happen if we did things differently. Maybe it'd be the same so it's down to the more philosophical question of whether it's ethical to benefit from things that have been made in an unethical way. Though this requires today's use not to have any effect on future demand. Like the nazi example where me using medicine is not going to bring back nazi experiments in the future. And that's not exactly the situation of AI. They're still there and actively working on the next iteration. So the logic is more complicated than that.

    And I'm a bit wary because I have no clue about the true motive behind why Meta gifts us these things. It costs them money and they hand control to us, which isn't exactly how large companies operate. My hunch is it's mainly the usual war, they're showing off and they accept cutting into their own business when it does more damage to OpenAI. And the Chinese are battling the USA... And we're somewhere in the middle of it. Maybe we pick up the crumbs. Maybe we're chess pieces and being used/exploited in some bigger corporate battles. And I don't think we're emancipated with AI, we don't own the compute necessary to properly shape it, so we might be closer to the chess pieces. I don't want to start any conspiracy theory but I think these dynamics are part of the picture. I (personally) don't think it's a general and easy answer to the question if it's ethical to use these models. And reality is a bit messy.

    But you don’t have to. I can run small models on my NITRO+ RX 580 with 8 GB VRAM, which I bought 7 years ago. It’s maybe not the best experience, but it certainly “works”. Last time our house used external electricity was 34h ago.

    I think this is the common difference between theory and practice. What you do is commendable. In reality though, AI is in fact mostly made from coal and natural gas. And China and the US ramp up dirty fossil fuel electricity for AI. There's a hype in small nuclear reactors to satisfy the urgend demand for more electricity and they're a bit problematic with all the nuclear waste due to how nuclear power plants scale. So yes, I think we could do better. And we should. But that's kind of a theoretical point unless we actually do it.

    it makes sense to train new models on public domain and cc0 materials

    Yes, I'd like to see this as well. I suppose it's a long way from pirating books because they're exempt from law with enough money and lawyers... to a proper consensual use.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 2 points 1 year ago

    I can’t buy salami in the supermarket and justify it by saying the cow is dead anyways and someone already sliced it up. It’s down to demand and that’s really complex.

    You pay for the salami and thus entice them to make more. There is monetary value for them in making more salami.

    Does Mark Zuckerberg really gift an open-weights model to me out of pure altruism?

    I don't really know why they initially released their models but at least they kicked off a pissing contest in the open weight space on who can create the best open model.

    Meta has not released anything worthwhile in quite a while. It's pretty much Chinese models flexing on American models nowadays.

    Still, their main incentive to train those models lies with businesses subscribing to their paid plans.

    However that’s not the entire story either, we still buy the graphics cards from Nvidia and we also set free some CO2 when doing inference, even if we didn’t pay for the training process.

    True, I exclusively run inference on AMD hardware (I recently got a Strix Halo board) so at least I feel a little bit less bad and my inference runs almost purely on solar power. I expect that is not the norm in the local AI community though.

    If I use some research mode from one of the big AI services, it’ll randomly google things, but some weird blog post or a wrong reddit comment will show up on the same level as a reputable source.

    I rarely use the commercial AI services but also locally hosted the web search feature is not really that great.

    It’s awesome to sift through documentation, though. Or a company’s knowledgebase. And I think those are the real use-cases for RAG.

    Yes, I prefer to use RAG with information I provide. For example, ask a question about Godot and provide it the full Godot 4 documentation with it.

    Still working on getting this automated though. I would love to have a RAG knowledge base of Wikipedia, Stackoverflow, C documentation, etc. that you can query an LLM against.

  • permalink
  • fedilink
  • source
  • parent