37
What are your LocalLLaMA "hot takes"? (lemdro.id)
submitted 1 year ago by to c/localllama@sh.itjust.works
31 comments fedilink hide all child comments
 

Or something that goes against the general opinions of the community? Vibes are the only benchmark that counts after all.

I tend to agree with the flow on most things but my thoughts that I'd consider going against the grain:

  • QwQ was think-slop and was never that good
  • Qwen3-32B is still SOTA for 32GB and under. I cannot get anything to reliably beat it despite shiny benchmarks
  • Deepseek is still open-weight SotA. I've really tried Kimi, GLM, and Qwen3's larger variants but asking Deepseek still feels like asking the adult in the room. Caveat is GLM codes better
  • (proprietary bonus): Grok 4 handles news data better than GPT-5 or Gemini 2.5 and will always win if you ask it about something that happened that day.
you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 1 year ago (4 children)

I’m flip-flopping between running local models on my PC with solar power vs. using OpenAI’s free ChatGPT to drive them into ruin, which most of the time ends with me having stupid a stupid argument with an AI.

impact on society

Local AI will likely have a long lasting impact as it won’t just go away. The companies who released them can go bankrupt, but the models stay. The hardware which runs them will get faster and cheaper over time.

I have some hope with accessibility and making FLOSS development easier/faster. Generative AI can at least quickly generate mockup code or placeholder graphics/code. There are game projects who would release with generated assets, just like for a long time there were game projects who released assets which were modifications or redistribution of assets they didn’t have the rights for. They are probably less likely to get sued over AI generated stuff. It’s unethical but they can replace it with something self-made once the rest is finished.

Theoretically even every user could generate their own assets locally which would be very inefficient, also ethically questionable, but legally fine as they don’t redistribute them.

I like how Tesseract already uses AI for OCR and Firefox for realtime website translations on your device. Though I dunno how much they benefit from advancements in generative AI?

Though a different point/question: At what point is generative AI ethically and legally fine?

  • If I manage to draw some original style it transfers? But I’m so slow and inefficient with it that I can’t create a large amount of assets that way
  • When I create the input images myself? But in a minimalist and fast manner

It still learned that style transfer somewhere and will close gaps I leave. But I created the style and what the image depicts. At what point is it fine?

Like coding

I actually use it often to generate shell scripts or small simple python tools. But does it make sense? Sometimes it does work. For very simple logic they tend to get it right. Though writing it myself would probably been faster the last time I used, though at the moment I was too lazy to write it myself. I don’t think I’ve ever really created something usable with it aside from practical shell scripts. Even with ChatGPT it can be an absolute waste of time to explain why the code is broken, didn’t get at all why its implementation lead to a doubled file extension and a scoping error in one function … when I fixed them it actually tried to revert that.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 2 points 1 year ago* (last edited 1 year ago) (3 children)

    Your experience with AI coding seems to align with mine. I think it's awesome for generating boilerplate code, placeholders including images, and for quick mockups. Or asking questions about some documentation. The more complicated it gets, the more it fails me. I've measured the time once or twice and I'm fairly sure it's more than usual, though I didn't do any proper scientific study. It was just similar tasks and me running a timer. I believe the more complicated maths and trigonometry I mentioned was me yelling at AI for 90min or 120minutes or so until it was close and then I took the stuff around, deleted the maths part and wrote that myself. Maybe AI is going to become more "intelligent" in the future. I think a lot of people hope that's going to happen. I think as of today we're need to pay close attention if it fools us but is a big time and energy waster, or if it's actually a good fit for a given task.

    Local AI will likely have a long lasting impact as it won’t just go away.

    I like to believe that as well, but I don't think there's any guarantee they'll continue to release new models. Sure, they can't ever take Mistral-Nemo from us. But that's going to be old and obsolete tech in the world of 2030 and dwarfed by any new tech then. So I think the question is more, are they going to continue? And I think we're kind of picking up what the big companies dumped when battling and outcompeting each other. I'd imagine this could change once China and the USA settle their battle. Or multiple competitors can't afford it any more. And they'd all like to become profitable one day. Their motivation is going to change with that as well. Or the AI bubble pops and that's also going to have a dramatic effect. So I'm really not sure if this is going to continue indefinitely. Ultimately, it's all speculation. A lot of things could possibly happen in the future.

    At what point is generative AI ethically and legally fine?

    If that's a question about development of AI in general, it's an entire can of worms. And I suppose also difficult to answer for your or my individual use. What part of the overall environment footprint gets attributed to a single user? Even more difficult to answer with local models. Do the copyright violations the companies did translate to the product and then to the user? Then what impact do you have on society as a single person using AI for something? Does what you achieve with it outweigh all the cost?

    Firefox for realtime website translations

    Yes, I think that and text to speech and speech to text are massively underrated. Firefox Translate is something I use quite often and I can do crazy stuff with it like casually browse Japanese websites.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 1 year ago (2 children)

    But that’s going to be old and obsolete tech in the world of 2030 and dwarfed by any new tech then.

    My point was more the people they replace now they’ll replace indefinitely in the context of "impact on society"

    a question about development of AI in general, it’s an entire can of worms

    and

    So I think the question is more, are they going to continue?

    I just ran into https://huggingface.co/briaai/FIBO, which looks interesting in many ways. (At first glance.)

    trained exclusively on licensed data

    It also only works with JSON inputs. The more we split AIs into modules that can be exchanged, the more we can update pipelines module by module, tweak them…

    It’s unlikely that there’ll never be new releases. It’s always interesting for new-comers to gain market penetration and show off.

    What part of the overall environment footprint gets attributed to a single user?

    It’s possible that there’ll be companies at some point who proudly train their models with renewable energy etc. like it’s already common in other products. It just has to be cheap/accessible enough for them to do that. Though I don’t see that for GPU production anytime soon.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 1 year ago* (1 child)

    Thanks.

    FIBO, which looks interesting in many ways.

    Indeed. Seems it has good performance, licensed training material... That's all looking great. I wonder who has to come up with the JSON but I guess that'd be another AI and not my task. Guess I'll put it on my list of things to try.

    It’s possible that there’ll be companies at some point who proudly train their models with renewable energy

    I said it in another comment, I think that's a bit hypothetical. It's possible. I think we should do it. But in reality we ramp up natural gas and coal. US companies hype small nuclear reactors and some some people voiced concerns China might want to take advantage of Russia's situation for their insatiable demand for (fossil-fuel) energy. I mean they also invest massively in solar. It just looks to me we're currently overall headed the other direction and we need substantial change to maybe change that some time in the future. So I categorize it more towards wishful-thinking.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 1 year ago* (last edited 1 year ago)

    I wonder who has to come up with the

    I haven’t tried to run any of that yet, but they have these models on HF:

    that’s a bit hypothetical

    Yes, absolutely. It can happen, but we shouldn’t make decisions based on the assumption that it might happen. In other fields there are companies who try to make their products better recyclable, less energy hungry (production and run time), made from sustainable resources, repairable, more ethically sourced resources etc. So it’s not out of question, but it often starts with people who just wanna see it happen, not with a business case. There are also many black sheep who only do green washing by just letting it sound like they do that without actually doing it.

    Ecosia already tries to sell their chatbot as green, but it only uses OpenAI’s API and they plant trees how they always do. Though I generally don’t like their compensation concept, at least they claim their own servers run 100% renewable energy. I haven’t tried their chatbot(s) yet, but it looks like it’s still only OpenAI. If they do it like duckduckgo at some point in the future, they could run open models on their own servers. Whether they can produce enough energy and get their hands on hardware to get that working etc is a different question though. There isn’t any indication yet that they plan to go that way.

    It’s probably already possible to let an EMS start AI training when there is solar overproduction. That’s only worth it when the pace of new break throughs have slowed down or when they use outdated techniques anyways. I dunno where the current balance currently is between electricity prices, hardware cost, energy efficiency of the hardware and time pressure.

    EDIT: Sounds like Ecosia is on it for runnning AIs at least https://blog.ecosia.org/what-we-are-doing-with-ai/. They probably push that renewable energy into grid somewhere else than where the AI is consuming it.

    concerns China might want to take advantage

    I don’t think they’ll say no to cheap energy, but they definitely don’t wanna be dependent on other countries for their energy. As far as I understand they push solar, electric cars etc for energy dependency reasons.

  • permalink
  • fedilink
  • source
  • parent