I tried using LocalAI to get some kind of agent not being used to entrench fascism and destroy the environment and consume irreplaceable clean water and haven't had much luck. Even on a beefy machine with a chonky graphics card, it's slow. That was after trying it on my laptop and damn near burning my legs off with the graphics card going nuts and the fans spinning up like jet turbines.

I did an experiment where I tried to get it to write some code that I had already written myself, just as a test case. Even when I found a model that worked more quickly, the code it produced didn't do the thing I had told it to do. I had to tell the LLM that, contrary to its "thinking," inline HTML does, in fact, make for valid Markdown, and that it could actually do the thing I had already explicitly told it to do.

The second iteration worked, but the code was far from ideal. If I had been in a professional situation where i would have been putting that up for review, it would have taken me even longer to make the code presentable and up to my standard. In the end, the code I wrote was better by a mile and I wrote it maybe twice as fast as the LLM.

I'm sure this is faster if I use Claude or something and engage the Torment Nexus, but that's kind of my point. It takes so much juice to make these things viable, and the hardware you need at home to replicate that usefulness is expensive and becoming less available. And removing the energy cost from the user's machine (where you can literally hear and feel the energy consumption) to some bank of servers you'll never see in person does a lot to distance a user from the effects of their usage of the tool. Running a local agent only made my view on AI lower than it already was.

I did not try the Qwen model, though I suppose I ought to since I've now seen you claim success with it a few times on here. Will I finally be impressed? Man, I doubt it, but I'll give it a go.