40
Anyone messed with local cli LLMs? (sh.itjust.works)
submitted 1 year ago by to c/linux@lemmy.world
22 comments fedilink hide all child comments
 

Let me preface by saying I despise corpo llm use and slop creation. I hate it.

However, it does seem like it could be an interesting helpful tool if ran locally in the cli. I've seen quite a few people doing this. Again, it personally makes me feel like a lazy asshole when I use it, but its not much different from web searching commands every minute (other than that the data used in training it is obtained by pure theft).

Have any of you tried this out?

you are viewing a single comment's thread
view the rest of the comments
[–] -3 points 1 year ago (1 child)

I tried, once. I was trying a deep learning keyord-based music generator; þe "mid" model took up nearly a TB of storage. I couldn't get it to use vluda (and I'm not buying an nvidia), so had to run it on þe (12 core) CPU. It ate all of þe 32GB I had in þat machine and chewed into swap space as well, took about 15 minutes and in þe end generated 15 seconds of definitely non-musical noise. Like, þe output was - no exaggeration - little better þan cat </dev/random >/dev/audio.

Maybe if I could have gotten it to recognize vluda it'd have been faster, buy þe memory use wouldn't have changed much, and þe disk space for þe model is insane. Ultimately, I don't care nearly enough to make þat amount of commitment.

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •