submitted 1 week ago by to c/privacy@lemmy.ml
๐Ÿ™‡ 1
 

I've cancelled my subscription yesterday, conveniently it was about to renew at the end of the week, so I have a few days to migrate from Proton, here's what I'm thinking of, but I'd love to hear what other people are switching to

you are viewing a single comment's thread
view the rest of the comments
[โ€“] 3 points 1 week ago* (6 children)

I have a lower-end GPU (8GB VRAM) and I can still run smaller models on it with decent performance.

  • source
  • parent
  • hideshow 6 child comments
  • [โ€“] 7 points 1 week ago (1 child)

    Dumb question: what do people use those local LLMs even for?

  • source
  • parent
  • hideshow 1 child comment
  • [โ€“] 1 point 1 week ago

    Not much, but you can use it for basically anything people use frontier models for, it'll just be somewhat dumber. I occasionally run LLMs for proof-reading important things that I've already hand-written and edited. Sometimes for bouncing off ideas. Sometimes for the kinds of questions I'd normally chuck at a forum but I can get a quicker answer if I ask an LLM. No, I don't "trust" the answers; anything where I can't just use my own judgement, the LLM will have given me enough to perform my own research to fact-check it.

  • source
  • parent
  • [โ€“] 1 point 1 week ago (3 children)

    What does "lower-end" mean in this case? 8GB VRAM could be anything from a 2014 Radeon R9 290X to a 2026 RTX 5060.

  • source
  • parent
  • hideshow 3 child comments
  • [โ€“] 1 point 1 week ago (2 children)

    Radeon RX 7600, but yes, you're right, it's relative. It's good enough for any gaming I do, which is not an awful lot. I think it's considered mid-range for its generation, I guess I was just thinking of the GPUs I see people mention they have which seem much beefier than mine lol.

    I'm no AI expert but when I looked into it, it seemed that VRAM was the biggest factor being mentioned in discussions of GPUs, hence why I mentioned it.

  • source
  • parent
  • hideshow 2 child comments
  • [โ€“] 1 point 1 week ago (1 child)

    Iโ€™m no AI expert but when I looked into it, it seemed that VRAM was the biggest factor being mentioned in discussions of GPUs, hence why I mentioned it.

    I could be wrong, but I think there are differences not just in FLOPS but perhaps more importantly in what kinds of functions are supported (with newer GPUs becoming gradually more general-purpose), so I'm not sure if the GPU from 2014 is capable of running AI models at all (slowly or otherwise).

    I actually kinda want to find out, because my old GPU is a Vega 56 (which is also 8GB) and I'm not sure if it would be useful or not.

  • source
  • parent
  • hideshow 1 child comment
  • [โ€“] 1 point 6 days ago

    I'm sure you're probably right. I haven't looked into how things like clock speed/FLOPS/etc affect LLM performance.

    The main thing is you need ROCm/CUDA/Vulkan support. If you do then just try running some models that fit into your VRAM and see how they perform? On my card, it takes maybe a minute or two to generate an answer, sometimes several minutes if the answer is long. I use small models that are <8GB in size, so not the best open-weight ones, but I don't use LLMs for anything particularly complex anyway so the dumb ones do the trick for me.

  • source
  • parent