211
The circle of life (lemmy.ml)
submitted 5 months ago by to c/programmerhumor@lemmy.ml
10 comments fedilink hide all child comments
 
you are viewing a single comment's thread
view the rest of the comments
[–] 5 points 5 months ago (3 children)

One jas to Knox how expensive LLMs are by trying to run one at home. Tried running it myself, figured out I need more than 128GiB of RAM and thousand dollars in grahics card. Figured out a $5 openai per month is cheaper, and also understood they are burning money by providing a free service and 20$ subscription.

  • permalink
  • fedilink
  • source
  • hideshow 3 child comments
    •
  • [–] [S] 7 points 5 months ago (2 children)

    You should be able to get very decent performance with 128gb vram running Qwen 3.6 with something like https://github.com/itigges22/ATLAS especially if you run MTP https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF

    A friend of mine gets something like 50 tokens a second with it, and output quality is quite decent.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 5 months ago (1 child)

    How does it compare to largest deepseek ans Claude opus 4.6? I hot used to blazing fast speed and accurate results. I'm not buying a server and 128 GB of RAM just to run a model similar to gpt-4.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] [S] 2 points 5 months ago

    ATLAS has some benchmarks in the repo, and it's comparable to opus 4.6, you don't actually even need 128gb model for that. An 8 bit quantized model will run with around 32gb and still perform quite well.

  • permalink
  • fedilink
  • source
  • parent