320
OpenAI Hoarding Tens Of Thousands Of Apple Mac mini And Mac Studio Devices, As ASUS And MSI Burn Through Their Entire First Batch Of NVIDIA RTX Spark Chip And Beg For More. (wccftech.com)
submitted 2 months ago by to c/technology@lemmy.world
78 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 32 points 2 months ago* (18 children)

Local 27b models are good enough for most tasks.

Can’t wait to buy one of these from Ebay for 10% of the price next year.

  • permalink
  • fedilink
  • source
  • hideshow 18 child comments
    •
  • [–] 4 points 2 months ago* (last edited 2 months ago) (9 children)

    Any 27b Model you can currently recommend for a 16gb AMD ? Mostly coding tasks but not exclusively.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 9 child comments
    •
  • [–] 5 points 2 months ago (4 children)

    9060xt 16gb is the most cost effective new GPU, but if you're going used look for a V620 on eBay. It's a 6800xt chip but in server form factor GPU with 32GB vram. Can be a bit of a pain to set up but by far the most cost effective option IMO

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 6 points 2 months ago (2 children)

    V620s were a good deal when you could get them for $350, now they’re $700+ and no longer a good deal.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 5 points 2 months ago* (last edited 2 months ago)

    There just aren't any good deals any more. Prices for everything have gone crazy in the last few months. For coding LLMs the cloud services may now be the least worst value, by design, until they hike the prices.

    That said, I still just paid way too much for a used graphics card so I could do many things locally, because I just don't want to give the likes of Sam Altman a penny.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 2 points 2 months ago* (1 child)

    For your hardware, the VRam is not enough to run 27b but, I’d recommend Qwen 3.5 9b for image / text to text.

    And I’m planning to experiment with Qwen 3.8 9b for text to text.

    4_k_m quantization is the sweet spot for performance and ram usage.

    Also, I find Llama cpp is better than Ollama in terms of performance.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 7 points 2 months ago (3 children)

    https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant

    I'm running gwen 3.6 with 131k context window on a 3090, it's fast enough and about as good as pay to play Claude at work.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •