582
Nobody Will Say Why Every Major AI Chatbot Suddenly Went Down Yesterday (futurism.com)
submitted 2 months ago by to c/technology@lemmy.world
149 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 2 months ago (7 children)

Correct, that's how I would do it, but then I need another machine to act as a head.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 7 child comments
    •
  • [–] 3 points 2 months ago (6 children)

    If your MB has onboard graphics, maybe you could mask the GPU and just pass it off to a container running the LLMs I guess

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] 1 point 2 months ago (5 children)

    No onboard graphics unfortunately, that would have been the easy way out.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 2 points 2 months ago (4 children)

    Well it works anyway even with a bit of occupied vram, but you could also buy a cheap videocard to use as an output. I have small intel card like that in my server for jellyfin transcoding, I think I paid 60€ for it, it hardly uses any power

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 1 point 2 months ago* (last edited 2 months ago) (3 children)

    I actually did exactly that previously! I had both an RX 6800 XT and an RX 6600 in my system and I used the 6600 for video output. Unfortunately, this cuts my RX 6800 XT from PCIe 4 16x to PCIe 4 8x and severely slows down model loading for llama-swap. Joys of the X570!

    And yes, I do have it running right now with a bit of occupied VRAM, but I need to limit my model to 14 GB to leave 2 GB free for GNOME Shell. I really want one of those 64 GB UMA Mac Mini, I heard they work really well because the GPU has direct access to system RAM.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 2 months ago (2 children)

    So I have a framework laptop with ryzen ai cpu that uses 48gb of shared ram, and it does run Q4 llms fine enough, but I'm not sure it compares to a real GPU.

    On my desktop I have an RX 7900 XTX but I've only dabbled in image generation so far, so right now I couldn't really tell you the difference.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 2 months ago (1 child)

    24 GB VRAM. Damn, jealous! My 16 GB seems pitiful in comparison 😅

    I do wonder if the Ryzen AI CPUs compare with Apple's UMA. I'm mostly interested in LLM inference for code generation and automation.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 2 months ago*

    Yeah I was lucky to buy a 6900XT when it was near the lowest price, so I sold that and added a couple hundred for the 7900, seemed like a good future-proofing move, given the times we're living in.

    I think Apple silicon is faster than my generation of Ryzen, but the newest (Strix something?) with the LPDDRGGFASEWARGH5 memory should be faster yet. Of course buying all that memory right now would be quite painful.

  • permalink
  • fedilink
  • source
  • parent