1009
Elon Musk’s Grok Goes Haywire, Boasts About Billionaire’s Pee-Drinking Skills and ‘Blowjob Prowess’ (www.mediaite.com)
submitted 11 months ago by to c/technology@lemmy.world
153 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 11 months ago (12 children)

Do you have a cluster with 10 A100 lying around? Because that's what it gets to run deepseek. It is open source, but it is far from accessible to run on your own hardware.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 12 child comments
    •
  • [–] 1 point 11 months ago*

    I run quantized versions on deepseek that are usable enough for chat, and it's on a home set that is so old and slow by today's standards I won't even mention the specs lol. Let's just say the rig is from 2018 and it wasn't near the best even back then.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 11 months ago (9 children)

    Yes, that's true. It is resource-intensive, but unlike other capable LLMs, it is somewhat possible—not for most private individuals due to the requirements, but for companies with the necessary budget.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 9 child comments
    •
  • [–] 5 points 11 months ago (8 children)

    They're overestimating the costs. 4x H100 and 512GB DDR4 will run the full DeepSeek-R1 model, that's about $100k of GPU and $7k of RAM. It's not something you're going to have in your homelab (for a few years at least) but it's well within the budget of a hobbyist group or moderately sized local business.

    Since it's an open weights model, people have created quantized versions of the model. The resulting models can have much less parameters and that makes their RAM requirements a lot lower.

    You can run quantized versions of DeepSeek-R1 locally. I'm running deepseek-r1-0528-qwen3-8b on a machine with an NVIDIA 3080 12GB and 64GB RAM. Unless you pay for an AI service and are using their flagship models, it's pretty indistinguishable from the full model.

    If you're coding or doing other tasks that push AI it'll stumble more often, but for a 'ChatGPT' style interaction you couldn't tell the difference between it and ChatGPT.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 8 child comments
    •
  • [–] 1 point 11 months ago* (7 children)

    You should be running hybrid inference of GLM Air with a setup like that. Qwen 8B is kinda obsolete.

    I dunno what kind of speeds you absolutely need, but I bet you could get at least 12 tokens/s.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 7 child comments
    •
  • [–] 1 point 11 months ago (6 children)

    Thanks for the recommendation, I'll look into GLM Air, I haven't looked into the current state of the art for self-hosting in a while.

    I just use this model to translate natural language into JSON commands for my home automation system. I probably don't need a reasoning model, but it doesn't need to be super quick. A typical query uses very few tokens (like 3-4 keys in JSON).

    The next project will be some kind of agent. A 'go and Google this and summarize the results' agent at first. I haven't messed around much with MCP Servers or Agents (other than for coding). The image models I'm using are probably pretty dated too, they're all variants of SDXL and I stopped messing with ComfyUI before video generation was possible locally, so I gotta grab another few hundred GB of models.

    It's a lot to keep up with.😮‍💨

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] 2 points 11 months ago (5 children)

    It’s a lot to keep up with

    Massive understatement!

    The next project will be some kind of agent. A ‘go and Google this and summarize the results’

    Yeah, you do want more contextual intelligence than an 8B for this.

    The image models I’m using are probably pretty dated too

    Actually SDXL is still used a lot! Especially for the anime stuff. It just got so much finetuning and tooling piled on.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 1 point 11 months ago (4 children)

    Yeah, you do want more contextual intelligence than an 8B for this.

    Oh yeah, I'm sure. I may peek at it this weekend. I'm trying to decide if Santa is going to bring me a new graphics card, so I need to see what the price:performance curve looks like.

    Massive understatement!

    I think I stopped actively using image generation a little bit after LoRAs and IP Adapters were invented. I was trying to edit a video (random meme gif) to change the people in the meme to have the faces of my family, but it was very hard to have consistency between frames. Since there is generated video, it seems like someone solved this problem.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 2 points 11 months ago* (3 children)

    Since there is generated video, it seems like someone solved this problem.

    Oh yes, it has come a LOONG way. Some projects to look at are:

    https://github.com/ModelTC/LightX2V

    https://github.com/deepbeepmeep/Wan2GP

    And for images: https://github.com/nunchaku-tech/nunchaku

    Video generation/editing is very GPU heavy though.

    I dunno what card you have now, but with text LLMs (or image+text input LLMs), hybrid CPU+GPU inference is the trend days.

    As an example, I can run GLM 4.6, a 350B LLM, with measurably low quantization distortion on a 3090 + 128GB CPU RAM, at like 7 tokens/s. If you would’ve told me that 2-4 years ago, my head would have exploded.

    You can easily run GLM Air (or other good MoE models) on like a 3080 + system RAM, or even a lesser GPU. You just need the right software and quant.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 11 months ago (2 children)

    Thanks a ton, saves me having to navigate the slopped up search results ('AI' as a search term is SEOd to death and back a few times)

    I dunno what card you have now, but hybrid CPU+GPU inference is the trend days.

    That system has the 3080 12GB and 64GB RAM but I have another 2 slots so I could go up to 128GB. I don't doubt that there's a GLM quant model that'll work.

    Is ollama for hosting the models and LM Studio for chatbot work still the way to go? Doesn't seem like there's much to improve in that area once there's software that does the thing.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 11 months ago*

    And IMO… your 3080 is good for ML stuff. It’s very well supported. It’s kinda hard to upgrade, in fact, as realistically you're either looking at a 4090 or a used 3090 for an upgrade that’s actually worth it.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 2 points 11 months ago*

    Oh no, you got it backwards. The software is everything, and ollama is awful. It’s enshittifying: don’t touch it with a 10 foot pole.

    Speeds are basically limited by CPU RAM bandwidth. Hence you want to be careful doubling up RAM, and doubling it up can the max speed (and hence cut your inference speed).

    Anyway, start with this. Pick your size, based on how much free CPU RAM you want to spare:

    https://huggingface.co/ubergarm/GLM-4.5-Air-GGUF

    The “dense” parts will live on your 3080 while the “sparse” parts will run on your CPU. The backend you want is this, specifically the built-in llama-server:

    https://github.com/ikawrakow/ik_llama.cpp/

    Regular llama.cpp is fine too, but it’s quants just aren’t quite as optimal or fast.

    It has two really good built-in web UIs: the “new” llama.cpp chat UI, and mikupad, which is like a “raw” notebook mode more aimed at creative writing. But you can use LM Studio if you want, or anything else; there are like a bazillion frontends out there.

  • permalink
  • fedilink
  • source
  • parent