▲ 496 ▼ DeepSeek Permanently Reduces The Price Of Its Flagship V4 Model By 75 Percent (tech.yahoo.com) submitted 5 months ago by jaykrown@lemmy.world to c/technology@lemmy.world 127 comments fedilink hide all child comments
[–] Mwa@thelemmy.club 6 points 5 months ago* (6 children) Still gonna self host it instead (maybe) permalink fedilink source hideshow 6 child comments replies: [–] qaz@lemmy.world 3 points 5 months ago (5 children) FYI the flash model is ~158 GB permalink fedilink source parent hideshow 5 child comments replies: [–] Tja@programming.dev 1 point 5 months ago (3 children) How are they running it? Doesn't the model have to fit in (V)RAM? Does Nvidia have such huge memories in the H cards? permalink fedilink source parent hideshow 3 child comments replies: [–] BlackLaZoR@lemmy.world 4 points 5 months ago There's tech for splitting model to run on multiple cards, but it requires really fast interconnect between GPUs. permalink fedilink source parent [–] MangoPenguin@lemmy.blahaj.zone 2 points 5 months ago Lots of GPUs together. permalink fedilink source parent [–] boonhet@sopuli.xyz 1 point 5 months ago* For self hosting it essentially needs to fit in VRAM + RAM but it'll take a lot of CPU for the part in RAM Deepseek probably uses those big fancy H cards and not one but several together to increase VRAM. permalink fedilink source parent [–] Mwa@thelemmy.club 1 point 5 months ago The destiled models? permalink fedilink source parent
[–] qaz@lemmy.world 3 points 5 months ago (5 children) FYI the flash model is ~158 GB permalink fedilink source parent hideshow 5 child comments replies: [–] Tja@programming.dev 1 point 5 months ago (3 children) How are they running it? Doesn't the model have to fit in (V)RAM? Does Nvidia have such huge memories in the H cards? permalink fedilink source parent hideshow 3 child comments replies: [–] BlackLaZoR@lemmy.world 4 points 5 months ago There's tech for splitting model to run on multiple cards, but it requires really fast interconnect between GPUs. permalink fedilink source parent [–] MangoPenguin@lemmy.blahaj.zone 2 points 5 months ago Lots of GPUs together. permalink fedilink source parent [–] boonhet@sopuli.xyz 1 point 5 months ago* For self hosting it essentially needs to fit in VRAM + RAM but it'll take a lot of CPU for the part in RAM Deepseek probably uses those big fancy H cards and not one but several together to increase VRAM. permalink fedilink source parent [–] Mwa@thelemmy.club 1 point 5 months ago The destiled models? permalink fedilink source parent
[–] Tja@programming.dev 1 point 5 months ago (3 children) How are they running it? Doesn't the model have to fit in (V)RAM? Does Nvidia have such huge memories in the H cards? permalink fedilink source parent hideshow 3 child comments replies: [–] BlackLaZoR@lemmy.world 4 points 5 months ago There's tech for splitting model to run on multiple cards, but it requires really fast interconnect between GPUs. permalink fedilink source parent [–] MangoPenguin@lemmy.blahaj.zone 2 points 5 months ago Lots of GPUs together. permalink fedilink source parent [–] boonhet@sopuli.xyz 1 point 5 months ago* For self hosting it essentially needs to fit in VRAM + RAM but it'll take a lot of CPU for the part in RAM Deepseek probably uses those big fancy H cards and not one but several together to increase VRAM. permalink fedilink source parent
[–] BlackLaZoR@lemmy.world 4 points 5 months ago There's tech for splitting model to run on multiple cards, but it requires really fast interconnect between GPUs. permalink fedilink source parent
[–] MangoPenguin@lemmy.blahaj.zone 2 points 5 months ago Lots of GPUs together. permalink fedilink source parent
[–] boonhet@sopuli.xyz 1 point 5 months ago* For self hosting it essentially needs to fit in VRAM + RAM but it'll take a lot of CPU for the part in RAM Deepseek probably uses those big fancy H cards and not one but several together to increase VRAM. permalink fedilink source parent