In a comment over at !selfhosted@lemmy.world someone wrote:

Containers are not a security barrier. They cannot be, they never will be.

I'm no expert but my mental model of containers was different in that I thought containerized processes live in somehow separate environments, at least filesystem- and network-wise.

I run self-hosted services in containers on my server to isolate them from each other, and now and then binaries I don't fully trust (things like suspected slop code or not well known python libraries) in containers on my laptop.

Does someone with more technical knowledge about how containers work feel like explaining whether that is a bad idea and if so why? That would be highly appreciated.

Edit: missing words and punctuation.

Edit 2: I would like to thank everyone who took the time to reply. My key take aways are that I will move my self-hosted services into VMs for proper isolation, but that it's probably fine to use a container when I want to make sure an LLM doesn't accidentally rm -rf my laptop system when it gets stuck.

Basically the title.

For me, a dense model around 20b, with an additional engram table of 10-20b, or a sparse MoE around 80b would be amazing.

I didn't even know they train their own models.

Have been playing around with this on my laptop's Radeon 780M, with 16GB of RAM assigned to the GPU. llama.cpp with Vulkan reaches a comfortable 30t/s TG with up to 1000t/s PP when running Q8_0 with unquantized KV-cache.

In the few rather simple summarisation tasks I gave it so far it performed better than expected, but of course it's very small MoE and you shouldn't expect too much. Looking forward to evaluating this in Pi on small agentic coding tasks.

Have some of you been playing around with it?

Some days ago I saw people who attended a Fridays for Future demonstration excitedly put political stickers on a shiny blue Lamborghini which was obviously parked at the wrong point in spacetime.

When discussing this with a friend, we concluded that there was quite strong symbolism in that situation - like direct payback for the unnecessary pollution of the planet, the car being the canvas where the activists were able to project their anger onto.

We also talked about luxury cars being a symbol of social inequality.

And only later it hit me, how luxury cars, among other things, are usually called status symbols and how actually they could also be called symbols of equality.

From the model card, sounds interesting:

The "Unified" in Gemma 4 12B Unified refers to its encoder-free architecture. Other Gemma 4 models use dedicated encoders to process multimodal data before passing it to the LLM. Gemma 4 12B eliminates these encoders entirely, projecting raw image patches and audio waveforms directly into the LLM's embedding space through lightweight linear layers. This unified approach means all modalities flow straight into a single decoder-only transformer, reducing multimodal latency and allowing the entire model to be fine-tuned in one pass.

The benchmarks put it closer to the 26b MoE than to the E variants of the Gemma4 series, but mostly below Qwen3.5 9b.

Looking forward to giving it a shot.

WTF? Do you see this as well?

In case you missed it, 2-3 weeks ago, experimental tensor-parallelism support was merged into llama.cpp.

In a nutshell, this allows in multi-GPU setups to not only combine the VRAM of the cards but also their computing power. The results depend a lot on the specific setup and model, but on my 3x RTX 2000e Ada rig running Qwen3.6-35b it almost doubled generation throughput (these are low-powered cards which are not very powerful on their own).

The option to turn it on is --split-mode tensor.

It's not yet officially documented, I assume because it's still experimental. But since #22362 was merged yesterday, in my case it now also work for the latest Qwen3.6 models.

The wait is over, most ggufs are already up. Nice to see there's models for many different hardware configurations.

It's out!

Hey everyone! I was just skimming through some inference benchmarks of other people and noticed the driver version is usually mentioned. It made me wonder how relevant this is. My prod server runs Debian 12 so the packaged nvidia drivers are rather old, but I'd prefer not to mess with the drivers if it won't bring a benefit. Does any of you have any experience or did do some testing?

Title says it - it's been 10 days already but I didn't catch the release. This might be huge for those of us running on multiple GPUs. At least for Gemma3, I was able to double inference speed by using vLLM with tensor parallelism vs. ollama's homegrown parallelism. Support in ExLlamaV3 could additionally allow to pair TP with lower-bit quants. Haven't tested this yet, but I'm looking very much forward to.

Tencent recently released a new MoE model with ~80b parameters, 13b of which are active at inference. Seems very promising for people with access to 64 gigs of VRAM.

Hey fellow llama enthusiasts! Great to see that not all of lemmy is AI sceptical.

I'm in the process of upgrading my server with a bunch of GPUs. I'm really excited about the new Mistral / Magistral Small 3.2 models and would love to serve them for me and a couple of friends. My research led me to vLLM with which I was able to double inference speed compared to ollama at least for qwen3-32b-awq.

Now sadly, the most common quantization methods (GGUF, EXL, BNB) are either not fully (GGUF) or not at all (EXL) supported in vLLM, or multi-gpu inference thouth tensor parallelism is not supported (BNB). And especially for new models it's hard to find pre-quantized models in different, more broadly supported formats (AWQ, GPTQ).

Does any of you guys face a similar problem? Do you quantize models yourself? Are there any up-to-date guides you would recommend? Or did I completely overlook another, obvious solution?

It feels like when I've researched something yesterday, it's already outdated again today, since the landscape is so rapidly evolving.

Anyways, thank you for reading and sharing your thoughts or experience if you feel like it.