I just meant that you have to be cognizant of what went into the quantization.

As an example, a “Q4_K_M” could be too much quantization to be usable on one model, and an inefficient waste of space on the other. Two Q4_K_Ms of the exact same model could be completely different, one totally borked. Or one particular Q4_K_M could excel in one task, but be totally useless for another, even with the exact same settings, when a slightly different sized or type of quantization would excel.

It’s a deep rabbit hole. It’s not random either; there are distinct technical reasons behind every case mentioned above.

And that’s not even at the cutting edge quantization anymore, though what’s “cutting edge” completely depends on your particular hardware and use case.

I’m trying to make this sound daunting on purpose.

Many people have really horrible experience with a default “ollama run” for this exact reason, because the defaults are terrible and the customization is critical to getting coherent, performant output.

Unquantized LLMs, on the other hand, are basically always run the same way: vllm docker image on a big server, official weights. There’s less to “go wrong” trying to squeeze it on hardware with unofficial runtimes and compressors.