Hi All, I just got a P100, and although it appears to work, i cant quite get gemma4 to run on it properly.

My setup is:

  • Ubuntu 22.04 VM
  • PCI passthrough the GPU (Connected via thunderbolt) (I know this is unideal, and I might try cut this out later)
  • Nvidia 580 driver installed.
  • llama.cpp built from source, with cuda enabled.

I got some mostly normal output from qwen, but whenever I try to load gemma4 I get a lot of <unused49><unused49><unused49><unused49>. I did have it briefly working, no idea what changed, but reboot and its back to unused spam.

Anyone hit this before? Ideas on what I should try change?

you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 2 months ago (1 child)

What version of llama.cpp are you using (exactly), what command are you using to run the model (exactly), and which version of Gemma are you trying to run (exactly)?

There are open issues on llama.cpp where people have gotten similar <unused49> spam under various circumstances, but the cause is not clear -- e.g. https://github.com/ggml-org/llama.cpp/issues/26088 -- if this can be consistently reproduced, you may want to let them.

Similarly, people mentioned it on some of unsloth's quants a few months back: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/discussions/2

As far as general workaround advice: try Vulkan instead of CUDA (or vice versa), try a different version of llama.cpp (especially if you're running either a bleeding edge version or a very old version), and try a different version of the model.

  • source
  • hideshow 1 child comment