you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 1 year ago

That fixed it.

I am a fan of this quant cook. He often posts perplexity charts.

https://huggingface.co/ubergarm

All of his quants require ik_llama which works best with Nvidia CUDA but they can do a lot with RAM+vRAM or even hard drive + rams. I don't know if 8gb is enough for everything.

  • source
  • parent