1146
hmmm (lemmy.blahaj.zone)
submitted 1 year ago by to c/programmer_humor@programming.dev
52 comments fedilink hide all child comments
 
you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 1 year ago (2 children)

If you've got a decent Nvidia GPU and are hoping on linux, look into the Kobold-cpp Vulkan backend, in my experience it works far better than the CUDA backend and is astronomically faster than the CPU-Only backend.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 3 points 1 year ago (1 child)

    Will look into that when I have some money to invest. Thank you 💪

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 3 points 1 year ago

    When/If you do, a RTX3070-lhr (about $300 new) is just about the BARE MINIMUM for gpu inferencing. Its what I use, it gets the job done, but I often find context limits too small to be usable with larger models.

    If you wanna go team red, Vulkan should still work for inferencing and you have access to options with significantly more VRAM, allowing you to more effectively use larger models. I'm not sure about speed though, I haven't personally used AMDs GPUs since around 2015.

  • permalink
  • fedilink
  • source
  • parent