you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 5 days ago (5 children)

I'm running gpt4all on a thinkpad T470, and although is slow I rather use my own hardware for running LLMs

  • source
  • hideshow 5 child comments
  • [–] 2 points 5 days ago

    If you have enough regular ram and an nvme ssd you can run huge models on modest hardware using Colibrí.

    I got a Lenovo mini PC with 64GB of ram and a 500GB nvme and a 38B model runs with 11 tokens per second on my machine.

    The 38B model fits in my ram but if I had used a 120B parameter model then it would stream the rest of the model from my nvme disk.

  • source
  • parent
  • [–] 3 points 5 days ago (2 children)

    Please don't run LLMs at all...

  • source
  • parent
  • hideshow 2 child comments