cross-posted from: https://sh.itjust.works/post/61139432

I seriously can't believe how much progress he's made for the FOSS community. He actually might take a bite out of the big 3's profits with this

you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 5 months ago (1 child)

Yes, offloads into system. Oh and i forgot to mention that's with the context set around 25k. That can vary greatly per model though, it's taken some experimentation to figure that out.

  • source
  • parent
  • hideshow 1 child comment