you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 2 months ago (1 child)

Thanks. I did get it working on the 3090 alone with reduced context and KV cache quantized to 4 bits, but it was getting itself muddled up and going round in circles more. Once I got it up to the full 262K context and 8-bit KV cache, it seemed noticeably less confused and more useful, but it needed the extra 10GB for that.

  • source
  • parent
  • hideshow 1 child comment