you are viewing a single comment's thread
view the rest of the comments
[–] 7 points 1 year ago (8 children)

Im not sure if it's a me issue but that's a static image. I figure you posted where they throw a brick into it.

Also, if this post was serious, how does a highly quantitized model compare to something less quantitized but with fewer parameters? I haven't seen benchmarks other than perplexity which isn't a good measure of capability?

  • source
  • hideshow 8 child comments
  • [–] [S] 7 points 1 year ago (4 children)

    It's a webp animation. Maybe your client doesn't display it right, i'll replace it with a gif

    Regarding your other question, I tend to see better results with higher params + lower precision, versus low params + higher precision. That's just based on "vibes" though, I haven't done any real testing. Based on what I've seen, Q4 is the lowest safe quantization, and beyond that, the performance really starts to drop off. unfortunately even at 1 bit quantization I can't run GLM 4.6 on my system

  • source
  • parent
  • hideshow 4 child comments
  • [–] 3 points 1 year ago* (2 children)

    What's higher precision for you? What I remember from the old measurements for ggml is, lower than Q3 rarely makes sense and roughly at Q3 you'd think about switching to a smaller variant. But on the other hand everything above Q6 only shows marginal differences in perplexity, so Q6 or Q8 or full precision are basically the same thing.

  • source
  • parent
  • hideshow 2 child comments
  • [–] [S] 4 points 1 year ago (1 child)

    As a memory-poor user (hence the 8gb vram card), I consider Q8+ to be is higher precision, Q4-Q5 is mid-low precision (what i typically use), and below that is low precision

  • source
  • parent
  • hideshow 1 child comment
  • [–] 6 points 1 year ago* (last edited 1 year ago)

    Thanks. That sounds reasonable. Btw you're not the only poor person around, I don't even own a graphics card... I'm not a gamer so I never saw any reason to buy one before I took interest in AI. I'll do inference on my CPU and that's connected to more than 8GB of memory. It's just slow 😉 But I guess I'm fine with that. I don't rely on AI, it's just tinkering and I'm patient. And a few times a year I'll rent some cloud GPU by the hour. Maybe one day I'll buy one myself.

  • source
  • parent