So Matt Pocock says models typically are smartest in the first 140k tokens. Hes using the big boy cloud models tho. Does that hold true for 27b-35b range? Or is it smaller and how much?

What would the ‘smart range’ be for tiny models that fit in 2gb? Would be it be like a sentence or two? I have an old 1050 was thinking of throwing in rig but im not sure what id even be able to do with such a small model

you are viewing a single comment's thread
view the rest of the comments
[–] 5 points 3 weeks ago*

There's scientific papers about that kind of degradation. Like this one from earlier this year.

In short, from what I read: No, I don't think it holds true. It'd be weird if it was some fixed number like 150k. If at all, I'd expect it to be a percentage of the maximum context length of a specific model. And it probably also depends on the exact model and how the attention and the parameters are set up.

  • source