you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 2 months ago (1 child)

Surprised the community hasn't found better ways to reduce context, 100k is insane. Is it possible to make the agent work on smaller pieces at a time, so less context is needed per piece?

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 2 months ago

    Oh for sure. Agents often use under 30k tokens, but the orchestrator needs to have enough information to instruct the agents so it typically is a fair bit bigger in context. Agents tend to save you money on frontier models, but I'm skeptical about local LLMs. I suppose if you aren't pressed for time it's probably just fine. I haven't played that much with it because anything big enough to bother with agents I typically feel is too big for local anyway. But I'm sure others have experimented more on that front than I have.

  • source
  • parent