99
I gave Qwen 3.8 27B a reverse-engineering job I assumed needed a frontier model, and it finished in 30 minutes (www.xda-developers.com)
submitted 2 months ago by to c/technology@lemmy.world
32 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 2 months ago (2 children)

Yeah so I've been working with Claude on that. Typically it greps to find what it's looking for and that can be a lot of tokens. So I created something halfway between rag and that (semantic search), and overall it lowers token usage a bit, but even if it optimistically reduces tokens 20%, Claude is hungry for docs.

There is automatic compaction, but I typically want to control that myself when I change focus (if I don't just /clear it). Still I'd say almost all of the stuff I do runs north of 100k tokens.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 3 points 2 months ago (1 child)

    Surprised the community hasn't found better ways to reduce context, 100k is insane. Is it possible to make the agent work on smaller pieces at a time, so less context is needed per piece?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 2 months ago

    Oh for sure. Agents often use under 30k tokens, but the orchestrator needs to have enough information to instruct the agents so it typically is a fair bit bigger in context. Agents tend to save you money on frontier models, but I'm skeptical about local LLMs. I suppose if you aren't pressed for time it's probably just fine. I haven't played that much with it because anything big enough to bother with agents I typically feel is too big for local anyway. But I'm sure others have experimented more on that front than I have.

  • permalink
  • fedilink
  • source
  • parent