49
[dev] New text model is still quite bad sorry - working on it (lemmy.world)
submitted 1 year ago by to c/perchance@lemmy.world
94 comments fedilink hide all child comments
 

I didn't expect it to take this long for me to get the new text model under control. It's much more intelligent in general, but the repetition and random fixations can get surprisingly bad.

Sorry about that. This is just a heads up that I'm still working on it, and it'll likely remain a bumpy ride for the next couple of months.

Thanks to those who have complained here and on reddit, especially the posts that give concrete examples of issues, and attempted workarounds, etc.

Please do keep posting any issues/discoveries, ideally with share links to give me specific examples that I can test.

Other news:

  • I'm still working on the next image gen update, and am hoping it will improve fine details, text rendering, and overall image quality. It should also be about 2x faster. If all goes well (which, from experience, it won't) I should be able to get that done in the next month or so.

  • For a long time I've been very close to shipping a multiplayer plugin and database plugin. I've actually made multiple 'ready to ship' draft plugins of both, but have been disatisfied with the design and ended up not publishing each time. Using these plugins won't trigger ads on the generator, so I need to be really careful about the overall architecture to ensure the the servers for them are dirt-cheap to run. I also want to make them fun/simple to use of course. Once I get text and images stable, I'm going to focus on these two plugins and finally get them done.

you are viewing a single comment's thread
view the rest of the comments
[–] 0 points 3 months ago

To: perchance dev / ai-text-plugin maintainer Re: Repetition degeneration (token loops) — request repetition-penalty option

I’m hitting repetition degeneration in my generator (FurAI / ai-furry-generator). During streaming the model in Simplified/Traditional Chinese both VERY frequently falls into a tight token loop and emits runs like:

他地大掌猛"地地地地" 死死"地地地" 按在… 地地地地地地地地地地地地地地地地地地地地地地地地地地地地地地地地地地地地

— a single CJK character repeated 4+ times, sometimes continuing for the rest of the generation. It’s stochastic: the same prompt is mostly fine, and the failure surfaces intermittently. Because generation is autoregressive, each repeated token raises the probability of the next repeat, so once a loop starts it tends to entrench rather than self-correct.

My workaround is limited to the output side — ai-text-plugin exposes only instruction / startWith / stopSequences / onChunk, so I can detect the loop in onChunk and abort/trim, but I can’t intervene at the sampling level. That catches symptoms, not the cause.

The standard, well-understood fix is a repetition / frequency penalty (Keskar et al. 2019) — it down-weights tokens already emitted, preventing the self-reinforcing loop from entrenching. It’s a default knob in llama.cpp, vLLM, and OpenAI’s API (frequency_penalty, presence_penalty).

Two possible asks, either helps:

Apply a modest repetition penalty server-side in the text service if one isn’t already enabled; or Expose it as an ai-text-plugin option (e.g. repetitionPenalty / frequencyPenalty) so authors can tune per-call. Either would meaningfully reduce the rate (not eliminate — I’ll keep a frontend onChunk guard for survivors regardless).

Thanks

  • permalink
  • fedilink
  • source
    •