1
To Perchance's owner regarding the new AI Chat Generator (lemmy.world)
submitted 1 year ago by to c/perchance@lemmy.world
5 comments fedilink hide all child comments
 

I am not sure if Perchance's owner will ever read my message, but the fact is that despite being a huge community, we have no way to communicate or speak directly to the person who gave us the chance to enjoy the wonders of AI for free.

I just want to thank you for finally updating the AI chatbot after 1.5 years of waiting, but now we are completely lost, so can you please publish an update on the site or post here and tell us about the new model? We are speculating that it may be Llama 3.3, DeepSeek, or Claude Sonnet 3.5. And for the love of god, can you increase the token context window? If we are really using models that can remember more than 100k tokens, having a tiny 6k token window is insane. Thanks.

you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 1 year ago* (2 children)

Where's the bot that's behaving bad? These 2 URLs look to be Anthropic API to me:

https://perchance.org/ai-story-generator

https://perchance.org/ai-chat

And for the love of god, can you increase the token context window? If we are really using models that can remember more than 100k token

This specification is an absolute lie, most of the time. Most '128K' LLMs fall apart after 24K tokens or so, even big API ones. There are only a few worth anything past 64K.

In addition, it's also very hardware intense and slow to host very long context windows (the way most folks host them without knowing any better). If they're hitting APIs instead, its usually very expensive that way too, as there's no caching and they get charged for submitting 100K tokens with every turn.

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
  • [–] [S] 1 point 1 year ago* (1 child)

    I know, I don't need to prompt an entire book as lore, I am just asking for 15-20k.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 3 points 1 year ago*

    Yeah, that's doable. But the context is likely low for cost reasons.

    The writing UIs like that are pretty 'normal', formatted like Kobold or SillyTavern: https://lite.koboldai.net/

    So you might consider dropping a few pennies on a legit dungeonmaster/text RPG finetune via some API, like openrouter. Hosting Claude or Llama like Perchance does for free is pretty generous, TBH.

  • permalink
  • fedilink
  • source
  • parent