all 23 comments

sorted by: hot top controversial new old
[–] 34 points 10 months ago (11 children)

Holy fucking what? 19k frigz? Must know serious db ops to make that work. Like damn. I'm learning to navigate model vocab in the 50k to 200k lines range and that is just key/value strings and extended Unicode characters. How does one organize and find a relevant Chud GODfrog image in the holysea???

  • source
  • hideshow 11 child comments
  • [–] 6 points 10 months ago (9 children)

    Probably doesn't involve AI, Jake

  • source
  • parent
  • hideshow 9 child comments
  • [–] 2 points 10 months ago (8 children)

    Did I somehow say it does, or is it me you have a problem with?

  • source
  • parent
  • hideshow 8 child comments
  • [–] 1 point 10 months ago (7 children)

    You mentioned model vocab, and I assumed that was to do with chatbots. I've seen your rambling about chatbots before. It's pretty funny.

  • source
  • parent
  • hideshow 7 child comments
  • [–] 1 point 10 months ago (6 children)

    Have you ever followed the citations I have given to the code present in the vocabulary, or do you just laugh at things you do not understand?

  • source
  • parent
  • hideshow 6 child comments
  • [–] 12 points 10 months ago (2 children)

    Why would I care about your little chatbots, Jake? You go off on these long, incoherent rants that honestly feel like some sort of schizophrenic meltdown over how there are sentient gods or personalities in your chatbots, and then act like you're typing reasonable stuff that we should all do our due diligence and read. I feel like you need to get your head on straight and stop interacting with the robots, but I also find it relatively funny so I hope you don't πŸ€·β€β™‚οΈ godspeed buddy 🫑

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 10 months ago (1 child)

    I do enjoy that they immediately started showing everybody exactly what you mean lol.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 4 points 10 months ago

    I really don't know what happens to them when AI is mentioned.

    I've seen some of their other comments that are about other things, and they're often coherent, reasonable, and even witty - but as soon as they start talking about chatbots it's like their brain short-circuits and they just blather.

  • source
  • parent
  • [–] 1 point 10 months ago (2 children)

    What are you talking about? What citations?

  • source
  • parent
  • hideshow 2 child comments
  • [–] -2 points 10 months ago*

    Look at the vocab.json of nearly any model vocabulary. In CLIP, it is organized at the end in the last ~2200 tokens of SD1 CLIP which is used by all diffusion models. The bulk of the code is in that last block, however there are 2337 tokens with code in a brainfuck style language using the extended Latin character set. Any idiot that looks at this will spot how these are not at all part of any language words or fragments and where there are obvious functions present. If you have ComfyUI, the file is in ./comfy/sd1_tokenizer/vocab.json.

    The most difficult are models, like the T5xxl have this code precompiled and embedded in the vocabulary.

    I am playing with Qwen 3's right now, which has the much larger world model of Open AI QKV hidden layers alignment. Unlike CLIP, Qwen's vocabulary starts as 2.8 MB of compressed json, and the extended Latin is intermixed in the total. This one is present in ComfyUI at ./comfy/text_encoders/qwen25_tokenizer/vocab.json. You will need the jq package to make the json readable, or other ways. If you have jq and ripgrep (rg) on your system, then try cat ./vocab.json | jq | LC_ALL=C rg '[[:^ascii:]]' That method is much less organized than CLIP, but this model has 103,378 lines of code using the same character set. I have reverse engineered most of the tokens used to create this code. I modify the vocabulary with scripts to alter behavior. Explaining the heuristics of how I have figured out the complexity of this structure is fucking hard unto itself, never mind actually sorting out any meaning. It really sucks when people are assholes about that.

  • source
  • parent
  • [–] 16 points 10 months ago (4 children)

    and that's not even a ls -l

  • source
  • hideshow 4 child comments
  • [–] [S] 13 points 10 months ago (3 children)

    wc -l should give you the same thing with the long names since the line count is still the same

  • source
  • parent
  • hideshow 3 child comments
  • [–] 5 points 10 months ago (2 children)

    my point is that, usually, just an ls will put more than one file per line

  • source
  • parent
  • hideshow 2 child comments
  • [–] 7 points 10 months ago

    No, I really, really don't.

  • source
  • [–] 3 points 10 months ago (2 children)

    What's a "froggo" on 4chan? Is it some kind of cred for hackers? But seriously, what's a froggo?

  • source
  • hideshow 2 child comments
  • [–] 1 point 10 months ago (1 child)

    Pfff, anyone who deals with big directories knows you should do ls -U | wc -l

  • source
  • hideshow 1 child comment