[–] [S] 2 points 5 months ago

hmmm..I looked into it and honestly dont see many similarities.

I gave up on constructing... my own tool call pipeline, since most LMs already come with one built in. only problem is that its craaaaazy verbose and eats up at least like 70 tokens per tool definition.

but its reliable... kinda. model has been trained to call tools this way. so.... thats how its meant to be iguess.

  • source
  • parent
  • context
  • [–] [S] 3 points 5 months ago

    yyyyes, highly recommend switching to llama cpp.

    the serving command is easy:

    llama-server -m <model file>
    

    and you can quantize the context for much better speed and use Qwen 3.5 MTP and such. that can look like this:

    llama-server -m Qwen3.6-35B-A3B-IQ2-M.gguf \
    -ctk q4_0 -ctv q4_0 \
    --spec-type draft-mtp --spec-draft-n-max 2
    

    it surprising howuvh more speed you get over long contexts with those context quantizations with percievably low quality loss.

  • source
  • parent
  • context
  • [–] 3 points 5 months ago* (1 child)

    I've been using ollama for most of my experiments so far (slowly switching over to using llama.cpp directly now). My UI is in the browser and I have a custom Tornado server in the middle (which handles local file access, etc. for tool calls). The UI talks to my server via a websocket, and the server talks to ollama via httpx. I wrote both parts using async -- which seemed like a natural choice at the time, but was not a good one in retrospect...

    Detangling which parts can block each other in my code from other issues like misunderstanding the sequencing of tool calls and assistant messages (e.g. LLMs can send assistant messages back before making a tool call -- not just during thinking -- and I also implemented tool call execution immediately upon receiving the call... which was a mistake) and running head-first into unexpected limitations of ollama/llama.cpp like the fact that I can't actually run two instances of the same MoE model concurrently in ollama (but can run multiple other models concurrently like Qwen3.6 + Gemma4 or two versions of Qwen that have separate modelfiles). I wanted some tool calls to be able to use other LLMs with different configuration... (It also seems like ollama just completely ignores temperature settings from HTTP requests... which sent me on a wild goose chase for a long time.) Add into all that mess the fact that I want to store the entire structure of the conversation history and be able to both (1) refresh the client page and reload the conversation from the server, and (2) export the conversation to a file and reload it from the client -- which I got working early on with the wrong history structure... -- and the whole thing just ends up making me want to tear my hair out.

    Interrupting the easy cases was easy by dropping the connection. Interrupting it in the middle of tool use with all my mistakes and cutting myself on the sharp edges in the ecosystem? Not so much...

    I know what I need to do now to get to where I want to be -- I'm not stuck -- but it's been a slog to untangle the whole thing.

    Edit: Oh, I also forgot to mention that I stream the responses (including thinking) to the client as they arrive. That was one more knot in the puzzle.

  • source
  • parent
  • context
  • [–] [S] 3 points 5 months ago (2 children)

    the "interrupting it, send a message and continue" problem is generally pretty easy to implement, at least in code (ive done it before).

    are you vibecoding your thing or actually stuck on something? or is it more of a UI problem rather than a logical messages array problem?

  • source
  • parent
  • context
  • [–] [S] 3 points 5 months ago

    oh nono, im happy to hear you work with their models!

    Plus, 27B and Max Qwen models are really good, no question.

    thank u for your two cents.

    if you plan on using some language model in... production or whatever, maybe look into deepseek V4 and Xiaomi Mimo 2.5 pro. both are gigantic models (1 trillion params +) and they are under a dollar for 1M putout and.... below two cents for cached 1M input.

    im not trynna sell anything, bzt those are currently the crazies permanent offers ive seen (cuz what you described sounded like you were paying for the API. may be worth trying deepseek)

  • source
  • parent
  • context
  • [–] 4 points 5 months ago (3 children)

    I saw your post! Fair enough if you're burning out on it. I'm about two months into my own LLM experience and while I haven't burned out yet, I can definitely feel the fatigue from how much effort I've poured into trying to make local models viable.

    I have actually gotten a fair bit of utility out for the effort I've put in. Actually good OCR without the ton of preprocessing I used to need to do for Tesseract. "Good enough" dictionary look-up, and "good enough" language translation for individual words or small bits of text. Brainstorming assistance. UI prototyping. (They're terrible at writing maintainable code, but getting something "good enough" for quick hands-on proof-of-concept checks has been nice.) "Catch my stupid mistake" code debugging. Structured data extraction from natural language for some tedious recurring work tasks I need to do. Plus a lot of just silly things for my own amusement.

    But yeah... Dealing with the rough edges in the ecosystem esp. churn in llama.cpp result in old advice leading me down rabbit holes of wrong paths for hours/days, and crappy/no documentation for things like the proper User/Agent/Tool Call/Tool Response interaction sequence has been draining lately. I think I've burned two weekends at this point trying to just make my custom harness stop reliably when I need to interrupt it without taking the whole program down and bringing it back up. (I have something that mostly works at this point, but it's got enough annoying issues that I'm still not satisfied...)

  • source
  • [–] 2 points 5 months ago (1 child)

    I don’t claim to be an expert on LLMs or AI, but having been closely testing free versions of big players, I have been impressed with Qwen and have regularly adopted it into my workflow. Sorry if this wasn’t the type of answer you were looking for but I have found Qwen to be quite in par with closed models from US companies. Just my two cents.

  • source
  • [–] 1 point 6 months ago (1 child)

    if you are looking to embed code into free-flowing text output, you can go with a format such as this:

    
    this is some free-flowing text output from the LLM to write to a file.
    
    > write("filename", "content")
    > someotherfunction()
    
    this is more free-flowing text
    
    

    you write the commands with indentations (>) and basically ignore every other line. some languages do it that way. i think PHP uses opening tags to denote where code starts/ends, the rest is just ignored by the PHP interpreter.

  • source
  • parent
  • context
  • [–] [S] 2 points 6 months ago (1 child)

    then i have "bad" news.

    i made thiup, it's not a standard.

    meaning: you would have to write the entire agentic scaffolding yourself.

    which honestly, is a fun project.
    ive done it, its nice being able to see exactly what's goin on.

    im sure ur super duper smart and already know this, but: all that an agent is, is just:

    • parsing the tokens as they come in
    • notice when the LM writes a tool call
    • parse it after its done
    • execute
    • return the output

    so umm... if thats what u feel like doing, good luck! πŸ’–

  • source
  • parent
  • context
  • [–] [S] 2 points 6 months ago* (3 children)

    ohgosh playing a whole game is a whole different level of complexity >o<

    it can work, but only under very short time horinzons and it will likely get stuck immediately trying to do the same thing. (im assuming you mean the text-based version)

    see claude opus trying to play runescape here and skip to 6:55 where it starts. you will notice: its interesting to watch, but gets stuck quickly. and that model was SOTA at the time.

    the model would also have difficulties telling what us where. sounds weird, but due to tokenization, it cant "infer" which characters are above which others. it will be guessing wrong a lot.

    you would be babysitting the model.

    right now, LMs and VLMs are being trained for computer use (where they click on a screen and use a computer "like a human would"), but exclusively for work tasks and not games.

    try this first: run ollama run qwen3:4b-instruct-2507-q4_K_M --experimental. that launches ollama in an "agent mode", where it can run shell commands, which essentially gives it one tool: bash(command:str).

    test it saying "yo what files and dies are in this directory?" and it will run ls probably and tell you.

    if you really - REALLY want it to work with the game, the best option would be:

    • download the VL version of the model ollama pull qwen3-vl:4b-instruct (or -thinking) so it can understand images
    • give it lots of context. 16k - 32k is...kinda the minimum
    • making some script which saves a screenshot of the game to a file
    • put the qwen into some pre-built agent, like opencode or goose (which has GUI)
    • tell it "start dwarf fortress as a background process, use this script to take screenshots, look at them and use xdotool to navigate the game. Perform the loop of take screenshot -> look at it -> take action on the game" (or some other mouse / keyboard usage command)

    theres no reason to use my format, its just a tolen-efficiency thing. with your fast GPU-throughput, this shouldnt be a problem.

    ohgosh long response-

  • source
  • parent
  • context
  • [–] [S] 2 points 6 months ago* (last edited 6 months ago) (2 children)

    thanks for sharing, but the goal here was not to make yet another key-value format, but to have natural feeling multiline-strings with minimal escaping.

    thats why i settled for the code blocks: they tokenize well, are universally understood as "this is some text", are easy to write and rarely ever have to deal with escape sequences.

    in this case, the [ and ] also serve as tool-call delimiters, which would usually be some heavy XMS ones <functions> </functions>...thats 6 - 10 tokens down the drain for delimiters! >o< aaaaa

    thank u for engaging with the post btw. i really appreciate it <3

  • source
  • parent
  • context