all 9 comments

sorted by: hot top controversial new old
[–] 4 points 1 month ago* (last edited 1 month ago) (5 children)

LLMs suck at counting and therefore suck at writing valid lisp. (Source: I've seen people trying to use code agents with clojure))

Sorry it seems I miscounted the (parenthesis)

You are totally right my fix made the syntax valid but doesn't work...

  • source
  • hideshow 5 child comments
  • [–] 1 point 1 month ago (4 children)
  • [–] 2 points 1 month ago (3 children)

    A linter will tell you that you counted wrong it will not tell you where.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 1 point 1 month ago (2 children)

    Citation needed.

    I asked a LLM to start a stdio REPL, write code and then copy the code into the REPL. It closely mirrors actual Lisp development process - you write a couple of lines, eval it, see why it didn't work, and repeat. Here's what it came up with:

    Starting Lisp

    $ : > /tmp/lisptest-repl.commands; : > /tmp/lisptest-repl.log; (tail -n +1 -f /tmp/lisptest-repl.commands | sbcl --noinform > /tmp/lisptest-repl.log 2>&1) & echo $! > /tmp/lisptest-repl.pid; sleep 1; printf '(require :asdf)\n(asdf:load-asd
    #P"/home/user/lisptest/lisptest.asd")\n(asdf:load-system "lisptest")\n' >> /tmp/lisptest-repl.commands; sleep 2; tail -40 /tmp/lisptest-repl.log (timeout 10s)
    
    * ("ASDF" "asdf" "UIOP" "uiop")
    * #<ASDF/FIND-SYSTEM:DEFINE-OP >
    #<ASDF/PLAN:SEQUENTIAL-PLAN {1204C49563}>
    * T
    *
    
    Took 3.1s
    

    Invalid input

    Feeding a single ) into the REPL. The LLM later recovered it by sending the abort command to the debugger.

    $ printf ')\n' >> /tmp/lima-repl.commands; sleep 1; tail -25 /tmp/lima-repl.log (timeout 10s)
    
    * #<ASDF/FIND-SYSTEM:DEFINE-OP >
    #<ASDF/PLAN:SEQUENTIAL-PLAN {1204C49563}>
    * T
    * #<SB-THREAD:THREAD tid=151 "lima-echo" RUNNING {1203EB14E3}>
    * Echo server listening on 127.0.0.1:5555
    T
    * NIL
    * #<SB-THREAD:THREAD tid=0 "lima-echo" RUNNING {1201C0DDC3}>
    * Echo server listening on 127.0.0.1:5555
    T
    *
    debugger invoked on a SB-INT:SIMPLE-READER-ERROR in thread
    #<THREAD tid=142 "main thread" RUNNING {12003D8003}>:
      unmatched close parenthesis
    
        Stream: #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDIN* {12014C9E73}>
    
    Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.
    
    restarts (invokable by number or by possibly-abbreviated name):
      0: [CONTINUE] Ignore it
      1: [ABORT   ] Exit debugger, returning to top level.
    
    (SB-IMPL::READ-RIGHT-PAREN #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDIN* {12014C9E73}> #<unused argument>)
    0]
    
    Took 1.0s
    

    Hotpatching

    This thing activates a new function definition. Next time it's called, the new one gets executed.

    printf '(load #P"/home/user/lisptest/src/server.lisp")\n' >> /tmp/lisptest-repl.commands
    
  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 1 month ago (1 child)
    (defn process-scores [user & scores]
      (println "User:" user) (reduce + scores  (filter pos? scores))
    

    can be either

    (defn process-scores [user & scores]
      (println "User:" user)
      (reduce + scores)
      (filter pos? scores))
    

    or

    (defn process-scores [user & scores]
      (println "User:" user)
      (reduce + scores (filter pos? scores)))
    
  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 1 month ago

    J as all the benefits of lisp, and more, without annoying ((())). LLMs are bad at understanding it though (which is one of the claims that Graham makes for lisp here). Lisp would have the same lack of training data compared to python/js to have models good at programming in lisp?

  • source
  • [–] 1 point 1 month ago (1 child)

    is the plan here to give an llm access to a running instance of the app and just... make changes? fantastic, can't imagine what might go wrong

  • source
  • hideshow 1 child comment