-8
What could go wrong with coding and AI usage, right? (jeferson.me)
submitted 3 months ago by to c/programming@programming.dev
21 comments fedilink hide all child comments
 

Few weeks ago I wrote about a side project that I'm using to learn, this is my reviewing what I missed on the project, the main issues after a few dozen games in the platform with friends. And what I learned in the process.

All comments, feedback is appreciated.

you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 3 months ago* (last edited 3 months ago) (12 children)

Reads like they started experimenting with an AI-driven workflow. Got some prototypes and early progress fast. And now the first draft kinda works, but it breaks left and right and they're stuck in a loop to fix things, another thing breaks and they're not getting anywhere.

Difficult to say what went wrong judging by what they wrote. That kind of information is missing in their rant. Could be a wrong choice of AI model, Agent framework, wrong workflow... Or AI in general. I also tend to get caught up in loops, yelling at the AI for half a day to fix things, write more strict tests so it can see itself what went wrong, and fix whatever it just broke in the process. Or yell at it for giving up and refactoring the entire project because it can't figure out what went wrong and why.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 12 child comments
    •
  • [–] 5 points 3 months ago (1 child)

    Sounds about right. Fits in with the 80:20 rule. Does the first 80 quicker than the 20% of time it usually takes but does the next 20% which takes 80% of the time even slower.

    Effectively, it optimises the quicker bit so you suffer long term and overall waste more time while screwing the planet. It's only fine for prototyping and even then, would that be the best base to build off? Is it modular and maintainable? Does it have low coupling and high cohesion? Or is it just trash?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] [S] 2 points 3 months ago

    Heavily coupled code with a lot of duplication of code. These are the main points of the refactor.

    There is some lack of "smart" use of the JS, like the positional arguments instead of objects (for quick fix/change), no separation of concerns, there is something lightly there, but lacks the gross thing.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 3 months ago (3 children)

    ai is a chimera in people minds.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] [S] 1 point 3 months ago (2 children)

    What aspect of the chimera? Like the mythological way? Or this is closer to the "parts of everything touching"?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 3 months ago (4 children)

    That's because vibe coding isn't a thing. AI is remarkably good at following instructions and terrible at reading your mind. Give it specific instructions about what needs to be done and how it needs to be done, and it does really well. That doesn't mean it's not stupid sometimes, but that's why you check what it does. My biggest problem was getting it to stop over-designing and implement the smallest necessary change. I had to explain the difference between an implementation task and a refactoring task. That made a huge difference in quality.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 2 points 3 months ago* (last edited 3 months ago) (3 children)

    Yeah Idk. While I do in fact always have issues prompting AI... I kinda also think it has a weird form of "intelligence". I wouldn't say it does really well at following instructions. It tries. Certainly. But sometimes it rather does silly things. Like I recently was trying to prototype some web app. The app didn't work. So I told it to write tests. And it wrote some stub which always passed. I had an argument with the agent how it can't just tell me the project works now, but it always has to follow the usual agentic pattern: Plan, write code, execute the test suite, fix mistakes, repeat and only then add all files to a commit and tell me it's done. And now we need some tests for it itself to see whether the program runs. It just wrote super convoluted tests, launching some headless browsers, didn't ever get it right so ALL the tests kept failing. I told it to test functions and API endpoints instead (which is kind of a no brainer to do anyways). Those also kept failing. And after several hours of this nonsense it told me: TOO HARD, NEED TO FOCUS! And staged a commit which just deleted the entire test suite 😆 Hilarious. But neither what I wanted, nor what I instructed it to do...

    I forgot what model that was. I was testing the Hermes Agent at that time.

    I never know what amount of hand-holding is expected from me as the human operator. Models keep getting better. A year ago I had them make up convenient, yet nonexistent functions. Like mathproblem.solve() That hasn't hapoened in a while. But I can't be expected to instruct and teach it every tiny detail?!

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 3 months ago* (last edited 3 months ago) (2 children)

    I'm not a big fan of "agentic" coding. Don't get me wrong, I am trying to create a process by which agentic programming is more reliable, but it's piling AI artifacts on top of AI artifacts. The error rate compounds and by the time it's working on the 5th story everything is garbage. You have to review everything at every step to find and fix those errors to keep everything on track.

    I have a lot more success giving it one story at a time — each one targeted toward small, incremental steps. It does well. Then clear the context out and start another small story. I don't trust agentic programming.

    I have an AI to help do production support when I'm in meetings or whatever. It comes to the dumbest conclusions about the root cause and the resolution path. But the logs and queries it pulls for me are very helpful. And sometimes it's analysis holds a bit of useful insight.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 3 months ago* (last edited 3 months ago) (1 child)

    Interesting. Most other people seem to use coding agents. I mean I also tried the polar opposite end of the spectrum. Copy-paste snippets into ChatGPT and back. I think that's how a lot of people got started. But it's tedious, still makes me let the AI do things which I later regret in the design. And I tracked my time once or twice for similar tasks. And I was slower than just typing it down myself without any AI assistance. Guess I still have to figure out what to do to get some proper use out of it.

    What kind of tools do you use for your workflow? Some IDE plugin or "agent" so it has access to the entire project files, but then give it stories of a size it can tackle in one shot? Or subdivide?
    I mean I also tried to subdivide tasks into chunks and hand those out. But that also has some tendency the AI doesn't really get what it's supposed to do because it's missing the broader picture. That was kind of the reason for me to try agents, so it has a plan, knows what's next in line so it doesn't write code which doesn't integrate (as is) into the next step to do.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 3 months ago*

    I use Claude code. I have access to do full on agentic shit, but mostly all it does is fan out an agent or two to read code and documentation. My process is describe specifically what I want sort of like a story. If it takes more than a paragraph or ten bullets to explain in exact detail, it's too big. Generate an implementation plan from a template. Look the plan over. Make any changes. It writes the code, then writes the tests, then updates documentation. I review everything at each step.

    The implementation plan is critical because you'll see if it wants to do something like create an enum to hold a single value or refactor the name of a method for no particular reason.

    We have extensive documentation of standards and practices, and generally how we structure code. I have templates for things like writing up the implementation plan. Questions that need to be answered. Important context. It has all the information to build it, but you have to double check that it's from the instructions.

    I generally have a "smart" model do the planning and a dumber one do the implementation. I make use of skills. I watch it talk to itself and if I see it going off the rails I interrupt it and correct it.

    It's not the sexy 10x productivity boost people claim when doing prototyping, but it's clearly faster than doing without. I can get 3 or 4 stories done in an evening after meetings and such. If I have five minutes between meetings I can see what it has done and set it in it's next step.

    Last night I had an instance of Claude collecting information for a production support ticket while I had another one work on a ticket in one app and a third working on another app. Every time one got going I'd go over to another and see what was going on.

    I tried to create a set of 11 stories to build an entire feature in one go. It was a mess. Yeah I got 11 stories done in two days, but then code review on one would require changes and rebasing dependent stories. I've done more rebasing in the past ten days than the rest of my career. That was a mistake.

    My boss is all gung ho about me doing it that way, but I just can't seem to make it work.

  • permalink
  • fedilink
  • source
  • parent