all 8 comments

sorted by: hot top controversial new old
[–] 30 points 1 year ago

Well done. More and more companies are deploying LLM-written code in production environments. Might as well be honest about the results so we can learn what does and doesn't work.

  • source
  • [–] 16 points 1 year ago (1 child)

    It's obvious that the LLM didn't understand the code at all. It chose to refactor the way it did because of a silly comment.

  • source
  • hideshow 1 child comment
  • [–] [S] 13 points 1 year ago (1 child)

    Why are we using tools that can't parse the comment and code via syntax for refactoring?

  • source
  • hideshow 1 child comment
  • [–] 23 points 1 year ago

    The first problem is they're letting AI touch their code.

    The second problem is they're relying on a human to pick up changes in moved code while using git's built-in diff tools. There's a whole bunch of studies that show how git's diff algorithms are terrible, and how swapping to newer diff algos improves things considerably.

    TL;DR on the studies:

    • Only supporting add/remove/move operations is really bad.
    • Adding syntax awareness to understand if differences in indentation should be brought to a reviewer's attention, improves code and makes code reviews more accurate. (But this is hard because it's language dependent)
    • Adding extra operations (indent/deindent/move/rename-symbol/comment/un-comment/etc...) makes code review easier, faster and more accurate. (But again, most of this requires syntax awareness.

    There's also a bunch of alternative diff algos you can use, but the best ones are paid, and the free ones have fewer features. See:

  • source
  • parent
  • [–] 51 points 1 year ago* (2 children)

    I gasped when I saw this:

    A bit of discussion indicated that the trigger for the CPU spikes both times was our CEO logging in. We re-deployed to get a clean start, permanently banned him from the service, and moved on.

    This is like finding a live grenade under your bed and putting it under the rug.

    They found a way to reproduce a system killing bug, and instead of taking the time to understand it, they threw away their test case.

  • source
  • hideshow 2 child comments
  • [–] 38 points 1 year ago

    They contained the impact. Root causing or “understanding” should come after impact mitigation. If needed find a safe way to reproduce the bug without customer impact.

    We reverted the refactoring, deployed, un-banned the CEO, and set about analysis.

  • source
  • parent