390
AI slows down some experienced software developers, study finds (www.reuters.com)
submitted 1 year ago by to c/technology@lemmy.world
122 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 134 points 1 year ago (53 children)

Experienced software developer, here. "AI" is useful to me in some contexts. Specifically when I want to scaffold out a completely new application (so I'm not worried about clobbering existing code) and I don't want to do it by hand, it saves me time.

And... that's about it. It sucks at code review, and will break shit in your repo if you let it.

  • permalink
  • fedilink
  • source
  • hideshow 53 child comments
    •
  • [–] 1 point 1 year ago (3 children)

    Same. I also like it for basic research and helping with syntax for obscure SQL queries, but coding hasn't worked very well. One of my less technical coworkers tried to vibe code something and it didn't work well. Maybe it would do okay on something routine, but generally speaking it would probably be better to use a library for that anyway.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 4 points 1 year ago (2 children)

    I actively hate the term "vibe coding." The fact is, while using an LLM for certain tasks is helpful, trying to build out an entire, production-ready application just by prompts is a huge waste of time and is guaranteed to produce garbage code.

    At some point, people like your coworker are going to have to look at the code and work on it, and if they don't know what they're doing, they'll fail.

    I commend them for giving it a shot, but I also commend them for recognizing it wasn't working.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 1 year ago (1 child)

    I think the term pretty accurately describes what is going on: they don't know how to code, but they do know what correct output for a given input looks like, so they iterate with the LLM until they get what they want. The coding here is based on vibes (does the output feel correct?) instead of logic.

    I don't think there's any problem with the term, the problem is with what's going on.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 18 points 1 year ago (6 children)

    I have limited AI experience, but so far that's what it means to me as well: helpful in very limited circumstances.

    Mostly, I find it useful for "speaking new languages" - if I try to use AI to "help" with the stuff I have been doing daily for the past 20 years? Yeah, it's just slowing me down.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] 6 points 1 year ago* (last edited 1 year ago) (4 children)

    I like the saying that LLMs are “good” at stuff you don’t know. That’s about it.

    When you know the subject it stops being much useful because you’ll already know the very obvious stuff that LLM could help you.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [+] 12 points 1 year ago (2 children)
    [removed by mod]
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 1 year ago (1 child)

    I...think that's their point. The only reason it seems good is because you're bad and can't spot that is bad, too.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 1 year ago

    To be fair, when you're in Gambukistan and you don't even know what languages are spoken, a smart phone can bail you out and get you communicating basic needs much faster and better than waving your hands and speaking English LOUDLY AND S L O W L Y . A good human translator, you can trust, should be better - depending on their grasp of English, but there's another point... who do you choose to pick your hotel for you? Google, or a local kid who spotted you from across the street and ran over to "help you out"? That's a tossup, both are out to make a profit out of you, but which one is likely to hurt you more?

  • permalink
  • fedilink
  • source
  • parent
  • [–] 4 points 1 year ago

    FreedomAdvocate is right, IMO the best use case of ai is things you have an understanding of, but need some assistance. You need to understand enough to catch atleast impactful errors by the llm

  • permalink
  • fedilink
  • source
  • parent
    • load more comments (1 reply)
  • [–] 26 points 1 year ago (4 children)

    Not a developer per se (mostly virtualization, architecture, and hardware) but AI can get me to 80-90% of a script in no time. The last 10% takes a while but that was going to take a while regardless. So the time savings on that first 90% is awesome. Although it does send me down a really bad path at times. Being experienced enough to know that is very helpful in that I just start over.

    In my opinion AI shouldn’t replace coders but it can definitely enhance them if used properly. It’s a tool like everything. I can put a screw in with a hammer but I probably shouldn’t.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 13 points 1 year ago (3 children)

    Like I said, I do find it useful at times. But not only shouldn't it replace coders, it fundamentally can't. At least, not without a fundamental rearchitecturing of how they work.

    The reason it goes down a "really bad path" is that it's basically glorified autocomplete. It doesn't know anything.

    On top of that, spoken and written language are very imprecise, and there's no way for an LLM to derive what you really wanted from context clues such as your tone of voice.

    Take the phrase "fruit flies like a banana." Am I saying that a piece of fruit might fly in a manner akin to how another piece of fruit, a banana, flies if thrown? Or am I saying that the insect called the fruit fly might like to consume a banana?

    It's a humorous line, but my point is serious: We unintentionally speak in ambiguous ways like that all the time. And while we've got brains that can interpret unspoken signals to parse intended meaning from a word or phrase, LLMs don't.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] a comment that isn't shown here: deleted, from someone you've blocked, or on another page. view the thread from it
  • [–] 6 points 1 year ago* (2 children)

    That's still not actually knowing anything. It's just temporarily adding more context to its model.

    And it's always very temporary. I have a yarn project I'm working on right now, and I used Copilot in VS Code in agent mode to scaffold it as an experiment. One of the refinements I included in the prompt file to build it is reminders throughout for things it wouldn't need reminding of if it actually "knew" the repo.

    • I had to constantly remind it that it's a yarn project, otherwise it would inevitably start trying to use NPM as it progressed through the prompt.
    • For some reason, when it's in agent mode and it makes a mistake, it wants to delete files it has fucked up, which always requires human intervention, so I peppered the prompt with reminders not to do that, but to blank the file out and start over in it.
    • The frontend of the project uses TailwindCSS. It could not remember not to keep trying to downgrade its configuration to an earlier version instead of using the current one, so I wrote the entire configuration for it by hand and inserted it into the prompt file. If I let it try to build the configuration itself, it would inevitably fuck it up and then say something completely false, like, "The version of TailwindCSS we're using is still in beta, let me try downgrading to the previous version."

    I'm not saying it wasn't helpful. It probably cut 20% off the time it would have taken me to scaffold out the app myself, which is significant. But it certainly couldn't keep track of the context provided by the repo, even though it was creating that context itself.

    Working with Copilot is like working with a very talented and fast junior developer whose methamphetamine addiction has been getting the better of it lately, and who has early onset dementia or a brain injury that destroyed their short-term memory.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 1 year ago

    From the article: "Even after completing the tasks with AI, the developers believed that they had decreased task times by 20%. But the study found that using AI did the opposite: it increased task completion time by 19%."

    I'm not saying you didn't save time, but it's remarkable that the research shows that this perception can be false.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 4 points 1 year ago* (last edited 1 year ago)

    On that last note, important thing they left out here being general news reporting tech stuff is that this was specifically bug fixing tasks. It can typically only provide the broadest of advice on that, and it’s largely incapable of tackling problems holistically when you often need to be thinking big picture while tackling a bug.

    Interesting that the AI devs thought they were being quicker though.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 1 year ago

    Sometimes I get an LLM to review a patch series before I send it as a quick once over. I would estimate about 50% of the suggestions are useful and about 10% are based on "misunderstanding". Last week it was suggesting a spelling fix I'd already made because it didn't understand the - in the diff meant I'd changed the line already.

  • permalink
  • fedilink
  • source
  • parent
  • [+] -10 points 1 year ago (33 children)

    Exactly what you would expect from a junior engineer.

    Let them run unsupervised and you have a mess to clean up. Guide them with context and you’ve got a second set of capable hands.

    Something something craftsmen don’t blame their tools

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 33 child comments
    •
  • [+] 62 points 1 year ago (25 children)
    [deleted]
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 25 child comments
    •
  • [–] -4 points 1 year ago (10 children)

    Yeah but a Claude/Cursor/whatever subscription costs $20/month and a junior engineer costs real money. Are the tools 400 times less useful than a junior engineer? I’m not so sure…

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 10 child comments
    •
  • [–] 4 points 1 year ago (3 children)

    This line of thought is short sighted. Your senior engineers will eventually retire or leave the company. If everyone replaces junior engineers with ai, then there will be nobody with the experience to fill those empty seats. Then you end up with no junior engineers and no senior engineers, so who is wrangling the ai?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 3 points 1 year ago (2 children)

    This isn’t black and white. There will always be some junior hires. No one is saying replace ALL of them. But hiring 1 junior engineer instead of 3? Maybe…and that’s already happening to some degree.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 4 points 1 year ago (1 child)

    And when the current senior programmers retire the field of juniors that are coming to replace them will be much smaller.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 3 points 1 year ago

    Not that I agree, but if you believe that the LLMs will continuously improve, then in 5-10 years you may only need 1/3rd the seniors, to oversee and prompt. Again, that's what these CEOs are relying on.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 5 points 1 year ago (4 children)

    Even at $100/month you’re comparing to a > $10k/month junior. 1% of the cost for certainly > 1% functionality of a junior.

    You can see why companies are tripping over themselves to push this new modality.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 2 points 1 year ago (3 children)

    I was just ballparking the salary. Say it’s only 100x. Does the argument change? It’s a lot more money to pay for a real person.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
    • load more comments (1 reply)
  • [–] -4 points 1 year ago (8 children)

    AI tools are actually improving at a rate faster than most junior engineers I have worked with, and about 30% of junior engineers I have worked with never really "graduated" to a level that I would trust them to do anything independently, even after 5 years in the job. Those engineers "find their niche" doing something other than engineering with their engineering job titles, and that's great, but don't ever trust them to build you a bridge or whatever it is they seem to have been hired to do.

    Now, as for AI, it's currently as good or "better" than about 40% of brand-new fresh from the BS program software engineers I have worked with. A year ago that number probably would have been 20%. So far it's improving relatively quickly. The question is: will it plateau, or will it improve exponentially?

    Many things in tech seem to have an exponential improvement phase, followed by a plateau. CPU clock speed is a good example of that. Storage density/cost is one that doesn't seem to have hit a plateau yet. Software quality/power is much harder to gauge, but it definitely is still growing more powerful / capable even as it struggles with bloat and vulnerabilities.

    The question I have is: will AI continue to write "human compatible" software, or is it going to start writing code that only AI understands, but people rely on anyway? After all, the code that humans write is incomprehensible to 90%+ of the humans that use it.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 8 child comments
    •
  • [–] 2 points 1 year ago* (5 children)

    I’m seeing exactly the opposite. It used to be the junior engineers understood they had a lot to learn. However with AI they confidently try entirely wrong changes. They don’t understand how to tell when the ai goes down the wrong path, don’t know how to fix it, and it takes me longer to fix.

    So far ai overall creates more mess faster.

    Don’t get me wrong, it can be a useful tool you have to think of it like autocomplete or internet search. Just like those tools it provides results but the human needs judgement and needs to figure out how to apply the appropriate results.

    My company wants metrics on how much time we’re saving with ai, but

    • I have to spend more time helping the junior guys out of the holes dug by ai, making it net negative
    • it’s just another tool. There’s not really a defined task or set time. If you had to answer how much time autocomplete saved you, could you provide any sort of meaningful answer?
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 2 points 1 year ago* (4 children)

    I've always had problems with junior engineers (self included) going down bad paths, since before there was Google search - let alone AI.

    So far ai overall creates more mess faster.

    Maybe it is moving faster, maybe they do bother the senior engineers less often than they used to, but for throw-away proof of concept and similar stuff, the juniors+AI are getting better than the juniors without senior support used to be... Is that a good direction? No. When the seniors are over-tasked with "Priority 1" deadlines (nothing new) does this mean the juniors can get a little further on their own and some of them learn from their own mistakes? I think so.

    Where I started, it was actually the case that the PhD senior engineers needed help from me fresh out of school - maybe that was a rare circumstance, but the shop was trying to use cutting edge stuff that I knew more about than the seniors. Basically, everything in 1991 was cutting edge and it made the difference between getting something that worked or having nothing if you didn't use it. My mentor was expert in another field, so we were complimentary that way.

    My company (now) wants metrics on a lot of things, but they also understand how meaningless those metrics can be.

    I have to spend more time helping the junior guys out of the holes dug by ai, making it net negative

    https://clip.cafe/monsters-inc-2001/all-right-mr-bile-it/

    Shame. There was a time that people dug out of their own messes, I think you learn more, faster that way. Still, I agree - since 2005 I have spend a lot of time taking piles of Matlab, Fortran, Python that have been developed over years to reach critical mass - add anything else to them and they'll go BOOM - and translating those into commercially salable / maintainable / extensible Qt/C++ apps, and I don't think I ever had one "mentee" through that process who was learning how to follow in my footsteps, the organizations were always just interested in having one thing they could sell, not really a team that could build more like it in the future.

    it’s just another tool.

    Yep.

    If you had to answer how much time autocomplete saved you, could you provide any sort of meaningful answer?

    Speaking of meaningless metrics, how many people ask you for Lines Of Code counts, even today?___

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 1 point 1 year ago (3 children)

    Shame. There was a time that people dug out of their own messes, I think you learn more, faster

    Yes, that’s how we became senior guys. But when you have deadlines that you’re both on the hook for and they’re just floundering, you can only give them so much opportunity. I’ve had too many arguments with management about letting them merge and I’m not letting that ruin my code base

    Speaking of meaningless metrics, how many people ask you for Lines Of Code counts, even today?

    We have a new VP collecting metrics on everyone, including lines of code, number of merge requests, times per day using ai, days per week in the office vs at home

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 2 points 1 year ago (2 children)

    I’ve had too many arguments with management about letting them merge and I’m not letting that ruin my code base

    I guess I'm lucky, before here I always had 100% control of the code I was responsible for. Here (last 12 years) we have a big team, but nobody merges to master/main without a review and screwups in the section of the repository I am primarily responsible for have been rare.

    We have a new VP collecting metrics on everyone, including lines of code, number of merge requests, times per day using ai, days per week in the office vs at home

    I have been getting actively recruited - six figures+ - for multiple openings right here in town (not a huge market here, either...) this may be the time...

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 1 year ago (1 child)

    Interesting idea… we actually have a plan to go public in a couple years and I’m holding a few options, but the economy is hitting us like everyone else. I’m no longer optimistic we can reach the numbers for those options to activate

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 1 year ago

    Always keep an open mind. I stuck around in my first job until the sad and pathetic end for everyone, and when I finally did start looking the economy was worse than it had been when the writing was first on the wall.

  • permalink
  • fedilink
  • source
  • parent
  • [–] -3 points 1 year ago (4 children)

    Is “way less useful” something you can cite with a source, or is that just feelings?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [+] 9 points 1 year ago (3 children)
    [deleted]
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 18 points 1 year ago (4 children)

    The difference being junior engineers eventually grow up into senior engineers.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] a comment that isn't shown here: deleted, from someone you've blocked, or on another page. view the thread from it
  • [–] 9 points 1 year ago (2 children)

    No, but that's the only way you get senior engineers!

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
    • load more comments (1 reply)
  • [–] 10 points 1 year ago (1 child)

    Exactly what you would expect from a junior engineer.

    Except junior engineers become seniors. If you don't understand this ... are you HR?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •