116
Linux Foundation Estimates 25% Of New Pull Requests To Be Low-Quality AI Slop (www.phoronix.com)
submitted 3 weeks ago by to c/linux@lemmy.ml
15 comments fedilink hide all child comments
 

Well, well, well...

The Linux Foundation published a new study today around the impact of generative AI on open-source software development. One key takeaway is that ~25% of new open-source software pull requests amount to AI slop.

The new Linux Foundation research study found that GenAI has the potential of cutting open-source coding time in half for increased productivity. They also found 59% of project maintainers found growing GenAI-based contributions. The 50% productivity boost was found to be across all types of open-source software development tasks. 71% of OSS maintainers agreed that GenAI reduces barriers to entry.

Very concerning though, their research found that 25% of incoming pull requests are estimated to be low-quality AI slop across open-source projects. Presumably that number will, sadly, continue to rise as more newcomers and users embrace AI/LLMs for working on patches on their behalf.

Those curious can find the Linux Foundation Research report in full via LinuxFoundation.org.

you are viewing a single comment's thread
view the rest of the comments
[–] -5 points 3 weeks ago* (8 children)

Productivity gains are substantial but vary by task complexity: GenAI accelerates OSS development in more ways than simply writing code faster. Respondents estimate average time reductions across all six OSS development activities measured, including proposal discussions (19%) and code authoring (52%). For high-complexity tasks, however, 21% of respondents report taking more time when using GenAI.

You lose some you also win some though.

Everything still goes through human review though regardless.

It just shifts the time spent from writing code and other things to rewiting code and reviews most likely

If they've never received a patch potentially from someone too, it's going to be more likely to be bad code too that can be reviewed fairly quickly

  • permalink
  • fedilink
  • source
  • hideshow 8 child comments
    •
  • [–] 18 points 3 weeks ago (7 children)

    This is just a survey though, people are notoriously bad at estimating their own speed.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 7 child comments
    •
  • [–] 11 points 3 weeks ago (4 children)

    Several MIT studies have shown that while people feel more productive they are basically actually equal because of overhead. They have to spend more time managing the LLMs and designing prompts and reviewing output and it basically is a wash.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 1 point 3 weeks ago (1 child)

    Well this study finds:

    For both contributors and maintainers, GenAI boosts productivity by up to 50% across all types of OSS development tasks.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 3 weeks ago* (last edited 3 weeks ago)

    Except this study is not peer reviewed from what I've seen and gathered data in very unscientific ways. There's no validation of answers and allowed users to just guess how much they feel they're being more productive. Like go read the questions, it's a person guessing at time saved. This study holds zero importance in the discussion in my mind because, as said, other MIT studies have shown people are actually incredibly bad at knowing how much time spent changed pre vs post LLM usage.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 3 weeks ago (1 child)

    One interesting thing I have noticed is that there seems to be a few people pointing to "studies". Are you a developer?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 5 points 3 weeks ago* (last edited 3 weeks ago)

    Yes, I'm a developer. And people point to studies because the scientific method and proper data gathering procedures have shown in the long run to be much more reliable and accurate than vague feelings.

  • permalink
  • fedilink
  • source
  • parent