you are viewing a single comment's thread
view the rest of the comments
[–] 8 points 1 day ago (2 children)

I'll start by saying I'm ethically against LLMs, but I see this as not an LLM thing but a corporate thing.

A large part of those unfound security flaws is companies really don't prioritize revisiting old code. They don't consider refactoring for stability a worthwhile endeavor at all. So much so that I've seen managers insist the proof of concept code get shipped. LLMs have kind of helped raise awareness of that, but not really because it just gets called a security flaw. If companies didn't chase maximum profit I guarantee a large chunk of what we're seeing with people patching right and left wouldn't be a thing.

LLMs have only less to this vulnerability avalanche because people are firing the things off left and right on old code. The solution isn't to keep doing that, it's to shift culturally to not shipping until things are properly developed.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 4 points 1 day ago

    Absolutely agree. I used to be a Mac guy and I remember when Mac OS X (as it was called at the time) 10.5, Leopard came out. It had a lot of problems and Apple acknowledged it by following up with Mac OS X 10.6, Snow Leopard. The pitch was that they weren't really going to ship new features but rather to refine 10.5 and make it suck less. I think it was pretty widely recognized as a successful effort. It's boring stuff for the tech press and media at large, but Jobs was able to sell it. I wish more companies would do this, but it doesn't make line go up in the same way "...Now, with AI✨™️" does

  • source
  • parent
  • [–] 2 points 9 hours ago* (last edited 9 hours ago)

    The statistical distribution of errors in LLM produced code is Random Uniform, unlike with people-produced code were bugs tend to be more around certain areas because people have common sense so certain things are "obviously wrong" for them so they don't make mistakes there and end up doing more mistakes in complex logic and interaction between multiple parts.

    This means that LLMs will makes really basic "nobody could possibly get this wrong" bugs right alongside incredibly complex ones.

    So on top of the effects you describe from the way corporate environments work when it comes to IT security, you now have something that will randomly introduce some of the most basic security mistakes imaginable (for example pushing access credentials plain text into Github).

    Then to add insult to injury, in a software development process where vibe-coding was introduced to reduce manpower costs (meaning that now there isn't enough manpower for humans to get to know and understand the codebase and do manual bug tracking and fixing) bugs are fixed by re-running the code generation, which doesn't actually reduces the number of bugs, just replaces old bugs with new bugs (as the probability of introducing bugs remains the same across code-generation runs).

    Further, people actually produce fewer bugs as time goes on, both because of their increasing familiarity and understanding of the code base and because they themselves first learn to code better and eventually even start learning to design software in ways that reduce the probability of mistakes, whilst the AI doesn't learn and the best you can do to try and reduce bugs is to make sure there are certain instructions always in its context and hope for the best.

    In summary, even on a "couldn't care less about IT Security" corporate environment LLMs increase the security flaws in the code base, never actually get better at doing less of it and, worse, are just as likely to produced stupidly basic and easy to exploit security flaws as they are complex and hard to exploit ones.

  • source
  • parent