you are viewing a single comment's thread
view the rest of the comments
[–] 76 points 3 weeks ago* (last edited 3 weeks ago) (61 children)

Lol, I could kind of understand (not really support) his use of AI. While I doubt it's something you can do for long without starting to make compromises, people who were a really good programmers before will probably use it responsibly for a while, until their skills inevitably erode and (more importantly) their willpower slowly runs out.

But this argument made me loose all my respect for him. It's so utterly stupid. I hate it when i have to check my assembly code line by line, because it's not guaranteed that the compiler will compile the same code I've written deterministically, and basically just randomizes what it outputs. Or when my editor does not save what I typed, and instead decides based on a random language statistics what to actually save, so I have to double-check my text word-by-word every time.

Oh, wait. No, because the tools are deterministic, and that doesn't happen. But of course, adding randomness into your workflow, where you have to focus and review everything line-by-line, is the same as using deterministic tools, lol.

  • source
  • hideshow 61 child comments
  • [–] a comment that isn't shown here: deleted, from someone you've blocked, or on another page. view the thread from it
  • [–] 8 points 2 weeks ago (6 children)

    Not parent commenter, but no, not to the same degree.

    The level of trust I have in a source greatly changes how I review it. If I know that I'm reviewing code from a competent developer, I'm going to see a for loop and look for why is this here, how does it play into the larger objective and if I'm happy with it, I move on. But if I don't trust the developer as much, I'm looking at far more like what data type is being used in the indexing, what's the break condition, can go off the rails into random memory.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 1 point 2 weeks ago* (last edited 2 weeks ago) (5 children)

    Competent developers make mistakes too. I think it's worth at least skimming every line in a PR. Things like indexing out of bounds should probably be handled by testing or using a memory safe language.

    I don't think code review is the place where you should be trying to catch bugs in the minutia of the code. You should be assessing readability/maintainability, architectural decisions, risk, and testing strategy. That does involve reading all of the code, even if only by skimming.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 3 points 2 weeks ago (4 children)

    I guess I don't really count skimming as reading haha

    When looking for readability, maintainability, etc what I mainly look for are design patterns. If you're using the same boilerplate and layout I'm going to start skipping over sections

    I also mainly work on high performance embedded systems that are difficult/nearly impossible to unit test and I dont always have line coverage metrics. A lot more verification by inspection in my corner of the world

  • source
  • parent
  • hideshow 4 child comments
  • [–] 1 point 2 weeks ago (3 children)

    high performance embedded systems that are difficult/nearly impossible to unit test

    Sort of a change in topic, but I'm curious why that would be so difficult to test.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 1 point 2 weeks ago (2 children)

    There's a couple big reasons, sometimes you only have to deal with one or two, other times you're hit with all of them.

    The data source for the algorithms often comes from arbitrary memory addresses that live outside the context of the C, C++, or Rust runtime. If I want to read CPU core temp I send magic numbers to one address to configure ADC channel conversion, another magic number somewhere else to do the conversion, wait a few dozen microseconds, then read a third new address that I squint one eye, spin around 10 times, and do a handstand and I'll get a number that looks like temperature.

    If I'm running on the microcontroller, I'm essentially in production environment and I can't control what the data is. If I recompile for a different system (say x86 Linux) then I'm almost certainly going to segfault. I could abstract out the reading of the data, but then on my tiny 8MHz microcontroller that has to run it's main loop at 1000Hz, I only have a few thousand clock cycles to get everything done, calling functions, pushing/popping the stack takes a lot of time.

    Adding to the pile-on most microcontrollers have <1MB of RAM, many in the tens of kilobytes range so I don't have much of a stack to work with and I need to keep even the number of instructions down.

    Even if you do all the abstractions, you then have extra logic you have to write to simulate what is happening and subtle behaviors that are timing dependant. It frequently is extremely difficult to do that as the documentation rarely fully describes everything, so you can lull yourself into a false sense of security that you've tested the code and it's correct, but then you get stuck on a register value that never changes and a watchdog timer comes along and resets the hardware on you.

    And that's not the getting into the runtime or build environments, flashing procedures, limited flash cycles, and a million other business constraints that technically you can do it, but at that point your dev team has all taken the month off to go play Half Life 3

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 2 weeks ago (1 child)

    That definitely increases the challenge of writing tests or trusting that they are faithfully representing your production environment.

    I could abstract out the reading of the data

    I was going to suggest this, and it sounds like that ends up being expensive. It seems like you need some kind of test harness that's able to emulate your specific hardware while still allowing for testing hooks that don't impede the tight clock scheduling.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 3 points 2 weeks ago

    If unit tests are being done, some kind of emulator is usually what gets used and the scope of testing is significantly limited.

    The more common approach is to abandon unit tests all together and go for board functional tests where as many of the external interfaces are controlled and put through representative loads. The complexity of the software is rarely that high so we can be pretty confident that we're covering everything we need to care about anyways.

  • source
  • parent
  • [–] 4 points 2 weeks ago* (3 children)

    You do that inherently when you are writing the code. Every line you write has already had that focus, because you had to come up with it.

    That you should also not trust every line of the compiled assembly after you wrote the code, and check it lime by line, is an extremely stupid argument.

    Of course, you should get someone else, with a fresh/unbiased opinion to review your code too, but just the act of figuring out every line you write is in some sense a rewiev, that there is at least a plan or reason why it was written.

    AI just randomly generates code without thought or reason, and you have to check everything. Unlike assembly generated from code.

  • source
  • parent
  • hideshow 3 child comments
  • [–] a comment that isn't shown here: deleted, from someone you've blocked, or on another page. view the thread from it
  • load more comments (1 reply)
  • [–] 9 points 3 weeks ago (7 children)

    His point is that an LLM is just another tool for increasing productivity, the same as a compiler or IDE. The quality of the code is the only metric that should be judged.

  • source
  • parent
  • hideshow 7 child comments
  • [–] 11 points 3 weeks ago (4 children)

    And the person you replied to is saying it is not just another tool because tools are deterministic.

  • source
  • parent
  • hideshow 4 child comments
  • [–] -1 points 3 weeks ago (3 children)

    Except they're not? Very few tools are deterministic. Only Mathematics is, and that's only a small subset of tools.

    You're free to argue that non-deterministic tools lack value, but that's clearly false: OCR only exists because of lossy training machine learning algorithms. Or, going even more basic, Monte Carlo simulations are by far the most efficient method of testing entire categories of problems that are intractable deterministically.

    LLMs can be shitty, as a business practice, due to their unpredictability, but (hopefully obviously) that depends on how it's used.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 6 points 2 weeks ago

    I'd argue that Monte Carlo is a bad example here, and is an example of a deterministic tool.

    You use a tool to create an outcome, you're looking for an answer or a solution that you can trust.

    If you want to examine behavior of a system that has input variables across some known range, you could brute force it, try every single possible combination and wait a millennia. Or you use a monte carlo strategy, let randomness enter the simulation to build out a basic characterization. You want more detail? More confidence? Increase the number of runs. I can trust that if I increase the number of runs I gain higher fidelity, if the tool/strategy was non deterministic I couldn't do that

  • source
  • parent
  • [–] 2 points 2 weeks ago

    The point is a tool is something you wield. Your skill is needed for and improved by the use of a tool. An LLM is an automation. You don't wield it, you give directives. Your skill is unnecessary for its use and is left untrained. The worst part is that, unlike a good, deterministic automation (like cruise control), where you understand the limitations and can take over to avoid them, you can't do that with LLMs. The best you can do is carefully review the slop output.

  • source
  • parent
  • [–] -1 points 3 weeks ago (3 children)

    "Deterministic" compilers and reproducible builds are actually fairly difficult problems. (Yes I know technically the compiler is deterministic, but AI is also technically deterministic since you can set the seed).

  • source
  • parent
  • hideshow 3 child comments
  • [–] 19 points 3 weeks ago (1 child)

    It's not just about being deterministic. A compiler crucially follows clearly defined formal rules on what is a correct output given some input, which the programmer can fully rely on.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 10 points 3 weeks ago

    which the programmer can fully rely on.

    Most of the time, today. Mid 90s I was constantly checking compiler output for bugs, and finding them. Early 90s the compiler bugs were finding me, particularly in Borland Turbo C++. I turned off the linker optimizer in 1992 and couldn't turn it back on until 2008 without getting difficult to identify bugs.

  • source
  • parent
  • [–] 2 points 3 weeks ago

    There is a clear line between deterministic code and what comes out of an LLM. One of the bigger problems there is that LLMs have a rather large "context window" they operate from, and human operators don't readily "grok" how the content of that context window enhances and degrades the desirability of the outputs you get.

    I almost exclusively use LLMs to generate, review and test deterministic code, then use that code for "the real tasks" with zero LLM involvement. The output of LLMs is still unpredictable, but once they've "said" something valuable, recognizing and using that value is a deterministic activity.

  • source
  • parent
  • [–] -4 points 3 weeks ago (2 children)

    people who were a really good programmers before will probably use it responsibly for a while, until their skills inevitably erode

    Damn, dude, I had mad 6502 assembly skillz in 1985, I could even write it out by hand - pencil on paper, and those slowly eroded through use of compilers for higher level languages until my last assembly experience in 1996, today... I'd be learning everyhing specific to assembly code from scratch. I guess I'm just useless now.

    the people I respect and call intelligent are not the ones that shy away from using new and interesting tools.

    Gotta agree there. We had two clear camps of programmers before LLM agents came along, and the camp I least enjoyed reviewing the code of before (and now) of is the camp that resisted adoption of new interesting tools the most loudly.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 10 points 3 weeks ago (1 child)

    I think there is no equivalency between compilers and other tools, and LLMs. As the comment states, they are just the statistical prediction of what would come next. Compilers may sometimes give unexpected outcomes, but they are working to a defined set of rules and there is potential to see how they actually came to that outcome and address it.

    Added to this is the prevalence of people thinking that LLMs can allow them to do anything, whether or not they understand it, it's even sold like that. In my own organisation I have an operations manager thinking he can make pages for our Intranet - where does it store the data? How is access controlled? So many basic questions that an actual programmer would know to think of. Sure you can just keep asking people if they can fly a plane and maybe one will do well enough that they get you all the way there, till it comes to landing and it's too late - maybe better to just ask a pilot to do it.

    Our software engineers use it really well, they can read what it's outputting to understand if it's wrong, they know how to prompt it to avoid issues, they use it for snippets and fixes and that it won't just write a whole program. However, it's benefit to us (and the world) is nowhere near worth the monetary and environmental cost. The investors pouring in money are hoping they can be the company that people become reliant on, so they can survive the bubble and their ultimate goal of replacing real people.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 3 weeks ago

    Added to this is the prevalence of people thinking that LLMs can allow them to do anything, whether or not they understand it

    1971, Willy Wonka and the Chocolate Factory depicted a computer which could (but was refusing to for comedic effect) predict the location of the remaining golden tickets. There were, and probably still are, a shocking number of people who believe that was a realistic scene.

    I have an operations manager thinking he can make pages for our Intranet - where does it store the data? How is access controlled? So many basic questions that an actual programmer would know to think of.

    Option 1: hold his hand and educate him on what he lacks

    Option 2: let him flounder and make fun of his mistakes (the making fun part should be done anonymously if you value your job)

    Option 3: he's bright enough to ask the important questions to you or the LLM and get reasonable answers

    Option 3 has a very realistic chance of working, if 1) you have documentation for how you do these things today - even informal documentation, anything the LLM can read and extract the relevant patterns from, 2) the LLM must have access to that documentation, and 3) your manager needs the patience and perseverence to work the discovery process until it's viable and sustainable.

    Odds are, your operations manager is going to fall short on the patience and perseverence dimensions and offload the boring parts to you, after all why did he become a manager in the first place?

    maybe better to just ask a pilot to do it.

    Yeah, except you can't make nightly backups of the plane's passenger manifest. If you're not making nightly backups, you're already in catchup territory LLM or no.

    it’s benefit to us (and the world) is nowhere near worth the monetary and environmental cost.

    I suggest doing a little research into what exactly the monetary and environmental cost of YOUR usage is. People engaged in low-value usages (too many to even begin to imagine, Rule 34 needs expansion to start to cover them) are burning a lot of un-necessary tokens.

    What I have read, consistently from various sources, is that my $20/month Claude code subscription burns $11 to 12 in electricity to serve my responses. My house burns over $13 in electricity per day in the summer, mostly for the air conditioning.

  • source
  • parent
  • [–] -3 points 3 weeks ago (33 children)

    You can add deterministic testing to AI output though?

  • source
  • parent
  • hideshow 33 child comments
  • [–] 12 points 3 weeks ago (18 children)

    Wouldn't it be just... writing code yourself? It's just instead of writing what you want directly you are writing tests for it.

  • source
  • parent
  • hideshow 18 child comments
  • [–] a comment that isn't shown here: deleted, from someone you've blocked, or on another page. view the thread from it
  • Well, depends. My thousand-line pet project does not need any tests, it needs to be feature complete and never touched again.

    Bigger projects are usually ones where multiple people are contributing and having code that practically nobody wrote there is a huge problem on it's own in my opinion, even if it passes a number of tests. Good tests themselves are usually harder to write than the code they are testing, I am not sure if delegating the actual fun part of development to an LLM and doing "chores" manually is a good idea.

  • source
  • parent
  • [–] 1 point 3 weeks ago (15 children)

    instead of writing what you want directly you are writing tests for it.

    LLMs write pretty good requirements, specifications, code, and automated tests from chat sessions. You do need to keep an eye on what it's writing to make sure it's not hallucinating, but hallucinations have been reducing in practice rather dramatically across the last 12-18 months of model development. They still happen, you will always need to guard against them, but usually a quick read of what is written followed by a quick test of the product will find the worst offenders. In the end, you need to check the final product to be sure it does EXACTLY what you want, just like you always did. And, it's still a good idea (and as possible as ever) to keep the product modularized and also check every module to ensure it works as you want. How small you make those modules has always been a matter of preference / dictated by levels of concern, etc.

  • source
  • parent
  • hideshow 15 child comments
  • [–] 10 points 3 weeks ago (14 children)

    If you are generating tests with an LLM, it's not deterministic.

    Also, have you actually read anything an LLM wrote that's longer than a hundered words? It's usually immediately obvious. In the open source world things I keep seeing are exchanges like this one. And Niri is a fairly good case with it's maintainer actually reading the PRs and catching all the bullshit that an LLM spits out. Very often nowadays people will also do reviews with an LLM and this sort of stuff will just accumulate in the codebase with nobody to reject it.

    But even if it was as useful as big tech paints it to be, I'll always be against using it based purely on morals. There is an enormous number of issues in that regard, most importatnly LLMs literally kill people both directly and indirectly. Condoning the adoption of this technology is either being ignorant or misanthropic.

  • source
  • parent
  • hideshow 14 child comments
  • [–] a comment that isn't shown here: deleted, from someone you've blocked, or on another page. view the thread from it
  • [–] 8 points 3 weeks ago (8 children)

    Why the are you getting more text than you can physically review at your job? That sounds like a recipe for poor reviews. How are you sure LLMs are not missing anything if nobody actually has the time to read through all the text? Why does the text that gets to a developer contain logical mistakes in the first place? Back at my last job we had people assigned to collecting feature requests, talk them through with whoever wanted the feature implemented and produce proper specifications that developers would implement without wasting time on specifications containing garbage.

    As does nearly every activity of life, both in human society and the natural world.

    Yea, all people who drank water died. Not a very good argument. Most things don't have a mental disorder named after them. If I leave a webpage with a calculator running unattended it won't magically persuade a random visitor into commiting suicide. Webpages with LLM-powered chatbots are doing just that. You may choose to neglect it, but to me it sounds like adding lead to your gasoline. Yea, your car runs better. Are you sure it's worth it, though?

  • source
  • parent
  • hideshow 8 child comments
  • [–] a comment that isn't shown here: deleted, from someone you've blocked, or on another page. view the thread from it
  • [–] 6 points 3 weeks ago (2 children)

    I'm sorry you are working in such conditions, I hope you get better management that actually understands how much data one person can process.

    I'm not sure how your story is at all relevant to what I said.

    I don't have a driver's license, I don't live in the US, I don't need one. The lead in gasoline example was just the first thing that came to mind that fit a "technology that is profitable and useful, but kills people" description. Please pay attention to the road while driving instead of perfoming mental math exercises.

    I would also like to ask you to think about what exactly you are arguing for right now, as I'm starting to think that our understanding of the value of human life differs quite a bit.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 2 weeks ago*

    I hope you get better management that actually understands how much data one person can process.

    Cynically, management doesn't care if we read or understand the material, only that we click a button that documents (records) that we have "been trained on it" and acknowledge that.

    In reality, after decades of exposure to the same basic concepts, most people who are "trained" on this material do know - vaguely - what's in the documents, and the real function of the training is to remind you that the procedures are there and when they are applicable to what you are doing (rarely, in my job function) then you should pull the official, current, procedure from the official document repository and reference that as a work instruction while you do what it applies to. Part of why the procedures are in such bad shape is that people rarely do exercise them like that, so flaws remain un-noticed for long stretches of time.

    I hope that one day, the world can openly accept that people are not perfect executors of written, spoken, or graven in stone instructions and that that is O.K. and we can still do a good job with our flaws. Some of the procedural safeguards are implicitly acknowledging that without coming right out and acknowledging that. I suspect that day will come after I retire, if ever, and I may work for another 30 years.

  • source
  • parent
  • [–] 1 point 3 weeks ago

    Human life is more valuable than money, property... but suicide is a reality of the human condition. It's not something that has been meaningfully improved with modern psychology or medications. It's worse among the poor but significantly prevalent across the wealth spectrum. Anything that interacts with people is going to interact with suicidal people.

    Next time you are in a train station, count 100 people as they pass you - same effect. If you're in a major station, look out across the big hall full of people, estimate how many hundreds of people you are looking at, and take in the reality of what 1 in 100 actually means.

  • source
  • parent
  • [–] 1 point 2 weeks ago (3 children)

    Sounds nice. We’ve got ~500 employees at our “site” (which is now distributed across 3+ states with remote workers), of those, about 8 do software development, so developers end up doing virtually all of the translation of customer “needs” to specifications / requirements. We also have a marketing department that changes their mind about twice as fast as the product development / delivery cycle.

    you're acting like the solution here is LLMs, when the solution is just hiring more people. LLMs aren't 'solving' anything, they're just masking the problem and allowing the millionaires at the top to keep making tons of money.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 0 points 2 weeks ago (2 children)

    you’re acting like the solution here is LLMs

    No, you're imagining that's what I wrote. LLMs are a tool, not a solution. A helpful tool, very helpful in some situations. The real problem that people and LLMs struggle with is: what should we be doing in the first place? LLMs don't solve that, they do accelerate the making of a software thing when you know what you want that software thing to be. They also accelerating the finding of flaws during the process, enabling cleaning those flaws out faster.

    just masking the problem and allowing the millionaires at the top to keep making tons of money.

    That's not a problem for tools to solve, that's a societal problem that goes back to - at least - the start of agriculture.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 2 weeks ago (1 child)

    you’re acting like the solution here is LLMs

    No, you're imagining that's what I wrote. LLMs are a tool, not a solution.

    Fine, you’re acting like the tool for the solution is an LLM.

    A helpful tool, very helpful in some situations.

    X to doubt. My job is currently filled with people using LLMs and most of what I’m doing is undoing their shit our detangling whatever LLM output they paste

    The real problem that people and LLMs struggle with is: what should we be doing in the first place? LLMs don't solve that,

    Correct

    they do accelerate the making of a software thing when you know what you want that software thing to be.

    Incorrect. They accelerate making a ton of code that is impossible to maintain, update, etc. If you know what the software is, it was never writing it that was the hard part. That’s why senior+ developers have been saying for decades that the hard part of the job isn’t coding. It’s everything else.

    They also accelerating the finding of flaws during the process, enabling cleaning those flaws out faster.

    Incorrect. Like I said, half my job is reading over extremely verbose LLM bullshit from colleagues to tell them every single thing the LLM said incorrectly. That didn’t happen before LLMs. People checked their work. LLMs don’t have a mind, they cannot check their work.

    That's not a problem for tools to solve, that's a societal problem that goes back to - at least - the start of agriculture.

    I didn’t say it was a problem for tools to solve! But it sure is a problem tools can cause! And that’s exactly what’s happening here. It’s making everything so incredibly worse that there is literally no justification for it!

  • source
  • parent
  • hideshow 1 child comment
  • [–] 0 points 2 weeks ago

    My job is currently filled with people

    My jobs have been filled with people misusing tools for 40 years, particularly new tools and computer oriented tools - the current trend is just an extension...

    They accelerate making a ton of code that is impossible to maintain, update, etc.

    When used improperly, sure, see above.

    If you know what the software is, it was never writing it that was the hard part.

    I'll agree, but still say that LLM agents accelerate the setup of toolchains, the learning of new languages, and many other things which do accelerate the writing part so instead of having to carve out 2 weeks of quiet time to get a good proof of concept ready from concept to ready for others to give feedback on, that can now be done in a day or two.

    reading over extremely verbose LLM bullshit from colleagues to tell them every single thing the LLM said incorrectly. That didn’t happen before LLMs.

    Your incompetent colleagues have been given a steamshovel for their bullshit, my condolences. It's not the steamshovel's fault it can work faster than a man with a spade on a stick.

    LLMs don’t have a mind,

    Correct.

    they cannot check their work.

    Incorrect. They're actually better at finding fault than they are at creating fault-free work. But they do require direction to keep looking for faults, they'll tell you they didn't find any (true statement) but when you tell them to look again (with a clean context window), unlike people, they'll look again just as carefully as the first time - and find more faults. I went bug hunting on a nice meaty module and found and corrected over 12 rounds of faults - they were getting rather trivial toward the end, and I had to direct to fix through removal rather than addition of code whenever possible - repeatedly, but after that it was actually clean and solid.

    it sure is a problem tools can cause! And that’s exactly what’s happening here.

    Tools aren't causing the problem here, managers are causing the problem. Any excuse for rank and yank will be gleefully embraced to its fullest potential.

  • source
  • parent
  • load more comments (1 reply)
  • [–] 1 point 2 weeks ago (3 children)

    Simple questions to an LLM like: “review this procedure, is it internally self consistent” will find issues, real issues - usually minor - but undeniable when you read the review, the problem it describes is absolutely present in the document, page 14 says “X” and page 23 says “not X” and the context is basically identical - so which are we really supposed to do: “X” or “not X”? Well, that depends. Depends on what? Additional context and detail that the already too long 50 page procedure doesn’t address.

    lol and I find issues like this in LLM output of only 1 or 2 pages. LLMs can't even be consistent across paragraphs, much less hundreds of pages of context window. what an insane claim.

  • source
  • parent
  • hideshow 3 child comments
  • [–] a comment that isn't shown here: deleted, from someone you've blocked, or on another page. view the thread from it
  • [–] 1 point 2 weeks ago (1 child)

    Asking LLMs to find flaws, then verifying those flaws are real, is more powerful than just looking for flaws without the tool.

    No it isn’t. Your example is so contrived. What was happening before these documents were reviewed by LLMs? Were the lines that were incorrect causing issues? If so then a human would have found it years ago. If not then they weren’t causing problems and all the LLM has done is invent work that a human now needs to clean up. The LLM didn’t solve a problem, it created a new one!

    Now you need to pay a human to make sure that the LLM isn’t actually making things worse, which wasn’t even a job before! Your company that won’t pay a human to review the document in the first place is now needing more humans to clean up a mess that wasn’t a problem in the first place.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 0 points 2 weeks ago

    Were the lines that were incorrect causing issues?

    Mostly not because mostly people gloss over the text and say "yeah, yeah, I know what you're getting at" and do what they think the page says instead of doing what the page actually says.

    The LLM didn’t solve a problem, it created a new one!

    If you have a road with mis-painted stripes, but it never causes a problem because everybody ignores the stripes anyway... yeah, in that sense fixing the stripes is un-necessary work...

    Your company that won’t pay a human to review the document in the first place is now needing more humans to clean up a mess that wasn’t a problem in the first place.

    Oh, the company pays plenty of people to create and review the documents. They have metrics for how many documents they review per quarter, and they do review those documents, and they do find these things, and other people are paid to revise the documents again with metrics for their performance, and we all get trained on the revisions as they come out - at a remarkably regular pace when you look across a scale of tens of years - and that training accounts for about 5% of my weekly chargeable time accounting.

    The difference is: the LLMs are finding problems faster, all at once, and the "continuous improvement process" is a little bewildered about what to do with all these, valid, observations coming out so much faster than they used to. We're just starting to think about how to come to terms with that, and in the meantime the process rolls on revising documents at about the same pace as always, because if we fixed them all at once it would put a kink in the system, like a python swallowing a buffalo.

  • source
  • parent
  • load more comments (1 reply)
  • load more comments (1 reply)
  • load more comments (1 reply)
  • [–] -1 points 3 weeks ago (13 children)

    Indeed, my code has never been more rich in unit tests now that coding assistants make them so easy to add. If the coding assistant subsequently does something weird that changes program behaviour it knows immediately.

  • source
  • parent
  • hideshow 13 child comments
  • [–] 4 points 3 weeks ago (11 children)
  • [–] a comment that isn't shown here: deleted, from someone you've blocked, or on another page. view the thread from it
  • [–] 4 points 3 weeks ago (9 children)

    Are these tests written by AI? Have you reviewed the tests? Do they cover all code paths and all the possible inputs and outputs? Do the tests have requirements cited (traceability)?

  • source
  • parent
  • hideshow 9 child comments
  • [–] 1 point 3 weeks ago* (last edited 3 weeks ago) (8 children)

    Unit tests have been around for a long time. They tended to be under-utilized in the past because they're such a maintenance headache - changes / improvements in the production code that used to be a simple 5 minute thing can easily balloon to an hour or more of headache in necessary adjustments to perfectly good unit tests for "the old way" that happened to be impacted by the change.

    Unit tests are a good way of keeping under-experienced programmers from accidentally breaking old things as they work on new things. They are VERY appropriate tools for LLM agent assisted development.

  • source
  • parent
  • hideshow 8 child comments
  • [–] 6 points 3 weeks ago (7 children)

    Absolutely not. In a world where hallucinating machines are writing code, humans should be writing the tests at a bare minimum.

    AI writing code to pass tests written by AI will get people killed.

  • source
  • parent
  • hideshow 7 child comments
  • [–] 0 points 3 weeks ago (6 children)

    Boeing managed to kill plenty of people without a single bit of help from AI / LLMs in their MAX 10 software.

    Used properly, LLMs could have articulated why their approach would lead to multiple crashes.

    Used improperly, any tool is dangerous, from a chainsaw to a plastic spoon, the potential for injuries and deaths are unlimited.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 6 points 3 weeks ago (1 child)

    Boeing's executives had motivated reasoning to skimp on actual engineering and give the results the exec wanted over the objections of actual engineers.

    Sound familiar?

    Frankly, no. LLMs are so sychophantic, there's little evidence they would have flagged a warning that what he executives wanted was dangerous.

    And no, LLMs are not just some tool. Never in human history has a hammer given it's users psychosis.

    This is a completely nonsense argument.

  • source
  • parent
  • hideshow 1 child comment
  • load more comments (1 reply)
  • [–] 3 points 3 weeks ago (3 children)

    The max issue on the Boeing planes wasn't a software issue in that sense. The software actually worked as intended. Boeing just hid the fact that the software existed and made it purposefully hard to override, giving pilots quite the surprise when the plane didn't operate the way they expected, especially when the angle of attack sensor wasn't working.

  • source
  • parent
  • hideshow 3 child comments
  • [–] -1 points 3 weeks ago* (last edited 3 weeks ago) (2 children)

    There are all kinds of aspects to software development. Under-training pilots and then having software lead them to crash a plane full of passengers counts as a software development fail in my book. Also a human factors fail, marketing fail, a management fail, a quality systems fail, probably a regulatory oversight fail... it's up there with ignoring the guy telling the Space Shuttle mission control not to launch when it has been under 32F the night before, which wasn't a software fail at all, but was a broad cultural failure. Boeing's MAX-10 situation just happened to have software in the crux of the matter.

    I do software development. When I see something, I say something. I'm frequently ignored. Several times over the years, I've had the people that ignored me come back asking for help fixing the issue, after they realized that it was more important than they thought it was when they initially ignored me. Luckily, none of the people who died in those situations died (or were harmed in any significant way) because of the problem, but people dying "near the problem" brought the seriousness of it to a level where it finally got addressed.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 5 points 3 weeks ago (1 child)

    If someone demands a piece of software that completely erases all available data when a certain button is pressed, no one is told about it, someone presses it by accident and deletes a whole bunch of data, that's not a software development error. That's a management failure. The software did exactly what it was supposed to do.

    And LLMs wouldn't make anything better in that regard. An LLM is built to do as the user wishes, with much, much less pushback than a human. So whereas a human getting the demand to write that software might say, "what in the actual fuck are you smoking and can I have some of it", an LLM will typically say, "that's a very bold decision!" In fact, in the case of MCAS, the issue of safety was raised by the engineers many times, and was flat out ignored by management. An LLM is much less likely to make a fuss and much more likely to just do it.

  • source
  • parent
  • hideshow 1 child comment
  • load more comments (1 reply)
  • load more comments (1 reply)
  • load more comments (1 reply)
  • load more comments (1 reply)