Amazon’s ecommerce business has summoned a large group of engineers to a meeting on Tuesday for a “deep dive” into a spate of outages, including incidents tied to the use of AI coding tools.

The online retail giant said there had been a “trend of incidents” in recent months, characterized by a “high blast radius” and “Gen-AI assisted changes” among other factors, according to a briefing note for the meeting seen by the FT.

Under “contributing factors” the note included “novel GenAI usage for which best practices and safeguards are not yet fully established.”

top 94 comments

sorted by: hot top controversial new old
[–] 173 points 8 months ago (19 children)

Well that's going to make your senior developers quit.

  • source
  • hideshow 19 child comments
  • [–] 119 points 8 months ago* (4 children)

    Exactly. If you're too stupid or lazy to adequately vet what your LLM puts out yourself, it shouldn't be somebody else's job to wade through the sewage you're producing. You either shouldn't be using one or, if you can't do your job without it, you shouldn't have that job.

    —Someone who doesn't use genAI but has spent way too much time digging through LLM slop

  • source
  • parent
  • hideshow 4 child comments
  • [–] 70 points 8 months ago (2 children)

    You know what my favorite pizza topping is? Bleach.

    Dominoes REFUSES to put bleach on my pizza, so I gotta do it myself. I found out about it from AI. Now my pizza tastes great! The downside is having to go to the hospital to get a stomach pump everytime.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 19 points 8 months ago*

    I mean honestly yeah, I'm not going to waste my time with some junior developer who can't explain how the code works and how it interacts with whatever framework I'm working on. I ain't got time for that nonsense, especially when the code I deal with involves safety critical sections of code.

    Honestly if my work ever decided to allow unfettered AI code generation into my code base, I would immediately look for a new job at that point.

  • source
  • parent
  • [–] 52 points 8 months ago (2 children)

    I am not a developer, but:

    I told the owner of the company recently that, and I quote, "I will fucking kill myself if my job becomes reviewing AI output"

  • source
  • parent
  • hideshow 2 child comments
  • [–] 29 points 8 months ago* (7 children)

    It's going to make snr devs get fired, surely?

    They either refuse to sign off when boss wants them to and get fired or sign off and get fired when ai code they signed off on causes issues.

  • source
  • parent
  • hideshow 7 child comments
  • [–] 26 points 8 months ago (3 children)

    Bingo.

    Maybe not outright fired, but absolutely open them up to career limits based on what you described.

    All of Amazon's code undergoes code reviews already. Accepting a PR is already spiritually a sign off.

    This is just explicitly a threat, explicitly trying to find someone to hold accountable because you can't hold ai accountable. What are they gonna do, fire the ai? Sign here to be the fall guy. Fuck off.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 17 points 8 months ago

    Or quit/find new jobs. I suspect that's by design by Business Idiots.

    *Get rid of the most expensive engineers and the cheaper ones can just use AI to make up the difference in output. And we can make the lower engineers the fall guy when convenient and replace them at our leisure *

    The disdain bosses have for average people is astonishing.

  • source
  • parent
  • [–] 17 points 8 months ago

    it's pretty fucking stark right? these are the devs that stayed after management mandated they USE the shit in the first place, now they want the same devs to become responsible for what the shit does to their codebases.

  • source
  • parent
  • [–] 145 points 8 months ago (7 children)

    Junior and mid-level engineers will now require more senior engineers to sign off any AI-assisted changes, Treadwell added.

    So instead of getting a human to write it and AI peer reviewing it you want the most expensive per hour developers to look at stuff a human didn't write and the other engineers can't explain? Yeah, this is where the efficiency gains disappear.

    I read stuff from one of my Jr's all the time and most of it is made with AI. I don't understand most of it and neither does the Dev. He keeps saying how much he's learned from AI but peer programming with him is the pits. I try to say stuff like, "Oops! Looks like we forgot the packages." And then 10 secs of silence later, "So you can go to line 24 and type..."

  • source
  • hideshow 7 child comments
  • [–] 21 points 8 months ago

    Lol I would be your Jr, except instead of 10 seconds of silence it would be 10 seconds of me frantically clacking on the keyboard "add a block to this for these packages with proper syntax, I forgot to include it" to claude. Then I'd of course be all discombobulated and shit so I wouldn't even bother to open code, I'd just ctrl-c about 100 lines somewhere around the general area of where I think the new code block should go, then ctrl-v the whole thing into the chat box because why not the company is paying out the dick for these tokens so might as well use them.

    And two weeks later half our website crashes which results in you having to go to a meeting where management tells you to keep a closer eye on me. Which is basically what you had been already doing before AI but now you get to babysit me and claude!

  • source
  • parent
  • [–] 23 points 8 months ago* (last edited 8 months ago) (2 children)

    Just to add to this:

    • When a senior dev reviews code from a more junior dev and gives feedback the more junior person (generally) learns from it.
    • When a senior dev reviews code from an AI, the AI does not learn from it.

    So beyond the first order effects you pointed out - the using of more time from more experience and hence expensive people - there is a second order effect due of loss of improvement in the making of code which is both persistent and cumulative with time: every review and feedback of the code from a junior dev reduces forever the future need for that, whilst every review and feedback of the code from an AI has no impact at all in need for it in the future.

    Given enough time, the total time wasted in reviews and feedback for code from junior devs is limited - because they eventually learn enough not to do such mistakes - but the total time wasted in reviews and feedback for code from an AI is unlimited - because it will never improve.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 11 points 8 months ago (1 child)

    Seniors reviewing code is fine but only when, as someone else mentioned, the code writer is learning from the review. The AI doesn't learn at all and the Jr Dev probably learns very little because they didn't understand the original code. Reviewing AI code often turns into me rewriting most of it.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 5 points 8 months ago*

    Exactly.

    The best way to learn is to have done the work yourself with all the mistakes that come from not knowing certain things, having wrong expectations or forgetting to account for certain situations, and then get feedback on your mistakes, especially if those giving the feedback know enough to understand the reasons behind the mistakes of the other person.

    Another good way to learn is by looking through good quality work from somebody else, though it's much less effective.

    I suspect that getting feedback on work of "somebody" else (the AI) which isn't even especially good, yields very little learning.

    So linking back to my previous post, even though the AI process wastes a lot of time from a more senior person, not only will the AI (which did most of the implementation) not learn at all, but the junior dev that's supposed to oversee and correct the AI will learn very little thus will improve very little. Meanwhile with the process that did not involve an AI, the same senior dev time expenditure will have taught the junior dev a lot more and since that's the person doing most of the work yielded a lot more improvement next time around, reducing future expenditure of senior dev time.

  • source
  • parent
  • [–] 5 points 8 months ago

    It'll be temporary, a gut reaction to add more experienced engineers in the loop. These folks will try to codify and then push better checks/guardrails into CI/CD and tooling to save themselves time. Given how new this all is, it's almost the blind leading the blind though.

    Amazon might also have some poor system boundaries, leading to non-critical systems/code impacting critical systems. Or they just let junior devs with their AI tools run wild on critical components without adequate guardrails... also likely. :-P

  • source
  • parent
  • [–] 273 points 8 months ago

    "Everyone must use AI."

    ...

    "No! Not like that!"

  • source
  • [–] 80 points 8 months ago (15 children)

    ...

    Do the senior engineers NOT sign off on changes to systems that can take down the production servers? Even if we take out the LLM created code, this sounds like a bigger problem

  • source
  • hideshow 15 child comments
  • [–] 38 points 8 months ago (8 children)

    We may start to see people realize that "have the AI generate slop, humans will catch the mistakes" actually is different from "have humans generate robust code."

  • source
  • parent
  • hideshow 8 child comments
  • [–] 27 points 8 months ago (6 children)

    Not only that, but writing code is so much easier than understanding code you didn't write. Seems like either you need to be able to trust the AI code, or you're probably better of writing it yourself. Maybe there's some simple yet tedious stuff, but it has to be simple enough to understand and verify faster than you could write it. Or maybe run code through AI to check for bugs and check out any bugs it finds…

    I definitely have trusted AI to write miniature pointless little projects - like a little PHP page that loaded music for the current directory and showed a simple JS player in a webpage so I could share Christmas music with my family and friends. No database, no file uploading or anything. It worked decently, although not perfectly, and that's all it needed to do.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 12 points 8 months ago* (last edited 8 months ago)

    This is true not just with code, but with many types of complex outputs. Going through and fixing somebody’s horrible excel model is much worse than building a good one yourself. And if the quality is really bad, it’s also just faster to do it yourself from scratch.

  • source
  • parent
  • [–] 6 points 8 months ago (4 children)

    I've been writing a slightly larger project with frontend, bff and backend and I need to take it in small batches so that I can catch when it misunderstands or outright does a piss job of implementing something. I've been focusing a lot on getting all the unit tests I need in place which makes me feel a bunch better.

    The bigger and more complex the projects get, the harder it is for the LLM to keep stuff in context which means I'll have to improve my chunking out smaller scoped implementations or start writing code myself I think. 

    All in all I feel pretty safe with my project and pleased with the agents work but I need to increase testing further before bringing anything live.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 5 points 8 months ago (3 children)

    Security testing will be the most important.

    I've done a couple of tiny projects that I didn't feel like coding. So far, I have not been terribly impressed. Well, it is impressive that it can make something functional at all, and in one case, what it made was fine enough to use as the temporary project it was intended (sharing christmas music with friends/family - reading files from a directory and writing a javascript player to play them in a shuffled order).

    In the other case, replicating a simple text-based old DOS game with simple rules (think a space-based game around the complexity of checkers or so), it failed to think of so many things that while it did what I told it for the most part, it wasn't a playable game. It was close, and fun enough for a nostalgic moment, but I had to work with it on logic like "If two fleets of ships arrive at the same planet in the same turn, you have to see how the first battle goes. If the first battle captures the planet, the second fleet is not attacking the first fleet's ships - we won the planet at that point". Very simple concepts that sure, you'd have to think of as a programmer, but if you were telling another person about how the game should work, were things I felt another person would think about.

    I hope AI works well for you. Anywhere security it needed like database sanitation or user credentials....... I hope you test thoroughly and I hope you can tell it enough to remind it to implement things like sanitation and other safety measures. An app can certainly appear to be working, but give many many fronts for attack. That's my main worry with AI code. I worry enough on the little projects I do if I'm being secure enough myself.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 3 points 8 months ago (2 children)

    Yeah I hope I am cautious enough. I use strict db models that were man written and have type checking and sanitation. That along with unit tests that cover everything I've been able to think of that can go right or wrong combined with the classic "obscurity===security" motto.

    Of course there are always vectors one hasn't thought of, but that goes for man made projects as well. If I decide to bring it live and scale up I'll probably order a pen test.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 28 points 8 months ago (4 children)

    I guarantee there's so much pressure on those engineers to deliver code that they rubber stamp a ton of it with the intention of "fixing it later"

    Source: I've worked in software for 20+ years and know a lot of folks working for and who have worked for Amazon

  • source
  • parent
  • hideshow 4 child comments
  • [–] 10 points 8 months ago (2 children)

    That's basically the story at all the big tech companies, from what I've heard. In my time at Facebook, I felt like the only person who actually read the merge requests that people sent me before hitting it with "LGTM"

  • source
  • parent
  • hideshow 2 child comments
  • [–] 4 points 8 months ago* (1 child)

    If companies are going to place increasing reliance on review due to having lower-quality submissions, then they should probably evaluate employees weighting review quality (say, oh, rate of bugs subsequently discovered in reviewed commits or something like that).

  • source
  • parent
  • hideshow 1 child comment
  • [–] 5 points 8 months ago

    When I worked there 20% of the work we had to do had to go through a senior engineer. And getting his time was like pulling teeth.

    More of the time he would just nitpick grammar in docs and then finally rubber stamp work. It was awful.

  • source
  • parent
  • [–] 9 points 8 months ago

    the way private companies work is that they require their employees to produce more than is reasonable given the work quality that is expected.

    when this discrepancy is pointed out, it's handwaved away. when the discrepancy results in problems, as it most obviously will, somebody is found to place the blame on.

    it's not the developer's faults. it's a management decision.

    source: I'm talking out of my ass I'm just a salty employee who is seeing this happen at their own workplace when it didn't used to, at least not to this level

  • source
  • parent
  • [–] 64 points 8 months ago*

    Yes, so now when there's a success, it gets attributed to AI. When there's an outage, that's the fault of humans not reviewing correctly. These senior engineers will get fucked in all scenarios.

  • source
  • [–] 55 points 8 months ago (7 children)

    What is AI good at? Creating thousands of lines of code that look plausibly correct in seconds.

    What are humans bad at? Reviewing changes containing thousands of lines of plausibly correct code.

    This is a great way to force senior devs to take the blame for things. But, if they actually want to avoid outages rather than just assign blame to them, they'll need to submit small, efficient changes that the submitter understands and can explain clearly. Wouldn't it be simpler just to say "No AI"?

  • source
  • hideshow 7 child comments
  • [–] 19 points 8 months ago* (last edited 8 months ago)

    AI's greatest feature in the eyes of the Epstein class is the ability to shift responsibility. People will do all kinds of fucked up shit if they can shift the blame to someone else, and AI is the perfect bag holder.

    Just ask the school of little girls in Iran which were likely targets picked by AI with out of date information about it being a barracks. Why bother confirming the target with current intel from the ground when no one's going to take the blame anyway?

  • source
  • parent
  • [–] 8 points 8 months ago* (2 children)

    If you ask a writer what is Ai good for? They will say it's good for art. But never use it for writing, because it's terrible at it.

    If you ask a artist what is Ai good for? They will say it's good for writing. but never use it for art, because it's terrible at it.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 8 months ago (1 child)

    Conclusion… it’s good at neither… or am I missing your point?

  • source
  • parent
  • hideshow 1 child comment
  • [–] 6 points 8 months ago

    The output looks good to people who are poorly versed in the segment for which AI is being asked to perform, but often inefficient or fails in ways that an expert in the field would never miss.

    ---ignore this part, I'm just rambling from here on Depending on the context, you'll almost certainly get something that looks correct on first glance, especially if you're not an expert. If you're an expert, you wouldn't need to ask for such a task and, if you did to save time, you'd probably end up adjusting, correcting, or fixing several things to produce a production-ready output. I use it regularly for code because the last language I had any training in proper syntax was Fortran 77. And eventually the simple tasks I ask it to code for me work. I've asked it to do some excel calculations (I'm not an excel expert, I do a lot of mathematic manipulation in custom sheets) and some of them work, but most are either wildly convoluted or relay on obscure calls/functions rather than simply using standard logic and mathematic operations which are easy to edit and change. I've also asked it to do some graphical illustration (because I'm not a graphic artist) and it has produced nice looking illustrations with zero basis in reality - i.e. "draw me an outline of Scotland in the style you'd see on a tourist map and label, with a star, these four cities". It produced what I would expect an average American would estimate the outline of Scotland looked like and was equally as accurate with the location of the four cities (i.e. utterly incorrect).

  • source
  • parent
  • [–] 1 point 8 months ago (2 children)

    Or I suppose add extra work by walking an AI tool through making small incremental changes.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 47 points 8 months ago (7 children)

    "Huge rich company responsible for hosting like half of the fucking internet spent the last year pushing code to global-scale production without so much as a review by a senior engineer."

    That's how I read that headline.

  • source
  • hideshow 7 child comments
  • [–] 13 points 8 months ago (6 children)

    I read it as "now a senior developer will be at fault for all AI code." Do you think they will have time to review all that code properly and do their jobs.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 10 points 8 months ago* (3 children)

    One of my first big jobs at NASA was as a lead engineer on a multi-experiment platform to fly on the space shuttle. I checked all the work and compiled all the data and trotted my 27 year old self down to Johnson to present my case to the Safety Board. When I stood up to present, the head of the panel asked if I knew why I was there. I confidently told him that I was there to walk them through my evaluation of each of the payload components and show that the payload was safe to fly. He smiled. He then said "You're here because if something goes wrong on this mission, there had to be one ass to kick. Proceed."

    Everyone needs an ass to kick, and AI doesn't offer that function.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 5 points 8 months ago (2 children)

    That sounds like an almost refreshing "you're one of us now / welcome to the real thing" type of brutal honesty.

    Did it have a friendly tone and/or serve as an ice breaker before your presentation?

  • source
  • parent
  • hideshow 2 child comments
  • [–] 2 points 4 months ago (1 child)

    Sorry for the necro reply, been away for a but. It was a great icebreaker, actually. Still a little nerve-wracking given it was my first presentation outside of my directorate, but it was clear that this was a no-bullshit zone.

    If you've seen NASA-related movies like The Martian, Apollo 13, or Hidden Figures the portrayal of the enthusiast/no-bs engineers and managers is actually true. There's no tiptoeing around or "whistling through the graveyard" - you see a problem, you talk it out. And it can get animated. But I can only think of a handful of cases where things ended in an impasse because we had the (math/science) resources to make sure things were right. Obviously that doesn't mean failures don't happen, but it really fine tuned the way I approach engineering problems to eliminate as many failure modes as was practical.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 39 points 8 months ago (1 child)

    How in the glorious fuck was this not a thing from the start? In a system this big and this critical all code should be reviewed by cognizant individuals. Anyone who thought an LLM would be perfect and not need code reviews has their heads so far up their asses they can see through their pee hole.

  • source
  • hideshow 1 child comment
  • [–] 14 points 8 months ago

    If you do this, you signal the AI isn't ready for production capabilities, which limits your sales groups capability to market it. Which is in reality the actual case and AI sucks and should never be trusted.

  • source
  • parent
  • [–] 35 points 8 months ago (1 child)

    I always saw a code review like a dissertation defense. Why did you choose to implement the requirement in this way? Answers like 'I found a post on Stackoverflow' or 'the AI told me to' would only move the question back one step; why did you choose to accept this answer?
    I was a very unpopular reviewer.

  • source
  • hideshow 1 child comment
  • [–] 24 points 8 months ago (1 child)

    The way AI is being pushed onto workers on a global scale has to be the dumbest thing to ever happen in the work space. Executives are getting hysterical over something they don't even try to understand and even governments shower companies in subsidies if they do anything with AI. Of course the only result so far are mass layoffs and exploding costs for energy and hardware. All the while economies are crumbling everywhere because of course they do when mass unemployment sweeps around the globe. And again, governments everywhere are subsiding this crap with tax payer money. What's even worse than all of that is the insane environmental damage all of this causes. But I'll have to cut myself short here because I'm just getting increasingly upset here.

    I guess what I'm trying to say is: We're funding our own decline in rapid speed. Human stupidity has found a new peak in 2026 and it's not even close. I knew the way AI was advertised was completely overblown years ago but I never anticipated it would get this bad this quickly.

  • source
  • hideshow 1 child comment
  • [–] 6 points 8 months ago*

    Unsurprisingly, there's a disconnect between executives/middle managers and people actually doing the job. The first group has fallen for the 10x productivity boost ads that the AI companies were selling them, while the actual boost for developers has been minimal, if any. That's why it's being pushed hard from the top.

  • source
  • parent
  • [–] 16 points 8 months ago* (1 child)

    or hear me out, they can build it themselves so they don’t have to chase hallucinations. as a matter of fact, let’s cut the ai out of the project and leave it to summarizing emails.

  • source
  • hideshow 1 child comment
  • [–] 9 points 8 months ago

    This 1000x. You think that senior dev got to that level hoping one day all they'd have to do is evaluate randomly generated code? No! They want to create, build, design, integrate, share. Cut out the middle, useless step and get back to the work these professionals have dedicated their careers to.

  • source
  • parent
  • [–] 14 points 8 months ago (1 child)

    Aren't their names already on the commits? Or is the AI given write access to their code repository?

  • source
  • hideshow 1 child comment
  • [–] 14 points 8 months ago (1 child)

    If my job ends up being reviewing AI code spammed at me by vibe coding juniors all day, I’m joining a nunnery.

  • source
  • hideshow 1 child comment
  • [–] 13 points 8 months ago (1 child)

    They want to move fast and break things but they still want a few meat bags around to blame when things inevitable blow up in their faces.

  • source
  • hideshow 1 child comment
  • [–] 12 points 8 months ago (1 child)

    LOL, so they can blame and fire SOMEONE.

  • source
  • hideshow 1 child comment
  • [–] 1 point 8 months ago

    Amazon saves the wages of a senior dev by doing that, but then they get outages costing them the wages of that senior dev for decades. I doubt the goal is to blame senior devs. If they wanted them gone they could easily fire them.

  • source
  • parent
  • [–] 12 points 8 months ago

    Couldn't they, I don't know, just go back to people writing the code, and stop using AI to do something it clearly can't handle? Just an idea.

    I guess they've invested (thrown) so much money at this thing, they're determined to make it work. Also, I know they've gone into insanely deep debt and if it doesn't work they're going to lose an eye watering amount of money, and perhaps the bubble bursting will be the catalyst to bringing down the entire world economy.

    Oh, so yeah, they do have great incentive to make this work, but I don't see it happening. As usual, they fuck up and the rest of us pay the bill. None of the billionaires will suffer any more than loss of face over this. Even if they've broken laws, all they ever get is a small fine and a slap on the back, "Better luck, next time, ol' boy!"

  • source
  • [–] 12 points 8 months ago (2 children)

    AI is an assistant, not a replacement. It amazes me that Amazon, Microsoft, Google, and all these "tech leader" companies are going to make the same tech fuckup multiple times.

  • source
  • hideshow 2 child comments
  • [–] 18 points 8 months ago

    as a sr, I would just keep rejecting them and make AI find "reasons" why.

  • source
  • [–] 9 points 8 months ago
    [–] 9 points 8 months ago

    xD

    Guess that all-in-on-AI attitude was not such a bold and brilliant idea after all.

  • source
  • [–] 8 points 8 months ago
    [–] 7 points 8 months ago
    [–] 13 points 8 months ago

    Keep taking Ls Amazon!

  • source
  • [–] 5 points 8 months ago

    Somwthing similar just happened to McKinsey, too.

  • source
  • [–] 5 points 8 months ago
    [–] 2 points 8 months ago (1 child)

    I wonder how well that is received by their staff :D

  • source
  • hideshow 1 child comment
  • Key is that AI assisted coding does not remove accountability. AI makes it more clearer that who must own the release decision. Senior reviews only help if it checks the diff against a predefined explicit acceptance criteria, rather than becoming a rubber stamp.

    I wrote more about this here: https://techwithramesh.com/blog/why-amazon-mandated-ai-code-review

  • source
  • [–] 1 point 8 months ago

    Hahaha IT'S ALL ON YOU NOW. HAVE FUN!

    • Fuckin no one who should ever not be beat to death with a printed stack of emails
  • source
  • load more comments
    view more: next ›
    23 comments in c/fuck_ai@lemmy.world view that post