603
A single DNS race condition brought AWS to its knees (go.theregister.com)
submitted 1 year ago by to c/selfhosted@lemmy.world
103 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[+] 100 points 1 year ago (25 children)
[deleted]
  • permalink
  • fedilink
  • source
  • hideshow 25 child comments
    •
  • [–] 77 points 1 year ago (15 children)

    Consequences? For Amazon?

    lol… lmao even

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 15 child comments
    •
  • [+] 35 points 1 year ago* (13 children)
    [deleted]
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 13 child comments
    •
  • [–] 26 points 1 year ago*

    Much of this stuff is automatic - I've worked with such contracted services where uptime is guaranteed. The contracts dictate the terms and conditions for refunds, we see them on a monthly basis when uptime is missed and it's not done by a person.

    I imagine many companies have already seen refunds for outage time, and Amazon scrambled to stop the automation around this.

    They'll have little to stand on in court for something this visible and extensive, and could easily lose their shirt with fines and penalties when a big company sues over breech when they choose to not renew.

    Just cause they're big doesn't mean all their clients are small or don't have legal teams of their own.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 6 points 1 year ago (2 children)

    Most services have a clause that they are not liable for unforseen issues.. Depends how good the lawyers were when formalizing the contracts.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 4 points 1 year ago (1 child)

    Good luck arguing that a missed config counts as an 'unforeseen issue'. If they go that route, people will be all over them for not being SOC compliant wrt change control.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 1 year ago

    They can try to argue that latency issue and the stale state were an unknown / unanticipated problem. Like when half of Canadas Rogers network went down affecting most debit payment systems. Testing of routing showed it OK, realworld flip went haywire.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 4 points 1 year ago (1 child)

    99% uptime in a year gives you 3.65 days of downtime, which I think would still be within SLA (assuming nothing else happened this year). Though, once you get to 1 9 reliability (99.9%), you've got a shift and change you can be down before you breach SLA.

    If their reliability metrics are monthly, 99% gets you less than a shift of down time, so they'd be out of SLA and could probably yell to get money back.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 10 points 1 year ago (2 children)

    These contracts do not stipulate reimbursement for lost revenue. The “uptime guarantee” just gets you a partial discount or service refund for the impacted services.

    It is on the customer to architect their environment for high availability (use multiple regions or even multiple hyperscalers, depending on the uptime need).

    Source: I work at an enterprise that is bound by one of these agreements (although not with AWS).

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 9 points 1 year ago

    SLA contracts can have a plethora of stipulations, including fines and damages for missing SLO. It really depends on how big and important the customer is. For example, you can imagine government contracts probably include hefty fines for causing downtime or data loss, although I am not involved with or familiar with public sector/ government contracts or their terms.

    You can imagine that a customer that is big enough to contract a cloud provider to build new locations and install a bunch of new hardware just for them, would also be big enough to leverage contract terms that include fines and compensation for extended downtime or missing SLO.

    I work at a data center for a major cloud provider, also not AWS

  • permalink
  • fedilink
  • source
  • parent
  • [–] 3 points 1 year ago

    Oh yea, other companies will sue them, and when amazon completely fails they will be bailed out with consumers' tax money. Or did we already forget that's what happens?

  • permalink
  • fedilink
  • source
  • parent
  • [–] 7 points 1 year ago (2 children)

    Amazon has more money than most countries. They can outlast any company in court, or just ban you from their services in the future.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 10 points 1 year ago

    Depends on who we're talking about. Companies like finance orgs are all about legal contracts and would be able to hold their feet to the fire.

    You don't want to go to court against a finance company or any very large org where contract law is their bread and butter (basically any large/multinational corp).

    Amazon's not hosting just small operations.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 14 points 1 year ago (5 children)

    Was it proven that AI wa the cause?

    In not saying it wasn't, just that if it really was, I'd like a source for that claim

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 18 points 1 year ago

    Mistakes happen with or without AI

    The problem is that the current internet is structured in a way that creates high risk systems that can cause a massive outage. We went from having thousands of independent companies to a handful of massive ones. A mistake by a single company shouldn't be able to black out half the internet.

  • permalink
  • fedilink
  • source
  • parent