1 comment

sorted by: hot top controversial new old
[–] 7 points 1 month ago

Customer data was irretrievably lost in one of three AWS availability zones in the United Arab Emirates region, specifically the mec1-az2 availability zone. Each availability zone is serviced by one or more Amazon data centers.

For those that are not regulars in enterprise cloud computing architecture, it is a known pattern that any critical data should exist in at least two availability zones (AZ). The assumption is that you should be able to lose and entire AZ and still be able to bring up your services again without significant loss of data.

But the destruction was apparently even more widespread for the data centers in the Bahrain region, because Amazon said it was unable to restore access to resources and data across all three availability zones there. “The damage to our infrastructure spanned multiple Availability Zones and exceeded what our regional and multi-AZ services are designed to withstand,” the AWS update said.

In this AWS region, even the general best practices didn't protect services/data. Usually only really important applications/services are designed to withstand multi-region failure, which is what this is for the AWS Bahrain region (me-south-1).

  • source
  • 89 comments in c/news@lemmy.world view that post
    [–] 220 points 1 month ago (3 children)

    The only way to get them to delete your data.

  • source
  • hideshow 3 child comments
  • [–] 63 points 1 month ago (2 children)

    "We don't need our own backups. The cloud takes care of everything."

  • source
  • hideshow 2 child comments
  • [–] 57 points 1 month ago (14 children)

    Anyone in IT who has ever been near a backup or had to do a DR exercise knows full well not to have a backup in the same geographic area as your primary workload. Usually it’s for natural disasters, but it very much applies to war too.

    This is just as much, if not more, on the companies that didn’t have geographicly dispersed backups.

  • source
  • hideshow 14 child comments
  • [–] 9 points 1 month ago (2 children)

    A not insignificant part of my first azure cert covered different redundancy options and pricing. If the data was that critical, the tenants would/should have mirrored it to another region.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 2 points 1 month ago

    Depending on the data requirements they may not have been legally allowed to move it to another region. Medical information or banking data usually has an on shore requirement. Sometimes funds have issues with storing Information overseas esspesully if there is concerns about other governments gaining access.

  • source
  • parent
  • [–] 6 points 1 month ago (9 children)

    I think pretty much every software dev has used AWS at some time, and when you set up new services, it specifically warns you to use multiple geographic areas for your services. It doesn't say "war" or something, only that it's good for redundancy, IIRC.

    So anyway, yes, I follow instructions, but I'm always surprised that you have software people and systems admins who can't even follow basic instructions. I don't know what percentage of people it is, but it feels like over 50% who can't follow even the simplest instructions.

  • source
  • parent
  • hideshow 9 child comments
  • [–] 6 points 1 month ago (8 children)

    I'm not surprised at all. It costs more to use more regions, both due to literal cost and in time to set it up.

  • source
  • parent
  • hideshow 8 child comments
  • [–] 2 points 1 month ago (7 children)

    The extra time to set up in more regions is pretty trivial, at least in my experience. It's either an option when setting up the service, or it's something that you need to cover in any case while documenting how to set up the service. But I suspect the devs I'm complaining don't read basic instructions also don't write basic instructions.

    As for the monetary cost, that's a business decision. The devs I am complaining about are not making business decisions. They're just ignoring basic instructions.

  • source
  • parent
  • hideshow 7 child comments
  • [–] 0 points 1 month ago (6 children)

    I haven't used aws for ages but "an option when setting up the service" does not encompass my recollection of it: you can set up (say) S3 in multiple regions, but that doesn't mean your application can actually make use of multiple regions. It doesn't mean your vlans actually enable anything to talk between the regions.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 1 point 1 month ago (5 children)

    I haven't used it for a while, either, but it was just an option sitting there when using the AWS UI to set up RDS, IIRC.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 0 points 1 month ago (4 children)

    Right, but how is your application going to handle it if you enable it?

  • source
  • parent
  • hideshow 4 child comments
  • [–] 1 point 1 month ago (3 children)

    You have to set up your application to use the service, regardless, so I would say that you just have to follow the instructions.

    I am not talking about fixing after a service has been set up for only one region, but about following the instructions when first setting it up.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 0 points 1 month ago (2 children)

    Yes, but the application setup is more complicated if it can talk to more than one region.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 1 month ago (1 child)

    The application setup to have the ability to talk to more than one region is the normal setup that you get when you follow instructions. You're not saving time to set it up a different way. You're simply creating technical debt for the person who has to come around and set it up in the standard way in the future. And it probably takes you longer to set up services locked into a single region since for the other way, you just have to follow instructions.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 36 points 1 month ago (1 child)

    War damage to data centers exceeded what AWS services are designed to withstand.

    Well good thing it isn't a war then

  • source
  • hideshow 1 child comment
  • [–] 2 points 1 month ago

    They are intentionally designed to withstand a little bit of "war damage"?

    Like an Amazon data warehouse construction guy needs to specifically add a little bit of extra cement or some shit to meet a war damage protection quota?

  • source
  • parent
  • [–] 34 points 1 month ago (4 children)

    Speaking as a member of a company who has just recently realized that we can have products in regions other than us-east-1, this is not surprising. If UE1 went down the economy would collapse overnight

  • source
  • hideshow 4 child comments
  • [–] 11 points 1 month ago (3 children)

    I've been working on Disaster Recovery for my company and the amount of sleep I'm losing on this is insane in general.

    Our devs just write their own IaC stuff so like, they can provision their own databases and shit without us knowing with no backups or customer keys and stuff to back up properly.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 12 points 1 month ago

    It's incredibly hard to reign that behavior in once it's entrenched as well.

    You basically gotta take away privileges from everyone and it becomes a huge fight if you don't have organizational support. Usually you have to wait for something to explode spectacularly for that to happen.

  • source
  • parent
  • [–] 7 points 1 month ago

    Fellow SRE here.

    The number of times I've heard "we have replication" followed by "we've tested it...on paper" make me cry.

    Trying to convince a CEO that the only way to know for sure that their backups work is to do an actual disaster test is harder than trying to convince my parents that I don't need to eat more.

  • source
  • parent
  • [–] 5 points 1 month ago*

    Our devs just write their own IaC stuff so like, they can provision their own databases and shit without us knowing with no backups or customer keys and stuff to back up properly.

    That'd give me nightmares too on top of the DR, hopefully at the minimum they can't spin up resources in anything but a dev/test environment.

    Best of luck to you, governance is super important, if it makes you feel any better you're definitely not alone.

  • source
  • parent
  • [–] 58 points 1 month ago (6 children)

    Never thought I'd be a cheerleader for Iran against my own country. But please, Iran, keep making America better!

  • source
  • hideshow 6 child comments
  • [–] 5 points 1 month ago* (5 children)

    They didn't attack US data centers... This hurt the companies in the area. Amazon almost certainly had it all covered by insurance.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 52 points 1 month ago (8 children)

    At this point, I wouldn't be surprised if a bombing on US territory was cheered if it was an AWS data center (or Meta, xAI, etc). Hell, Iran could probably even find US citizens for sabotage at this point if they just ask, lol.

  • source
  • hideshow 8 child comments
  • [–] 4 points 1 month ago (3 children)

    I expect Iran already hires Americans for unorganized sabotage, I know both Iran and Russia utilises Telegram to hire teenagers "torpedoes" as they're called among mobsters.

    It's a fully gangster technique, but it's effective. One example would be when the Israeli embassy in Copenhagen had two kids attempt to throw a grenade their way in 2024 - It's proven to be related to Foxtrot (the largest Swedish organised crime syndicate), which Iran has influence on through one of the leaders living in Iran

  • source
  • parent
  • hideshow 3 child comments
  • [–] 2 points 1 month ago (2 children)

    You have to be a special kind of brainwashed believe that story about Iran hiring children

  • source
  • parent
  • hideshow 2 child comments
  • [–] -1 points 1 month ago (3 children)

    Nobody will give their life for this, you must use drones for sabotage

  • source
  • parent
  • hideshow 3 child comments
  • [–] 8 points 1 month ago (2 children)

    Just give any American $1000 and they will be happy to do that.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 21 points 1 month ago

    Fuck yes, keep doing that please.

  • source
  • [–] 16 points 1 month ago

    That's one way to delete your Amazon account I suppose.

  • source
  • [–] 26 points 1 month ago (5 children)

    Do they have a Go Fund Me or something I can donate to or something to keep them rolling?

  • source
  • hideshow 5 child comments
  • [–] 4 points 1 month ago (4 children)

    Neither Amazon or Islamic Republic of Iran need your money, friend. Buy yourself a nice hat in stead.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 13 points 1 month ago (3 children)

    Good. Fuck Amazon, sucks for the customers. Also what kind of company like this doesn't have backups?

  • source
  • hideshow 3 child comments
  • [–] 5 points 1 month ago (2 children)

    Presumably, the kind that likes to cut corners and not do things properly, safe in the knowledge that they'll face no consequences. Or if they somehow do then the government will just bail them out anyway.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 12 points 1 month ago
    [–] 12 points 1 month ago
    [–] 11 points 1 month ago

    AWS makes so much cash that they won't even leave the region

  • source
  • [–] 10 points 1 month ago

    Making me wanna Inshallah up

  • source
  • [–] 10 points 1 month ago

    Oh no, not amazons marketing data!

  • source
  • [–] 10 points 1 month ago

    Someone needs to shine a laser at bezos' noggin for a better permanent loss.

  • source
  • [–] 18 points 1 month ago

    Iran out here doing everyone a favor

  • source
  • [–] 8 points 1 month ago

    So like... the opposite of a data breach, the headlines I am used to reading about "customer data"? Hmmm... Ok.

  • source
  • [–] 8 points 1 month ago

    No great loss.

  • source
  • [–] 8 points 1 month ago

    3,2,1 applies to cloud regions, so everyone who lost their data permanently deserved it.

  • source
  • [–] 7 points 1 month ago

    Finally! Data getting "lost" instead of "stolen". /s

  • source
  • [–] 6 points 1 month ago

    Hilarious, YES IRAN!

  • source
  • [–] 6 points 1 month ago

    Oh no! Anyway…

  • source
  • [–] 6 points 1 month ago
    [–] 6 points 1 month ago

    Oh no, ... anyway, don't have Amazon account

  • source
  • [–] 5 points 1 month ago

    The only drone delivery service I support.

  • source
  • [–] 5 points 1 month ago

    Get them all!

  • source
  • [–] 5 points 1 month ago

    Oh no. Anyway...

  • source
  • [–] 4 points 1 month ago (5 children)

    Strikes during war cause loss of many things, including life. Why is this news?

  • source
  • hideshow 5 child comments
  • [–] 4 points 1 month ago

    kinda surprising it didn't happen earlier

  • source
  • [–] 4 points 1 month ago

    Yall this is AWS not Amazon retail. There might be some regional retail data, but most of this is data from other businesses. Loss of customer data here is referring to AWS business customers, not retail.

  • source
  • [–] 7 points 1 month ago

    Do it again. Harder!

  • source
  • [–] 5 points 1 month ago

    good. more. do more.

  • source
  • [–] 4 points 1 month ago

    Based based based based

  • source
  • [–] 2 points 1 month ago

    I'm sure Alexa still has it.

  • source
  • [–] 2 points 1 month ago

    ik hope my data was deleted

  • source
  • [+] -8 points 1 month ago* (last edited 1 month ago) (1 child)