top 87 comments

sorted by: hot top controversial new old
[–] 245 points 1 year ago (30 children)

It's wild that these cloud providers were seen as a one-way stop to ensure reliability, only to make them a universal single point of failure.

  • source
  • hideshow 30 child comments
  • [–] 131 points 1 year ago (8 children)

    But if everyone else is down too, you don't look so bad 🧠

  • source
  • parent
  • hideshow 8 child comments
  • [–] 68 points 1 year ago (6 children)

    No one ever got fired for buying IBM.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 16 points 1 year ago (4 children)

    I wouldn't be so sure about that. The state government of Queensland, Australia just lifted a 12 year ban on IBM getting government contracts after a colossal fuck up.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 58 points 1 year ago* (2 children)

    It's an old joke from back when IBM was the dominant player in IT infrastructure. The idea was that IBM was such a known quantity that even non-technical executives knew what it was and knew that other companies also used IBM equipment. If you decide to buy from a lesser known vendor and something breaks, you might be blamed for going off the beaten track and fired (regardless of where the fault actually lay), whereas if you bought IBM gear and it broke, it was simply considered the cost of doing business, so buying IBM became a CYA tactic for sysadmins even if it went against their better technical judgement. AWS is the modern IBM.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 4 points 1 year ago

    Such a monstrous clusterfuck, and you'll be hard pressed to find anyone having been sacked, let alone facing actual charges over the whole debacle.

    If anything, I'd say that's the single best case for buying IBM - if you're incompetent and/or corrupt, just go with them and even if shit hits the fan, you'll be OK.

  • source
  • parent
  • [–] 13 points 1 year ago*

    One of our client support people told an angry client to open a Jira with urgent priority and we'd get right on it.

    ... the client support person knew full well that Jira was down too : D

    At least, I think they knew. Either way, not shit we could do about it for that particular region until AWS fixed things.

  • source
  • parent
  • [–] 60 points 1 year ago (8 children)

    It's mostly a skill issue for services that go down when USE-1 has issues in AWS - if you actually know your shit, then you don't get these kinds of issues.

    Case in point: Netflix runs on AWS and experienced no issues during this thing.

    And yes, it's scary that so many high-profile companies are this bad at the thing they spend all day doing

  • source
  • parent
  • hideshow 8 child comments
  • [–] 20 points 1 year ago (3 children)

    Yeah, if you're a major business and don't have geographic redundancy for your service, you need to rework your BCDR plan.

  • source
  • parent
  • hideshow 3 child comments
  • Absolutely this. We are based out of one region, but also have a second region as a quick disaster recovery option, and we have people 24/7 who can manage the DR process. We're not big enough to have live redundancy, but big enough that an hour of downtime would be a big deal.

  • source
  • parent
  • [–] 4 points 1 year ago (1 child)

    What's the general plan of action when a company's base region shits the bed?

    Keep dormant mirrored resources in other regions?

    I presumed the draw of us-east-1 was its lower cost, so if any solutions involve spending slightly more money, I'm not surprised high profile companies put all their eggs in one basket.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 4 points 1 year ago

    I presumed the draw of us-east-1 was its lower cost

    At no time is pub-cloud cheaper than priv-cloud.

    The draw is versatility, as change didn't require spinning up hardware. No one knew how much the data costs would kill the budget, but now they do.

  • source
  • parent
  • [–] 31 points 1 year ago (3 children)

    It is still a logical argument, especially for smaller shops. I mean, you can (as self-hosters know) set up automatic backups, failover systems, and all that, but it takes significant time & resources. Redundant internet connectivity? Redundant power delivery? Spare capacity to handle a 10x demand spike? Those are big expenses for small, even mid-sized business. No one really cares if your dentist's office is offline for a day, even if they have to cancel appointments because they can't process payments or records.

    Meanwhile, theoretically, reliability is such a core function of cloud providers that they should pay for experts' experts and platinum standard infrastructure. It makes any problem they do have newsworthy.

    I mean,it seems silly for orgs as big and internet-centric as Fortnite, Zoom, or forturne-500 bank to outsource their internet, and maybe this will be a lesson for them.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 5 points 1 year ago (2 children)

    It's also silly for the orgs to not have geographic redundancy.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 1 year ago (1 child)

    No it's not. It's very expensive to run and there are a lot of edge cases. It's much easier to have regional redundancy for a fraction of the cost.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 4 points 1 year ago* (last edited 1 year ago)

    The organizations they were talking about and I was referring to have a global presence

    Plus, it's not significantly more expensive to have a cold standby in a different geographic location in AWS.

  • source
  • parent
  • [–] 3 points 1 year ago (1 child)

    Well companies use not for relibibut to outsource responsibility. Even a medium sized company treated Windows like a subscription for many many years. People have been emailing files to themself since the start of email.

    For companies moving everything to msa or aws just was the next step and didn't change day to operations

  • source
  • parent
  • hideshow 1 child comment
  • People also tend to forget all the compliance issues that can come around hosting content, and using someone with expertise in that can reduce a very large burden. It's not something that would hit every industry, but it does hit many.

  • source
  • parent
  • [–] 79 points 1 year ago (15 children)

    I hate how Signal went down because of this... Wish it wasn't so centralised.

  • source
  • hideshow 15 child comments
  • [–] 40 points 1 year ago* (5 children)

    My friend messaged me on Signal asking if Instructure (runs on AWS) was down. I got the message. That being said, it's scary that Signal's backbone depends on AWS

  • source
  • parent
  • hideshow 5 child comments
  • [–] 7 points 1 year ago (2 children)

    Why is this scary? That's what e2ee is for, so that no one besides your recipient can view the contents of a message. It does not matter which server is used. If anything for a service like Signal, you want a server with high availability like AWS, Azure, Google Cloud or Cloudflare.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 2 points 1 year ago (5 children)

    Started moving to Element/Matrix this weekend when I attended a protest and wanted to have some kind of communication, but also wanted to leave my primary phone at home. I was using a de-googled android fork and an e-sim, but being a data-only e-sim, I couldn't use Signal due to the phone number requirement.

    Annoying to have try to get contacts to get another app, but at least it's decentralized and comes with the option of being self-hosted once I'm ready to tackle that.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 4 points 1 year ago (3 children)

    @MrMcGasion @Sunny Come to the dark side (xmpp, and jmp.chat) and get decentralized messaging and SMS support with that data-only sim!

  • source
  • parent
  • hideshow 3 child comments
  • [–] 4 points 1 year ago (1 child)

    @Sunny@slrpnk.net already has an XMPP account, as that is included in every slrpnk.net account automatically. It is very easy to set that up for most Fediverse software, and the user id is identical between Fediverse and XMPP.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 4 points 1 year ago

    Oh damn i did not even know about this! I will defo have a play around with this tomorrow, how very neat!

    However, it isnt me im really worried about in the grand picture, its family and friends. It was already difficult enough to convert them to using Signal.

  • source
  • parent
  • [–] 36 points 1 year ago (13 children)
  • [–] 19 points 1 year ago (6 children)

    You guys don't selfhost a registry?

  • source
  • parent
  • hideshow 6 child comments
  • [–] 11 points 1 year ago (2 children)

    I know this is selfhosted so most people here are hobbyists, but it's a ton of work to selfhost in enterprise setting. I'd wager 90%+ of people using image registries are using Docker Hub, GHCR, or AWS ECR.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 1 year ago* (1 child)

    For your personal use, you don't need an enterprise setting. It's just a simple compose file that you run.

    You can host a registry in pull through mode, so you still have all the images you use locally, but if it's not in your registry yet, it pulls it from docker hub or whatever.

    The only pain point is that a single registry can't do both. So if you want to push your own docker images AND have a "cache" of stuff from docker hub, you need to run two registries in two different modes. And then juggle the url's.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 5 points 1 year ago

    Pretty sure you could run Pulp in pull-through mode and add your local Forgejo/whatever registry as a remote, which would at least give you a unified "pull" URL. Then just use Forgejo actions to handle the actual build/publish for your local images whenever you push to main (or tag a release, or whatever).

    Pulp might actually be able to handle both on its own, I haven't ever tried though.

  • source
  • parent
  • [–] 3 points 1 year ago (2 children)

    I hadn’t actually considered that before. What’s your preferred way to do that?

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 1 year ago*

    I have just this (which ironically won't work now cause docker hub is down)

    services:
      registry:
        restart: always
        image: registry:2
        ports:
          - 5000:5000
        dns:
          - 9.9.9.9
          - 1.1.1.1
        volumes:
          - ../files/auth/registry.password:/auth/registry.password
          - registry-data:/var/lib/registry
        environment:
          REGISTRY_STORAGE_DELETE_ENABLED: true
          REGISTRY_HEALTH_STORAGEDRIVER_ENABLED: false
          REGISTRY_HTTP_SECRET: ${REGISTRY_HTTP_SECRET}
          REGISTRY_AUTH: htpasswd
          REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
          REGISTRY_AUTH_HTPASSWD_PATH: /auth/registry.password
          # REGISTRY_PROXY_REMOTEURL: "https://registry-1.docker.io/"
    
    volumes:
      registry-data:
    

    I don't even remember how and when I set it up. I think it might be this: https://github.com/distribution/distribution/releases/tag/v2.0.0

    Recently somebody has created a frontend, which I bookmarked but didn't bother to set up: https://github.com/Joxit/docker-registry-ui

  • source
  • parent
  • [–] 50 points 1 year ago (1 child)
  • [–] 23 points 1 year ago* (3 children)

    according to that page the issue stemmed from an underlying system responsible for health checks in load balancing servers.

    how the hell do you fuck up a health check config that bad? that's like messing up smartd.conf and taking your system offline somehow

  • source
  • hideshow 3 child comments
  • [–] 4 points 1 year ago

    I mean if your OS was "smart" as not to send IO to devices that indicate critical failure (e.g. by marking them read-only in the array?), and then thinks all devices have failed critically, wouldn't this happen in that kind of system as well..

  • source
  • parent
  • [–] 21 points 1 year ago (9 children)
  • [–] 14 points 1 year ago (8 children)

    Is there no way to check the doorbell video locally?

    An Amazon employee misconfigures something and now your doorbell doesn't work

  • source
  • parent
  • hideshow 8 child comments
  • [–] 5 points 1 year ago

    I don't have one (because of that point), so I don't know...

    Presumably the app and doorbell are hardcoded to go to an AWS URL (so it's "easier" for consumers), but in theory the data's all on your wifi.

  • source
  • parent
  • [–] 5 points 1 year ago (5 children)

    Obligatory

    https://reolink.com/

    Oh wow their front page doesn't mention at all that their products run locally and don't require subscriptions.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 2 points 1 year ago (2 children)

    It mentions push notifications and emails, so I guess they must require an account, or can you configure them to use SMTP directly, as with the Amcrest Pro cameras?

  • source
  • parent
  • hideshow 2 child comments
  • [–] 2 points 1 year ago* (1 child)

    TBH, I've never used any of those features. I just used it locally and plugged it into home assistant.

    But I just reinstalled their app and can confirm I can watch the feed and get push notifications without a cloud account. Haven't tried email tho

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 1 year ago

    That sounds worth investigating, thanks! Amcrest needs an account for notifications afaik, but the Pro cameras can work just on a local network.

    The app for them is awful. Then they made a new version that is awful in slightly different ways, so I'm interested in new options.

  • source
  • parent
  • [–] 1 point 1 year ago

    The last time I looked at time I missed something. I want to use my existing chime and have it also use that power supply as a power source. Turns out the battery one does that! I missed that months ago! I was going to get a Ring Pro or whatever to get it to do that! This is so much better! Moving up the todo list now!

  • source
  • parent
  • load more comments (1 reply)
  • [–] 18 points 1 year ago (5 children)

    That explains why my Matrix <-> Signal bridge was complaining about being disconnected.

  • source
  • hideshow 5 child comments
  • [–] 2 points 1 year ago (2 children)

    You can connect Matrix with Signal?

  • source
  • parent
  • hideshow 2 child comments
  • [–] 27 points 1 year ago (2 children)

    Who wants to bet Amazon gave AI full access to their prod config and it screwed it up.

  • source
  • hideshow 2 child comments
  • [–] 11 points 1 year ago

    that is an understatement 😂

  • source
  • [–] 27 points 1 year ago

    A bad day for Jeff Bezos is a good day for all of us

  • source
  • load more comments
    view more: next ›
    18 comments in c/sysadmin@lemmy.world view that post