all 100 comments

sorted by: hot top controversial new old
[–] 409 points 2 days ago (18 children)

No mails arrived due to the disruption [...]. So no one noticed that the maintenance safety system of the sauna landscape has long since sounded the alarm by mail. Because the automatic dosing system had pumped too much chlorine into the pool.

Kinda scary to rely solely on email for such alarms.

  • source
  • hideshow 18 child comments
  • [–] 222 points 2 days ago (13 children)

    Yeah, I hate Microsoft as much as the next person, but this is 100% on Phantasialand for trying to cut corners with excessive automation.

  • source
  • parent
  • hideshow 13 child comments
  • [–] 114 points 2 days ago* (last edited 1 day ago) (1 child)

    I think that automation is fine, but when you start talking about life-critical systems, the bar on failure modes and reliability becomes a lot higher. Like, if you have a sensor for excessive chlorine levels, the default probably should be not to keep pumping more of the chemical in, but to have some "hard" upper level where no more chlorine goes in (and if the two systems aren't directly-integrated, to at least let the maintenance safety system power off the chlorine system), and for a secondary alarm to be triggered (like, start playing a loud audible alarm) if maintenance doesn't respond to the email notification and indicate that they've dismissed the alert within a given amount of time. That's disruptive and undesirable compared to just emailing maintenance, but as long as there aren't any failures, you shouldn't ever see it. But if you do have it come up, you don't want the failure mode to be chemical-keep-being-dumped-into-the-system-and-people-being-unaware-of-it.

    You want fail-safe behavior.

    Automation can increase safety by decreasing time until an issue is addressed, for example. Automation isn't intrinsically incompatible with safety. Aircraft computers today use TCAS to automatically instruct pilots what to do if they are about to collide (and in fact, today the automated system is explicitly given higher priority than human air traffic controllers


    if the two give pilots conflicting directives, the human is the one to be disregarded). But you gotta make your automated procedures have high reliability and handle failure the way you want when the consequences of a failure are high (and that goes for non-automated procedures too).

  • source
  • parent
  • hideshow 1 child comment
  • [–] 7 points 1 day ago

    It’s like those supermarket fridges.

    When these things go off a literal alarm needs to be screaming that there’s an issue.

    Yes, you can set up additional email alerts and logging. But the primary mechanism needs to be an in your face loud uncomfortable alarm, and safety cutoffs like you said.

  • source
  • parent
  • [–] 59 points 2 days ago (2 children)

    Excessive automation? That actually seems like a great thing to have automated... just not by email. Nothing stopping them from having a sub ghz radio that sets off a physical alarm somewhere. Improper automation I think would be more apt. A purely local automated alarm sounds great, but the internet shouldn't be involved imo, at least not on the critical path to remediation.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 98 points 2 days ago (2 children)

    It's not excessive automation -- it's a lack of redundancy for critical safety features.

    Anything safety-critical should never have a single point of failure if you can help it. And there was definitely an opportunity here to have some kind of secondary backup alarm system.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 15 points 2 days ago

    I for one think that IT team should be kicked to the curb, but they will probably get raises to figure out that they can use email, healthchecks.io, and ntfy all at the same time for redundancy or they will say we need money to switch providers and set up a gmail account to do the same thing and call that redundant cause you know getting two emails saying the same thing is not going to end up just being ignored.

  • source
  • parent
  • [–] 93 points 2 days ago* (last edited 1 day ago) (2 children)

    The technical standards for Email would allow for 3 days of delivery failure before finally giving up. That's the amount of leeway email has (in theory - practically we assume that everything longer than 3 minutes means something has gone wrong). Sending alerts by email is fine for non-critical applications; if one of my docker containers crash, an email is fine because the amount of damage is nil.

    Why wasn't there an actual fail-safe mechanism employed that measures the chlorine levels and shuts the whole thing down BEFORE it gets dangerous?

    This is not a Microsoft problem. This is a negligence problem on Phantasialands side, using inappropiate tools for a system that can hurt people on failure, and should be followed up by lawsuits.

  • source
  • hideshow 2 child comments
  • [–] 14 points 1 day ago (1 child)

    From: noreply@carbondioxidedetector.com.au

    RE: Carbon Dioxide leak detected

    Message: To whom it may concern, an elevated carbon dioxide level was measured in the walk-in fridge in the bar kitchen. Probably one of the keg lines is leaking pressurised gas. Gas PPM is classified as alert level 5 - Immediately harmful to life. Please evacuate staff from the area and ventilate the fridge thoroughly before attempting entry with ELSRD hood donned to shut down gas flow.

    Folder: Spam

  • source
  • parent
  • hideshow 1 child comment
  • [–] 10 points 1 day ago

    Yeah

    “Why do all our alerts go to spam”

    Oh, well everyone just ignores them, then Jeff who isn’t even on the IT team but was on the mailing list was getting them. So Jeff marked them as spam instead of asking to be removed from the mail list, and now our corporate email provider just black holes them.

  • source
  • parent
  • [–] 89 points 2 days ago (1 child)

    This is why I always use Yahoo mail for my critical life safety notifications.

  • source
  • hideshow 1 child comment
  • [–] 156 points 2 days ago (3 children)

    Can't put any blame on MS here. Using email for safety warnings with no redundancy is negligent.

  • source
  • hideshow 3 child comments
  • [–] 34 points 2 days ago (1 child)

    One can put some blame on them, legally speaking.

    • First, MS was outside their SLA. That's right in their service agreement.
    • Second, they were so far outside their SLA that it gets into reasonable expectation.
    • Third, there will be litigants who assert negligence.

    Which of those claims will stick remains to be seen, but "no blame" is incorrect.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 8 points 1 day ago* (last edited 1 day ago)

    Your answer is 100% incorrect. The hotel is solely responsible for the incident as it is their responsibility (liability) for maintaining the system. The hotel failed to properly maintain the system and provide adequate monitoring of said system. If the alerts were delivered by traditional mail and the letter was late, you wouldn't blame (i.e. sue) the post office.

    If the hotel wants to sue Microsoft, they can try. They will get the damages from the missed SLAs, everything else will be laughed out of court.

    Your post reads like anti-Microsoft hate rather than actual legal argument. As much as MS sucks, there's no jurisdiction here much less a burden of care or duty on MS's behalf.

  • source
  • parent
  • [–] 3 points 1 day ago

    Yup. They put a safety critical system behind an alert system that is roughly the equivalent of sending a 6 year old across town to deliver a message. Sure they failed to deliver a message, but they never should have been relied upon to do that in the first place. The failure isn't the kid's fault; it's the idiot's fault that trusted them with a life and death situation.

  • source
  • parent
  • [–] 64 points 2 days ago* (last edited 1 day ago) (45 children)

    Yeaaah not MS fault here, no sane person would rely solely on email for something like this

  • source
  • hideshow 45 child comments
  • [+] -17 points 2 days ago (44 children)

    What else should they rely on?

  • source
  • parent
  • hideshow 44 child comments
  • [–] 23 points 2 days ago (2 children)

    Good engineering practices for starters.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 3 hours ago (1 child)

    Which are? Please tell me and give examples. I'm not an engineer and don't know what's code

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 1 minute ago

    An important one is something called a fail-safe. It's quite literal; it dictates that a system must be safe in it's failure state. In this case when it detected a failed pump, the system should have shut down. Fail safes are assistant passive and don't rely on external systems.

    As an example, image a room with a magnetic lock on the door. It should be design so that when the power goes out, people inside can still exit via either a mechanical push, or the latch just releasing from not having power.

    https://en.wikipedia.org/wiki/Fail-safe

  • source
  • parent
  • [–] 33 points 2 days ago* (last edited 1 day ago) (14 children)

    An actual automation system. Ideally, a self-hosted one.

  • source
  • parent
  • hideshow 14 child comments
  • [–] 2 points 1 day ago (3 children)

    I wouldn’t advise self-hosting unless you can guarantee the resources to do it right, and with human lives on the line plus negligent management, that’s a lot to ask.

    And for modern SMS alerts you probably need someone else’s platform anyways, so you’re halfway to paying for this.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 2 points 1 day ago

    I wouldn’t advise self-hosting unless you can guarantee the resources to do it right,

    Bit of a pickle. If you can't trust the biggest name in the industry to get it right...

    Parallel: Pagerduty, Slack, NTFY.

    you can self host NTFY then you have 1/3 under your express control.

  • source
  • parent
  • [–] 2 points 1 day ago (1 child)

    I mean, the pump is self-hosted. The water is self-hosted. Smoke detectors are self-hosted.

    Making an exceedingly simple system that sounds an alarm and/or flashy light when a threshold is crossed is hardly new. It doesn't take that much and if you're really worried, set up a duplicate system to run in parallel.

    Using email for something important is dumb. This is why 911 doesn't have an email address.

  • source
  • parent
  • hideshow 1 child comment
  • [+] -15 points 2 days ago (9 children)

    and it would alert everyone how?

  • source
  • parent
  • hideshow 9 child comments
  • [–] 28 points 2 days ago

    Notifications on the automation portal, a push notification if you have an app, an sms message, an email, a physical alarm or all of the above. Hell, you could even have a physical alarm at the actual pool to alert customers in case someone from maintenance missed the alarm. I don't know how this is an actual question for you xd

  • source
  • parent
  • [–] 16 points 1 day ago (4 children)

    Imagine being so slop brained that you forget that physical alarms and local system alerts are a thing.

  • source
  • parent
  • hideshow 4 child comments
  • [–] 3 points 1 day ago (1 child)

    This is going to get worse as more and more people turn their brains to mush with chatgpt

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 3 hours ago

    My god is it ever. Idiots are going to be asking "how can I say something to make it sound like I'm an expert about something when someone asks a question without knowing a thing? Phrase it in a way to get upvotes."

  • source
  • parent
  • [–] 1 point 3 hours ago (1 child)

    Ok smart ass. Where are the alarms in retrospect to anyone that can hear them? Tell me all about your experience setting up alarm systems for people that aren't in the building.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 2 hours ago

    You’ve never worked a building engineering or maintenance system at all, have you?

    I’m in residential at the moment, but at my last job, there was always someone on call for equipment issues like this. Alarms would pop up on any BAS terminal if it happened during the day. Building I worked at had high pressure steam boilers, so there was always someone on-site as well. You’d get a push notification from the BAS app on your work phone as well if you were away from a workstation. Hell it would automatically text and then call you too. It’s 50/50 if the call was triggered by an email alert or if it would just dial from the main line and do direct text to speech. Don’t remember how it was configured. But I do know the text message alerts were built into the software. There were 3 ways to reach someone, and all the major systems are tied into it, so can be remotely accessed via a laptop for troubleshooting or shutting down equipment.

    Actual automation systems have redundancy built in, because they are necessary for safety and thus reduced liability. We had high pressure steam boilers. If the fucking building blew up, I couldn’t exactly go to my boss and say “well you see, Microsoft had an oopsie and that’s why the multi million dollar building doesn’t exist anymore and why we’re on the news, not my fault, boss” of course, systems that are actually designed well will have hardware safeties built in with the monitoring and alerting as well.

    Oh and I was also required to do rounds at every shift change and physically check the analog gauges to make sure they matched what the BAS was reporting, and log any discrepancies for investigation if they could not be immediately identified.

    Fucking hell you are slop brained. Not everything should or does run though a fucking cloud. Email has always been a backup to a backup for critical systems because sometimes it takes minutes for an email to arrive on a good day. Physical alerts on equipment, electronic notifications on local hardware, and telephone communication for remote alerts have always been the industry standard. I pray to god no one like you ever steps within 50 yards of a boiler with a “cost saving” improvement to how things are actually run.

  • source
  • parent
  • [–] 5 points 1 day ago (3 children)

    Chlorine level monitors that staff check before opening the pool? That doesn't require internet and would only be a simple embedded device. You might be able to have an electricity-free option too, not a chemistry expert but I imagine there are tests for chlorine levels that don't require electricity?

  • source
  • parent
  • hideshow 3 child comments
  • [–] 2 points 18 hours ago* (2 children)

    Yeah they exist. Most people who own home pools use them and then take their water to be tested if something is getting way off.

    I helped out at a hotel for a couple months at one point and their setup was simple (to use). Every morning we would check everything manually to ensure it was correct, it was also controlled by machine. Essentially if the machine detects the level of something gets to low, it automatically adds the amount needed. Essentially all of the water is cycled into a room near the pool, and that's where all water is added and tested automatically from.

    The manual tests we would do from 2 different areas of the pool to ensure they matched what the pool sensors detected in the behind the scenes area. If something was way off somehow alarm lights would show on the panel, and no chemicals would be added and a contracted maintenance group would be dispatched to come address the major issues.

    Also pools are large amounts of water, so they don't just suddenly change drastically unless someone is intentionally pouring chemicals of some kind into them.

    Id say 98% of chemical issues are dealt with by normal staff and the automated system. The pool company with the contract comes to verify everything working once a month, and maybe once or twice a year you'd call them out for something, most likely after winter when opening the pool just to make sure they clear everything, and once a year when something weird happens.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 3 hours ago (1 child)

    How much were you paid for this?

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 3 hours ago* (last edited 2 hours ago)

    I got confused and thought you said this to another comment, paid to help out at the hotel? Not to much it was basically a maintenance job that had this as a morning and afternoon side quest. At the time it was probably like $15/hr,

  • source
  • parent
  • [–] 5 points 2 days ago* (last edited 1 day ago) (7 children)

    Idk, any self-hosted system, there are so many stuff one can do...even having displays or alarms...

  • source
  • parent
  • hideshow 7 child comments
  • [+] -6 points 2 days ago (5 children)

    MS wants businesses to sign up for their cloud services, promises the world, and says don't worry. Businesses rely on this nd when it goes down it's not MS's fault even tho MS is the service that went down? Come on.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 4 points 1 day ago (1 child)

    It is their fault for not having a fallback system when lives are on the line. And the whole strategy of "outsource something so we can blame that when it stops working" is transparent, lazy, and negligent (criminally in this case IMO). Downtime has always been a thing and it's not like MS has a reputation for reliability (I would have said the opposite based on using their software over the last few years).

    If you haven't realized by now that marketers will say whatever they think will get them the sale, that doesn't say great things about your intelligence, even if there's a lot of people falling for that BS.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 3 hours ago

    And who are the people that fall for that bs? Spoiler: they're the same idiots that think they know more than engineers and won't buy the systems with more contact options because it costs more (or not enable those alert systems because they are extra services and cost more).

  • source
  • parent
  • [–] 4 points 1 day ago (1 child)

    It's not MS fault if you don't also have a fallback system or any system that can work even offline/dipendently on your network. Regardless of microsoft being down or not, emails for such stuff is a VERY bad idea and should be changed with a better system that cannot be accidentally ignored.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 5 points 1 day ago (2 children)

    There are a ton of on-call systems available that have hard contractual uptime reliability requirements.

    Every time I’ve been on a team with on call we had a required app that pushed critical notifications to your phone, plus SNS, plus email, plus slack, plus an actual phone call system.

    And I’m not even an SRE or anything like that.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 4 points 21 hours ago (2 children)

    Public pools here are required to have audible alarms and flashing lights on redundant electric circuits in the dosing room and main hall, plus a shitload of ventilation.

    Every walk in fridge is required to have CO2 alarms, just like smoke detectors.

    Not everything needs to go through a computer. A loud ass alarm is sufficient.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 54 points 1 day ago (3 children)

    Subject: Fire.

    Dear Sir/Madam,

    I am writing to inform you of a fire that has broken out on the premises of 123 Cavendon Road.

    Fire! Fire! Help me!

    123 Cavendon Road. Looking forward to hearing from you.

    Yours truly,
    Maurice Moss

  • source
  • hideshow 3 child comments
  • [–] 7 points 1 day ago (1 child)

    I know he tried calling 0118 999 881 999 119 725 3 first, but had he not ended up with another country on the line it would have been a much quicker response.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 23 points 2 days ago

    Industrial automaton engineer here, I've worked on a bunch of safety critical systems and audited systems after incidents.

    Best practice in situations were you are sending a remote start command, is for the stop command to be local to the pump controller.

    I audited a system where the main control system sheet a start command to the dosing shed (which stored the concentrated chemicals), unfortunately the stop command was over the same link that got interrupted. The start was received but the stop was never was. Radio is not a reliable link.

    For chemical dosing, I'd expect at least two types of command signals, such as concentration and Max allowed time; you could also measure the weight loss of the concentrate as a third check. Loss of communications to the main controller isn't necessity an instant stop, but I would expect a prolonged loss to trigger a stop. As long as the local controller has stop control, dosing can usually continue.

  • source
  • [–] 42 points 2 days ago

    Email is not a guaranteed delivery system. Now they know.

  • source
  • [–] 17 points 2 days ago (1 child)

    We were all exposed to this problem: on Friday there was an outage that crippled the US software company Microsoft, including the email program Outlook.

    Phantasialand was affected as well. Because of the disruption, no emails arrived, the company told WDR on Saturday. So nobody noticed that the maintenance safety system for the sauna area had long since raised the alarm by email. The automatic dosing system had pumped too much chlorine into the pool.

  • source
  • hideshow 1 child comment
  • [–] 16 points 2 days ago

    Damn, Microsoft hurting people they didn't even mean to (in addition to the ones they mean to).

  • source
  • [–] 15 points 1 day ago

    What the fuck?

  • source
  • [–] 14 points 2 days ago (1 child)

    This isn't MS fault, what happened to Normal Alarming System

  • source
  • hideshow 1 child comment
  • [–] 12 points 1 day ago

    I've been to Phantasialand. It was a nice park.

    I have to agree with the other commenters. I feel like this is a task for a loudly audible alarm system and maybe a flashing strobe in the operators office, not an email.

  • source
  • [–] 23 points 2 days ago* (last edited 1 day ago)

    Wow. Why wire up a blinking lite and speaker in the managers office to the system when you can just rely on fucking email. Pretty sure whoever made it possible to use email intended it to be a backup solution in case no one in charge was on site when the alarm went off.

  • source
  • [–] 9 points 1 day ago (4 children)

    Why the fuck would you open the pool without getting a mail saying it is safe?

  • source
  • hideshow 4 child comments
  • [–] 11 points 1 day ago (3 children)

    The email is that there is a dangerous condition, not that it is safe.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 1 point 22 hours ago (1 child)

    That's such a rookie mistake to make in safety design..

  • source
  • parent
  • hideshow 1 child comment
  • Yeah, standardly you'd have someone manually check chemicals daily, and the automated system would correct chemicals when not in use, but where I grew up it isn't so much of a pool being "opened" as it was just that it's open around 365 days a year and less people just use it in "winter."

    But if its 81F/27C outside on Christmas day, people will be swimming on Christmas..

  • source
  • parent
  • [–] 9 points 18 hours ago

    My job puts me in frequent contact with the safety team at a chemical plant. There are sensors all over the plant and warehouse and if the tiniest bit above nominal levels of quite few different chemicals is detected in the air, an audible alarm sounds and lights flash, along with automated reporting to safety team members.

    My job does not involve interacting with chemicals, and my duties on the production floor are basically just rebooting printers and plugging computers back in as needed, so my instructions are basically "if the alarm goes off, immediately return to the office and listen out for any further instructions"

    The fact that the monitoring mechanism literally only reported overage by email is a terrible design and the organization deserves whatever regulatory and legal backlash they receive for that decision

  • source
  • [–] 14 points 1 day ago (1 child)

    Germans gassing people

    2026

    did not expect that on my bingo card.

  • source
  • hideshow 1 child comment
  • [–] 5 points 1 day ago
    [–] 7 points 2 days ago
    [–] 5 points 1 day ago

    GTFO of the cloud, you idiot humans

  • source
  • [+] 1 point 2 days ago* (last edited 1 day ago)
    load more comments
    view more: next ›
    2 comments in c/tech@feddit.online view that post
    2 comments in c/hackernews@lemmy.bestiver.se view that post