If you were waiting at a U.S. airport on the afternoon of Friday, November 13, your flight probably wasn't going anywhere.

The Federal Aviation Administration stopped most departures nationwide after air traffic controllers lost reliable communications and radar coverage in places. The stop lasted about five hours. By the evening, at least 827 flights had been diverted, according to flight-data firm Cirium, as the Associated Press reported.

Three days later, some Chase customers opened their banking app and saw a message telling them some services weren't available. On the same Friday as the ground stop, staff at an NHS trust in England couldn't rely on their email. And in October, part of a Google Cloud data center in London lost power for most of a working day, disrupting some UK banks and payment services.

These incidents had different causes, and I'm not suggesting a common thread in how they started. What connects them is the question every operator faced once they did:

When a critical system fails, what should still work for the people who depend on it?

I believe that question deserves more attention than it gets. Most resilience planning is organized around systems: the primary, the backup, the recovery time. The people on the other end experience something simpler. Either they can still do the few things they need to do, or they can't.

Start With the Functions, Not the Systems

For any critical service, there's usually a short list of things people must still be able to do during a failure.

A traveler needs to know whether their flight is going and how to get on another one.

A bank customer needs to see their balance and know whether a payment went through.

A hospital employee needs to reach colleagues and keep patients informed.

None of those require the full system. They require a small set of functions that keep working when the main system doesn't, and that don't depend on the same components that just failed.

That is the standard I'd use to judge each of these incidents. Not whether the outage happened, but what remained available while it did.

Aviation: Stopping Was the Safe Response

Here is what has been confirmed so far.

Officials initially said that new equipment being added to the air traffic control network had overwhelmed older parts of it, causing intermittent communications and radar gaps. Homeland Security Secretary Markwayne Mullin said there were no signs of a cyberattack (AP).

By Saturday, the root cause still hadn't been established. FAA Administrator Bryan Bedford said routers and networking equipment appeared to be involved and that the agency "was unwilling to take any risk" that the problem could cause a sensor to lose its connection with aircraft. "What we saw was instability. We couldn't explain the instability," he said, CNBC reported. L3Harris, which operates the network, said it was supporting the FAA's investigation.

Grounding flights was a safety response, and I think it was the right one. When you can't explain what a system is doing, and the system keeps aircraft apart, stopping is the safe output. A well-designed critical system should fail toward safety, even when that's expensive.

The more useful questions are upstream. The investigation will establish what actually happened, and I'm not claiming any of the following would have prevented it. But in my judgment, these are the measures that reduce the risk of this kind of event:

  • Testing against the old equipment, not just the new. A modernization program connects new components to legacy ones. The test environment should include realistic versions of the older equipment and the traffic it actually carries.
  • Staged network changes with rollback criteria written in advance. Change one site or one segment at a time, and decide before the change what signal means "roll back," so nobody is debating it during an incident.
  • Drills for switching to independent communications. If the primary network becomes unstable, controllers and facilities need a separate way to talk, and the switch has to be practiced, not just documented.

The airline side of the same day raises a different question. Airlines can't fly during a ground stop, but they can still tell passengers what's happening. In my view, flight status updates and basic rebooking should run on a path that doesn't depend on the airline's operations systems being healthy, so a passenger can still get an answer and a new seat even while the rest of the system is working through the backlog.

Banking: "Pending" Is an Answer

On Monday, November 16, Chase said, "Due to a technical issue, some transactions are taking longer than usual to complete," and added that "customers still have access to their accounts and funds" (AP). Some customers couldn't sign in, and others saw a message asking them to try back later. Chase said it wasn't a cyberattack.

The effect reached beyond Chase. U.S. Bank said its own systems weren't directly affected, but that customers sending or receiving Zelle payments with another institution could see delays. Early Warning Services, which owns Zelle, said its network was fully operational and attributed the delays to "an issue at the financial institutions."

That pattern is common in payments. One participant has trouble, and everyone who transacts with it feels the delay.

Here is what I'd want a bank's systems to do in that situation, as a general design principle rather than a judgment about Chase:

  1. Isolate the failing connection. If one payment network or one counterparty is slow, stop sending new work to it for a while instead of letting requests pile up and drag everything else down. Engineers call this a circuit breaker, after the electrical device that cuts a circuit before it overheats.
  2. Keep basic account access separate. Viewing a balance and recent transactions shouldn't depend on the payment path that's struggling.
  3. Show the true status. "Pending" is an honest answer. A customer who can see that a transfer is in progress is far less likely to send it again.
  4. Reconcile before replaying. Before resending anything after the problem clears, compare your records with the network's and confirm which payments actually went through.

Point 4 is where duplicate payments come from, and the protection is simpler than it sounds.

Every payment request gets a unique reference number when it's created, something like a claim ticket. If the same request arrives twice, because a customer tapped "send" again or a system retried automatically, the receiving system checks the ticket, sees it already handled that payment, and returns the original result instead of moving the money a second time. Engineers call this an idempotent request. It means a retry can be safe, which matters most on exactly the days when retries happen.

Cloud Services: A Second Way to Talk

On November 12 and 13, Microsoft 365 services, including Outlook, were disrupted for users around the world. Microsoft's Charles Lamanna later explained that "a latent defect caused servers to become overwhelmed and unresponsive under certain load conditions," and that a surge in load combined with reduced server capacity triggered a bug introduced in July. "Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed," he wrote (ABC News).

That last point applies to any operator. Bringing a service back all at once can recreate the overload that took it down. Letting traffic back in gradually is part of recovery, not a delay to it.

In the UK, the Isle of Wight NHS Trust said on November 13 that the main impact was "on staff emails and some administrative systems." It told patients to keep attending planned appointments, to call 999 or use NHS 111 as normal, and to contact services "by telephone rather than email where possible" (Isle of Wight NHS Trust). The trust said ambulance services continued to respond and that its teams had established plans to maintain safe patient care.

That's what a fallback looks like when it works: the trust had another channel, and it told people which one to use.

For most organizations, the same principle applies to internal communications. When the primary collaboration platform is down, an independent channel, with its own accounts, its own infrastructure and a contact list that doesn't live inside the system that failed, lets staff:

  • reach the people on call and the managers who make decisions;
  • send one clear status message instead of dozens of uncoordinated ones;
  • tell customers or patients what to do in the meantime;
  • coordinate the recovery itself.

If that channel signs in through the same identity system as the primary platform, it may fail at the same moment. Independence has to be checked, not assumed.

Redundancy Only Helps When the Backup Can Survive the Same Failure

The Bank of England and the Financial Conduct Authority described the October incident in detail in a letter to the Treasury Committee on November 4.

According to that letter, a data center facility supporting Google Cloud's europe-west2-b zone, one of the zones Google lists in London, lost all power at 07:00 BST on October 20. Service from that zone was affected for about seven hours. Google attributed the incident to an underground cable fault outside the facility. The data center's backup generator didn't receive the signal it needed to start, and the batteries ran down.

The regulators reported disruption to customer-facing banking and payments services at some UK firms, and to internal systems supporting areas such as asset management and trading. Third-party providers serving regulated firms were also affected, which created knock-on effects through supply chains. Firms activated contingency arrangements, and almost all normal operations had resumed by the end of the day, though some firms needed extra time to restore their own services after Google's zone came back. The regulators also noted that they first learned of the disruption from affected firms, not from Google.

That generator detail is the lesson. The facility had a backup power source, but the backup depended on a signal that didn't arrive. A backup only helps if it can survive the failure it's meant to cover.

The same logic applies one level up. Running in a second zone or region protects you only if the things your service needs to function are also available there. The dependencies that tend to get missed are the unglamorous ones:

  • the identity service people sign in through;
  • DNS, certificates and encryption keys;
  • the database replica, and whether it's current enough to use;
  • a third-party provider that itself runs in the zone that failed;
  • the deployment tooling needed to move traffic in the first place.
Diagram of a primary and a backup environment that both depend on one shared component, so a single failure takes out both, beside a second version where each environment has its own copy and the backup keeps running.
A second site helps only if the services it depends on survive the same failure.

And then there's load. A backup environment that has never carried full production traffic is a hypothesis. The switch needs to be tested under realistic load, with the people who'd actually perform it, before the day it's needed. I made a smaller version of this point about restores in Before You Move to the Cloud, Know What You're Paying For: a backup you haven't restored isn't a plan yet.

What Catches a Problem Before Rollout, and What Limits It After

It helps to separate the two, because they're different investments.

Before a change reaches production, the goal is to find the problem while it's still cheap:

  • test environments that include the older systems and realistic traffic, not only the new components;
  • staged rollouts, one site or a small share of traffic at a time;
  • rollback criteria agreed before the change starts;
  • a dependency map showing what each critical function actually relies on;
  • failover tests that run under real load.

Once a disruption starts, the goal is to protect the people who depend on the service and recover cleanly:

  • reduced-function modes that keep the essentials available;
  • an independent communications channel that staff have practiced using;
  • honest status, including "pending" for anything not yet confirmed;
  • safe retries, with reconciliation before anything is replayed;
  • a gradual return of traffic so recovery doesn't recreate the overload.

Neither list can rule out an incident. As the UK regulators put it in their letter, "not every operational incident can be prevented." The aim is to make failures smaller, shorter and easier for the people affected to live through. In The Future of Cloud Infrastructure, I argued for resilient systems without excessive duplication. This is how I'd decide where the duplication actually belongs: around the few functions people can't do without.

Three Questions for Your Own Critical Services

If you run, buy or depend on a critical service, these are the questions I'd put to the team this week:

  1. Which functions must stay available? Not which systems, but what customers, patients or staff must still be able to do when the main system is down.
  2. What dependencies do the primary system and the backup share? Power, network, identity, data, vendors and people. Anything on both lists can take both down.
  3. When did we last prove the fallback works? Not on paper. Under realistic load, with the people who'd actually run it.

If the answer to the third question is "we haven't," that's the place to start.

Build Resilient Infrastructure With PreciseMP

This is the work behind PreciseMP's Cloud Solutions & Infrastructure and DevOps & Platform Engineering services: mapping what a critical service depends on, designing the fallbacks around the functions people need, and testing the switch before it matters. For the operational side, including monitoring and incident response, see Managed IT & Cybersecurity.

Sources and Further Reading