The Typo That Broke a Quarter of the Web

Four hours in us-east-1

On the morning of February 28, 2017, a large portion of what people call "the internet" stopped working, and much of it did so in the same way at the same time: images failed to load, apps hung, dashboards went blank, smart home devices stopped responding. The common thread was Amazon Simple Storage Service, S3, in the us-east-1 region in Northern Virginia, the oldest and busiest region of the world's largest cloud. For roughly four hours, S3 in that region was severely impaired, and because a staggering number of websites, applications, and services store their files, images, and even their operational data in S3, the outage rippled outward into thousands of unrelated products. Amazon published a candid summary a few days later, and everything technical below comes from it.

What actually happened

An authorized S3 team member was debugging an issue with the billing subsystem. Following an established playbook, they ran a command intended to remove a small number of servers from one of the S3 subsystems. The command was entered with an incorrect parameter, and it removed a larger set of servers than intended, including servers that supported two other critical S3 subsystems. One was the index subsystem, which manages the metadata and location information for every object in the region, the catalogue that knows where every file lives. The other was the placement subsystem, which manages new storage allocation. Removing enough capacity from the index subsystem meant it could no longer serve requests, and it required a full restart to recover.

Here the story turns on scale. These subsystems had not been fully restarted in years, because S3 had grown enormously in that time and the region had never needed to. The restart, and the safety checks that ran during it, took far longer than anyone anticipated, because the systems had to process an amount of stored metadata that had grown by orders of magnitude since the restart logic was last exercised. The recovery was not a matter of turning something back on; it was a slow, careful validation across a massive dataset. Amazon's engineers could not simply hurry it.

There was a now famous secondary irony. The AWS Service Health Dashboard, the page customers check to see whether AWS is healthy, depended on S3 in us-east-1 to display its status indicators. So during the outage, the dashboard could not turn red. For a period, Amazon had to update customers via Twitter because the tool built to report outages was inside the outage.

The fixes and the fairness

Amazon's response was the model the industry now expects, and notably it contained no name and no blame. The summary asked why one command could remove so much capacity, and answered with engineering: the tool was modified to remove capacity more slowly and to refuse to take any subsystem below a minimum safe capacity level. The company also committed to changes that would let subsystems recover faster and audited other services for the same "long time since restart" risk. This blameless framing, focused on why the system permitted the error rather than who made it, is deliberately contrasted elsewhere in this series with responses that singled out individuals, and the contrast is instructive: the S3 fix made the next engineer safer, which is the only kind of fix that scales.

What it teaches

Three lessons carry beyond cloud operations. First, recovery paths decay when unused: any restart, failover, or rebuild that has not run at production scale in years is an untested claim, and the S3 outage lasted as long as it did precisely because the recovery logic met a dataset it had never faced. Rehearse the recovery, not just the failure. Second, blast radius is a design parameter: a routine tool should not be able to reach across subsystem boundaries and remove critical capacity, and the fix was to build the floor that should always have been there. Third, your status and recovery tooling must not share fate with the thing it monitors: the dashboard that cannot turn red is the same failure family as the outage responder locked out of the building, and it recurs throughout this series because the instinct to build monitoring on the same convenient infrastructure is nearly universal. One mistyped parameter is not a story about one engineer. It is a story about every guardrail that was not yet there when the parameter was typed.