AWS S3 outage, 2017
On February 28, 2017, Amazon’s S3 storage service — which backs a huge portion of the internet — had a more-than-four-hour outage in its Northern Virginia region starting at 9:37 AM PST. Thousands of dependent websites, apps, and devices failed along with it, not because they ran on the broken subsystem but because they depended on S3 for images, files, and configuration.
What it looked like at the time
From the outside, S3 in us-east-1 started returning errors for basic operations — reads, writes, and listings all failed. Because the dashboard and administrative tools also relied on S3, Amazon’s own status reporting was delayed, so for a stretch of time operators everywhere could see their systems failing without being able to see why.
Root cause
An authorized S3 engineer was debugging a slower-than-normal billing subsystem and ran an established command to take a small number of servers out of service. One of the command’s inputs was mistyped, and a much larger set of servers was removed — including servers behind two other S3 subsystems. One of them, the index subsystem that tracks metadata and object locations for the region, could not be restarted quickly, and S3 cannot serve objects without it. The AWS postmortem is candid that the tooling allowed far too much damage from a single command, and that the team had planned for component failures but underweighted the failure mode of one quick operator action hitting several subsystems at once.
How it was found and fixed
The affected subsystems were restarted and S3 returned to normal by about 1:54 PM PST. Amazon then removed the ability of that tool to take out so many servers at once, added recovery tooling and a rehearsed recovery plan for the index subsystem, split control-plane operations so one region’s maintenance can’t take down global tooling, and published the full postmortem within two days.
The lasting lesson
Operational tools deserve the same design scrutiny as product features: a maintenance command that can remove the wrong servers is a loaded weapon with no safety catch. Blast radius is a design decision — dangerous actions should be segmented, confirmable, and reversible, especially for the humans having the worst day of their month.
Practice the skill
A debugging challenge that works the same muscle:
- Works On My Machine — a debugging challenge in the same discipline
- The Silent Crash — a debugging challenge in the same discipline