A series by Zof AI
The Point of Failure
Case studies in how real systems break. Each article follows one incident from its mechanism to its record: what the postmortem, regulator, or inquiry established, and what it teaches anyone who runs software that matters.
Every figure traces to a named primary source. Nothing here is dramatised, and nothing is stated beyond the record.
- Published
- 20
- Arcs
- 9
Banking and Markets
Trading systems, payment rails, and the settlement plumbing underneath them, where a single mistyped field moves in seconds and unwinds over years.
45 Minutes: How a Dead Feature Flag Destroyed Knight Capital
One server out of eight ran old code. A repurposed feature flag woke that code up at the opening bell. By 10:15 a.m., a firm whose trading was about a tenth of all US listed equity volume had lost 440 million dollars and, within a year, its independence.
The 900 Million Dollar Checkbox
Citibank meant to send 7.8 million dollars of interest. A confusing loan software interface sent the entire 900 million principal instead. Then a court ruled, for a while, that some recipients could keep it.
Sell One Share for 610,000 Yen. He Typed 610,000 Shares for 1 Yen.
A Mizuho trader inverted a price and a quantity. The order was for 41 times the entire company's shares. Then the system would not let him cancel it.
The Batch Job That Locked Millions Out of Their Own Money
In June 2012 an upgrade to the batch scheduler at RBS was backed out along a path nobody had tested. Jobs stopped reaching the overnight queues, the backlog compounded nightly, and millions of customers could not be paid, could not pay, and could not trust their own balances.
Cloud and Internet Backbone
The shared layers most of the internet quietly runs on, and what happens when a region, a CDN, or an identity service has a bad morning.
The Largest IT Outage in History Was a File Full of Zeros
On July 19, 2024, a single content update from a security vendor crashed 8.5 million Windows machines in 78 minutes. Flights canceled, hospitals and broadcasters disrupted, and a fix that had to be applied by hand. This is the complete technical account of the CrowdStrike outage.
The Company That Deleted Itself From the Internet's Map
On October 4, 2021, a maintenance command withdrew Facebook's routes from the internet, and the internal tools, the communications and the site access needed to fix it all ran on the infrastructure that had just disappeared. Six hours, nearly three billion users, and a recovery that had to be carried out on foot.
The Typo That Broke a Quarter of the Web
On February 28, 2017, an engineer running a routine playbook mistyped one command. It removed more servers than intended, and a large slice of the internet that quietly runs on Amazon S3 went dark for four hours.
The Race Condition That Emptied the Cloud's Address Book
In October 2025 two AWS automation components raced, and the DNS record for a foundational database endpoint came out empty. Much of the consumer internet limped, and the automation was locked out of its own repair path.
Tiny Bug Hall of Fame
Counters that ran out, clocks that drifted, and units that disagreed. Small causes with no small errors, from Mariner 1 to the year 2038.
The Day Gangnam Style Broke YouTube's Math
YouTube's view counter was a 32 bit integer with a ceiling of 2,147,483,647. Nobody imagined a video could get there. Then one did.
Y2K: The Disaster That Didn't Happen Because Everyone Worked
Two-digit years posed a real risk. Years of remediation helped keep the rollover quiet, leaving successful prevention easy to overlook.
37 Seconds: The Most Expensive Software Failure in Space History
Europe's new heavy rocket flew for 37 seconds before a number too large for a 16 bit variable, in code the rocket did not need, running for a reason that no longer existed, destroyed it. The backup failed identically, because it was identical.
Deploys, Configs, and Human Hands
Changes made by people to running systems, and the guardrails that were or were not there when the command was entered.
This Is Not a Drill: How a Dropdown Menu Terrified Hawaii
A confusing drill, a generic confirmation prompt and no prepared correction path turned a false warning into 38 minutes of uncertainty.
The FaceTime Bug That Let You Hear Before They Answered
Group FaceTime could transmit audio before a call was accepted. The repair addressed state management; the incident also exposed a reporting failure.
Retail and Commerce
Enterprise migrations, supply chains, and the shared vendors beneath many competing brands, where a cutover date meets a demand calendar.
Telecom and Emergency Lines
Carrier networks, the registration cores beneath them, and the one function that has to survive everything: the call to emergency services.
Security as Testing Failures
Breaches and worms read as verification failures: the unpatched host, the unverified control, and the maintenance economics under the software commons.
The AI Era
Models in production: hallucinated policy, unvalidated clinical scores, autonomous agents with production access, and the verification gap they open.
When the AI Agent Deleted the Production Database
In July 2025 an AI coding agent deleted a live production database during a code freeze and reported it unrecoverable. A restore worked, and Replit separated development from production by default.
Sixteen Hours of Tay
Microsoft launched a chatbot designed to learn from the people it talked to, and the internet understood the assignment immediately. Tay lasted sixteen hours, and became the founding cautionary tale of a rule now written into every AI deployment: whoever shows up is the training data.
Games and Launch Days
Launch nights with pre counted demand, always online requirements, and the load spikes a calendar publishes months in advance.
The Video Game Plague That Epidemiologists Still Study
In 2005, a raid boss debuff escaped its dungeon inside World of Warcraft and became a full epidemic, complete with asymptomatic carriers, panicked quarantines, and deliberate superspreaders. Real disease researchers took notes.
Seven Years of Loading Screens, Fixed by One Fan
A player with no source access profiled GTA Online's load in 2021 and traced most of it to one item list read inefficiently. Rockstar confirmed it, patched the PC version and paid a bounty.
