A series by Zof AI
The Point of Failure
Case studies in how real systems break. Each article follows one incident from its mechanism to its record: what the postmortem, regulator, or inquiry established, and what it teaches anyone who runs software that matters.
Every figure traces to a named primary source. Nothing here is dramatised, and nothing is stated beyond the record.
- Published
- 8
- Arcs
- 4
Banking and Markets
Trading systems, payment rails, and the settlement plumbing underneath them, where a single mistyped field moves in seconds and unwinds over years.
45 Minutes: How a Dead Feature Flag Destroyed Knight Capital
One server out of eight ran old code. A repurposed feature flag woke that code up at the opening bell. By 10:15 a.m., a firm whose trading was about a tenth of all US listed equity volume had lost 440 million dollars and, within a year, its independence.
The 900 Million Dollar Checkbox
Citibank meant to send 7.8 million dollars of interest. A confusing loan software interface sent the entire 900 million principal instead. Then a court ruled, for a while, that some recipients could keep it.
Sell One Share for 610,000 Yen. He Typed 610,000 Shares for 1 Yen.
A Mizuho trader inverted a price and a quantity. The order was for 41 times the entire company's shares. Then the system would not let him cancel it.
Cloud and Internet Backbone
The shared layers most of the internet quietly runs on, and what happens when a region, a CDN, or an identity service has a bad morning.
The Largest IT Outage in History Was a File Full of Zeros
On July 19, 2024, a single content update from a security vendor crashed 8.5 million Windows machines in 78 minutes. Flights canceled, hospitals and broadcasters disrupted, and a fix that had to be applied by hand. This is the complete technical account of the CrowdStrike outage.
The Company That Deleted Itself From the Internet's Map
On October 4, 2021, a maintenance command withdrew Facebook's routes from the internet, and the internal tools, the communications and the site access needed to fix it all ran on the infrastructure that had just disappeared. Six hours, nearly three billion users, and a recovery that had to be carried out on foot.
The Typo That Broke a Quarter of the Web
On February 28, 2017, an engineer running a routine playbook mistyped one command. It removed more servers than intended, and a large slice of the internet that quietly runs on Amazon S3 went dark for four hours.
Tiny Bug Hall of Fame
Counters that ran out, clocks that drifted, and units that disagreed. Small causes with no small errors, from Mariner 1 to the year 2038.
Games and Launch Days
Launch nights with pre counted demand, always online requirements, and the load spikes a calendar publishes months in advance.
