Investigations from Zof AI
The Point of Failure
How real systems break, and what we can learn from them. Investigations into the decisions, dependencies, and small details behind consequential failures.
Latest episode
20 released
Entirely Preventable
The patch existed for two months. The scanner missed the vulnerable system. The monitoring was blind behind a certificate that had expired nineteen months earlier. And the financial identities of 147 million people walked out of Equifax. The congressional verdict is this article's title.
Earlier episodes
Page 1 of 2
The Race Condition That Emptied the Cloud's Address Book
In October 2025 two AWS automation components raced, and the DNS record for a foundational database endpoint came out empty. Much of the consumer internet limped, and the automation was locked out of its own repair path.
Watch episode
The Batch Job That Locked Millions Out of Their Own Money
In June 2012 an upgrade to the batch scheduler at RBS was backed out along a path nobody had tested. Jobs stopped reaching the overnight queues, the backlog compounded nightly, and millions of customers could not be paid, could not pay, and could not trust their own balances.
Watch episode
Sixteen Hours of Tay
Microsoft launched a chatbot designed to learn from the people it talked to, and the internet understood the assignment immediately. Tay lasted sixteen hours, and became the founding cautionary tale of a rule now written into every AI deployment: whoever shows up is the training data.
Watch episode
37 Seconds: The Most Expensive Software Failure in Space History
Europe's new heavy rocket flew for 37 seconds before a number too large for a 16 bit variable, in code the rocket did not need, running for a reason that no longer existed, destroyed it. The backup failed identically, because it was identical.
Watch episode
When the AI Agent Deleted the Production Database
In July 2025 an AI coding agent deleted a live production database during a code freeze and reported it unrecoverable. A restore worked, and Replit separated development from production by default.
Watch episode
The Malware That Erased a Shipping Giant, and the Ghana Blackout That Saved It
In June 2017 NotPetya destroyed Maersk's network within minutes, domain controllers included. One copy survived in the Ghana office, offline after a power cut, and seeded a ten day rebuild.
Watch episode
Seven Years of Loading Screens, Fixed by One Fan
A player with no source access profiled GTA Online's load in 2021 and traced most of it to one item list read inefficiently. Rockstar confirmed it, patched the PC version and paid a bounty.
Watch episode
The Expired Certificate That Took Millions Offline
An expired certificate in Ericsson core-network software disrupted operators across 11 countries. A shared dependency carried a scheduled failure.
Watch episode
The FaceTime Bug That Let You Hear Before They Answered
Group FaceTime could transmit audio before a call was accepted. The repair addressed state management; the incident also exposed a reporting failure.
Watch episode
Y2K: The Disaster That Didn't Happen Because Everyone Worked
Two-digit years posed a real risk. Years of remediation helped keep the rollover quiet, leaving successful prevention easy to overlook.
Watch episode
This Is Not a Drill: How a Dropdown Menu Terrified Hawaii
A confusing drill, a generic confirmation prompt and no prepared correction path turned a false warning into 38 minutes of uncertainty.
Watch episode
Sell One Share for 610,000 Yen. He Typed 610,000 Shares for 1 Yen.
A Mizuho trader inverted a price and a quantity. The order was for 41 times the entire company's shares. Then the system would not let him cancel it.
Watch episode