The Batch Job That Locked Millions Out of Their Own Money

The invisible machinery of a bank

Most people picture banking as real time: you tap a card, money moves. Underneath, a great deal of the work that keeps accounts correct still happens overnight, in batches. Scheduled jobs apply the day's transactions, update balances, move payments in and out and prepare the next day, and a batch scheduler runs them in the right order, because most of them depend on the ones before. Ulster Bank told the Northern Ireland Assembly in July 2012 that the group processed about 20 million transactions on an average day this way, and that the process had run without serious failure for 25 years.

What the regulators found

On 17 June 2012 the group's technology function upgraded the batch scheduler software that processed overnight updates to NatWest and Ulster Bank accounts, moving from what the Financial Conduct Authority's notice calls Version 1 to Version 2A, a version carrying a patch. On 19 June the team backed the upgrade out. The notice is precise about why that was the fault line: the consequences of backing out Version 2 to Version 1 had been tested, and the consequences of backing out Version 2A to Version 1 had not. The two were not compatible.

What followed was not a pause. A significant number of jobs failed to appear in the batch queues and the unprocessed jobs began to multiply. Because each night's processing assumes the last one finished, the backlog compounded: by 21 June Ulster Bank's batch processing was more than a day behind, so the next day's run began before the current one had finished, and multiple days of files and jobs interfered with each other in the same queues. Clearing that is not a restart. It is replaying days of financial processing in order without applying anything twice.

At least 6.5 million customers in the United Kingdom were affected, 92 per cent of them retail customers, about a tenth of the population. Disruption to most RBS and NatWest systems lasted until 26 June, and to most Ulster Bank systems until 10 July. In the Republic of Ireland, which the UK notices exclude, the Central Bank of Ireland found that about 600,000 Ulster Bank Ireland customers were deprived of essential and basic banking services over a 28 day period.

Not an old system

The most repeated explanation of this incident is that a decades old system finally gave way. The regulators found close to the opposite, and said so plainly. The FCA's notice records that the incident was not the result of insufficient investment in IT, that the group spends over a billion pounds a year maintaining its infrastructure, that its mainframe technology is under five years old and that it uses up to date software. The fault it identifies is in IT risk management, in controls that failed to identify and actively manage IT risks, and in testing.

What was old was the assumption. The group's own board minuted, with hindsight, that batch processing had been taken for granted while attention went to technology that had failed or to future developments. The scheduler was not neglected because it was ancient. It was neglected because it worked.

The reckoning

In November 2014 the FCA fined the banks 42 million pounds and the Prudential Regulation Authority fined them 14 million, the first joint enforcement action by the two and the PRA's first penalty. Both figures are after a 30 per cent settlement discount. The Central Bank of Ireland separately fined Ulster Bank Ireland 3.5 million euro. The group had already provided 175 million pounds in 2012 for customer redress and related costs, which is a different number again.

What it teaches

A rollback is a change, and it deserves the rehearsal the change got. Backing something out is routinely treated as the safe direction, which is how an untested path ends up being taken under pressure, by people who believe they are returning to a known state. Test the reverse, from the version you will actually be reversing.

Sequential systems do not fail gracefully. Where each cycle depends on the last, an outage does not cost you the outage; it costs you the outage plus the pile behind it, and recovery is arithmetic you do not control once it starts.

And a system that has worked for 25 years accumulates confidence rather than attention. The batch run was invisible to customers right up to the moment it was the only thing that mattered, and the board said as much afterwards in its own words.