The legacy systems problem

A comment I made on BBC 5Live's Wake Up To Money programme has attracted quite a bit of attention. It's quoted in the BBC's report about RBS's recent IT failure:

"We just have a lot of legacy systems out there," said Frances Coppola, a former RBS employee and an independent banking analyst.
"Some of those systems haven't been replaced for a long time," she said.

And my colleague on the programme added the following:

"The bank has given too much priority to grand schemes and acquisitions, rather than running a day-to-day bank," said Alastair Winter, chief economist at Daniel Stewart Securities.

Both of these remarks need further explanation, and to do so I need to talk about what I actually did in banking. I am usually described as a "former banker", but that isn't entirely true. I worked in systems. I started as an IT analyst/programmer, moved on into IT project management, then crossed the fence into the business (doing an MBA in the process) and became a business analyst and project manager working on joint business/IT projects. So when I talk about bank IT systems, I'm not making it up. I really do know what they are like. I worked on them.

Admittedly, I left banking ten years ago, and systems have changed in that time - considerably. But the principal areas of development have been in front-end applications (customer interface) and in settlement processing and payments. The core systems, that run the basic banking processes, have not been upgraded. That's why despite the massive increase in processing power and storage capacity in modern IT technologies, banks like RBS are still running massive traditional mainframe computers. Their software won't run on anything else. I suppose it keeps IBM in business, but it is not what we might expect for a modern real-time, mission critical banking system.

The existence of ancient "legacy" systems within the modern banking systems architecture is not necessarily to do with lack of investment, as Alastair Winter suggested, though fast growth and acquisitions complicate IT architectures and can make systems vulnerable. I shall return to the likely effect of RBS's aggressive expansion strategy shortly. But the real problem is the size, complexity and criticality of these old systems - plus the fact that many of them are written in progamming languages that are not widely used now, so there are skills shortages among IT staff. Many of these systems are also poorly documented (comments in code were something of a rarity when these systems were written) and their functions are poorly understood. Replacing them without affecting functionality is therefore not an easy task. Even replacing a single program can have adverse effects if the program is not properly understood, as I discovered when my team replaced a start-of-day batch program in a systems upgrade on one occasion: the old program was complex and poorly documented, we (perhaps inevitably) failed to understand exactly what it did and we therefore subtly changed its functionality without realising it. Fortunately the changes we made didn't cause major problems, and it wasn't a major retail banking system anyway. But imagine that, scaled up to an entire suite of retail banking applications running millions of bank accounts, with trillions of transactions going through every day? No wonder banks have shied away from replacing these systems. The risks, and the associated costs, are terrifying.

However, there is another reason why banks have avoided replacing legacy systems. Banks are driven by the need to make profits, and profits don't come from upgrading basic infrastructure. No, they come from new business lines, risky mergers and applications, swanky front-office applications to support fancy new products. Infrastructure is boring and the cost of replacing it is a hit to short-term profits. Therefore banks do whatever they can to avoid expensive replacement of legacy systems. Keeping the things running, patching them up and circumventing them if necessary is the order of the day. Banks that are expanding very rapidly are particularly prone to develop a patchwork of unintegrated and incompatible systems, because every time they acquire a new business they also acquire the systems that support it and they seldom allow the time to integrate these before moving on to the next expansion opportunity. I saw this happening very clearly during my time at UBS, and I have no doubt that RBS was doing the same during its fast expansion in the run-up to its failure in 2008. Piecemeal, incompatible systems are a risk, but banks don't worry about that when the business opportunities are good.

It would be nice to think that if banks that are expanding won't invest in new core systems, maybe damaged banks that are trying to reduce their risks might do so. Sadly that isn't true either. Core system replacement is very expensive, and damaged banks are trying to reduce costs as well as risks. Very expensive IT infrastructure projects simply aren't acceptable to management or staff when their jobs are on the line. So since 2008, despite their supposed commitment to reducing risk, banks such as RBS still haven't addressed the legacy system problem. They've reduced their balance sheet risk, but not their operational risk.

But as I noted above, bank IT systems have developed enormously in the last couple of decades despite the existence of old systems. That is because to avoid replacing legacy systems banks have adopted a practice known as "wrapping". This approach was recommended by consultancies as a lower-cost and perhaps more importantly, lower-risk approach to improving banking system functionality. Basically you treat your legacy system as a "black box" which remains untouched at the core of your system: around it you create a "shell" of additional applications that provide your customer interface, your straight-through settlement processing, your point-of-sale functionality and your real-time updates (yes, this functionality can be added even if the "black box" is a batch system). This is akin to the way in which off-the-shelf package applications are typically customised, but it is of course on a much larger scale.

The "wrapping" approach has been all too successful. Customers now expect real-time banking services twenty-four hours a day: in fact the entire economy has come to depend on them. But the core banking systems still do not support this. The functionality to support real-time banking services is in the shell. So as far as customers are concerned, the shell applications ARE the banking system. The balances that we see on our internet banking screens ARE our real balances, to us - but not to the bank. To the bank, the real balances are in the core system, which is invisible to customers and is probably updated in overnight batch processing, not in real time as customers think. There is divergence between how financial information appears to customers and how it appear to the banks themselves. And this opens up the possibility of system errors causing data corruption, with a serious impact on customers. Just to give an example of what MIGHT happen, suppose that a customer deposits cash into their account. That cash shows up immediately in their balance in the customer-facing applications - online banking screens, telephone services, branch information points. And they can use it for payments. But the core system doesn't apply that cash to the balance, or the payments made using that cash, until the overnight process. If that process fails, or the customer application doesn't transfer the information correctly, the result could be an imbalance between what the customer thinks they have in their account and what the bank says they do.

Over time, the "shell" of additional applications becomes ever larger and more complex, and customers come to rely on it more and more. And because the core uses old technology, and the shell applications are much newer, there are problems with technological compatibility and connectivity. That increases the risk of failures. The more fragmented your systems architecture, and the more it relies upon stable interconnections between different technologies, the riskier it becomes. Efforts have been made to reduce risks, of course - to improve the stability of connections, and to provide "fail-safe" backups for critical components - but in the end the "pasta rule" still applies: the more your systems architecture looks like spaghetti, the higher risk it will be.

So the problem for banks is the balance of risk: the risk of replacing a critical legacy system and it all going horribly wrong (and costing a fortune) versus the risk of increasing instability in an ever-more-complex systems architecture founded on diverse technologies. It's rather like the risk of a major operation (which could result in death but might lead to full recovery) versus medical treatment to control symptoms - you get iller but you don't die, at least not for a while. But eventually the operation becomes necessary. The question is whether IT systems in banks have reached the point where radical surgery is the only option.

This is not simply a question for banks to consider. It is a matter for political consideration. The economy as a whole has become critically dependent on the real-time performance of bank systems. If the payments network goes down because of the failure of one component - a large banking system - the entire economy stops working. The RBS system was only out for 3 hours, but in that time people couldn't make debit card payments, they couldn't get cash, automated payments weren't made, wages weren't paid into accounts. Just imagine what would have happened if it had gone down for days - as nearly happened in 2008. Those who think that RBS should have been allowed to fail do not understand how damaging that would have been to ordinary people and businesses. Even twenty years ago, the damage from a three-hour outage in the late evening would not have been so extensive.

The increasing fragility of banking systems poses a real risk to the economy as a whole. Unless this is addressed, we run the risk of a major systems failure in one or more banks at some point. The two RBS failures are warnings. Banks and politicians need to address this problem before there is a real disaster.