The bankruptcy of the Lehman brothers was a huge loss for everyone involved, except the super-wealthy who often emerge from financial crises without a scratch. While everyone remembers this crash with post-traumatic clarity, there are other glitches that are lesser known but that were almost equally destructive. This glitch described by Zero Hedge illustrates just how easily global financial collapse can occur.
Imagine a bank with a process called straight-through processing that allows for an entire trade process for payment transactions to be completed electronically without manual intervention. The bank in question used this process because it allowed for same-day settlement. But one day, one of the employees of the company made a change to the system without authorization to add more storage for an already bogged-down system. Yet, although this was a routinely, albeit unauthorized change, this one day it brought everything, including all the trades from that day, to a screeching halt:
As my friend applied his change that morning, he triggered a bug in a notoriously crap script responsible for bringing new data disks online. The script had been coded in-house as this saved the bank about £300 per year on licensing fees for the official ‘storage agents’ provided by the vendor. Money that, in hindsight, would perhaps have been better spent instead of pocketed. The homebrew code took one look at the new configuration and immediately spazzed out. This monged scrap of pisspoor geek-scribble had decided the best course of action was to bring down the production end of the system and bring online the disaster recovery (DR) end, which is normal behaviour when it detects a catastrophic ‘failure’. It’s designed to bring up the working side of the setup as quickly as possible.
The system that the bank used for redundancy made the problem even worse. Instead of acting as an agent of support, the system simply multiplied the problem:
Sadly, what with this system being fully-replicated at both sites (to [cough] ensure seamless recovery), the exact same bug was almost instantly triggered on the DR end, so in under a minute, the hateful script had taken offline the entire system in much the same manner as chucking a spanner into a running engine might stop a car. The databases, as always, were flushing their precious data onto many different disks as this happened, so massive, irreversible data corruption occurred. That was it, the biggest computer system in the bank, maybe even the world, was down.
Fortunately, for the bank and for the euro, the workers were able to recover from this error and get the system back up and running. If they had been just a couple of hours late on their fix, however, it could have led to disaster.
Allegedly (as we were later told), the volume of financial goodies contained within this beast was so great that failure to clear the trades would have had a significant negative effect on the value of the Euro currency.
Imagine the euro being devalued based on a single fat-fingered mistake. This isn’t so far-fetched now, is it? It seems as though global financial ruin is just one glitch away.
Sources:
