Over the years I have had the benefit of working as a security practitioner in numerous corporate environments, not as a consultant but as a defender working with a team to shore up the security of the organization. Along the road, I have developed a healthy disdain for meteors. Time and again this menace would loom in the sky above, ever threatening to reduce our building to ash and rubble. Hopefully, it wouldn’t be quite as dire as what created the Vredefort Crater 2 billion years ago.
What am I talking about? Well, simple really. As anyone who has worked to build the defenses of a company can attest, business continuity planning (BCP) and disaster recovery (DR) is an unenviable task. It is a necessary exercise of course but it’s much like manual log reviews: no one wants to do it.
The question that has lingered in my mind for years is why is it always a meteor? Every single time there is a BCP or DR document it has the stalwart “smoking hole” scenario. Why? Are we so bereft of imagination that the most extreme case for DR is a meteor hitting the building? Not being an astronomer myself, I can’t say for certain, but I can make an educated guess that if a meteor struck the data center, that particular recovery would not be something you’d be overly concerned about. I watched Deep Impact.
I have seen my share of disaster movies and I’d hazard that the odds are not in our favor. The other wrinkle that has always bemused me is the line that I have seen all too often as a step for recovery, “buy laptops from Best Buy.” Um, if everyone has a similar plan in their BCP and DR plan, one would hope that Best Buy has some massive inventory stocked. This isn’t a practical thought process.
In addition to the meteors and the laptops, there was one other piece to the BCP and DR stories that I have never fully understood. In at least two different companies that I had done some work for, they had a failover facility. In both cases, this location had empty desks ready for people to set up shop in the event that something horrible happened to the primary location.
The difficulty I had was that the backup finance systems in both cases had a remote dialer attached to them so that people could manage them remotely if the network was down. I took down the number and dialed into it from a second location. Sure enough, in both cases, it let me straight in. Didn’t even challenge me. No password let alone MFA. Both times these devices were taken offline.
What may have seemed like a good idea to the Finance team was in fact opening a massive hole in the network. The failing, in this case, was that the security team, of which I was a part, had not done a good enough job to educate the wider corporate audience on security awareness.
When dealing with BCP and DR there needs to be a concerted effort to do it well. To paraphrase the household name in home renovations, Mike Holmes, “Do it once, do it right.”