So recently I had an opportunity to help restore a company in a real world disaster recovery situation. A company we don’t really look after as they had moved back to having an internal IT guy.
You may have seen my post on Restoring Exchange, this fire was why I ended up writing the post. When I was first asked to help recover the business, I was told that they did have a backup. However, when I received the backup it was only the database file, no logs of CHK file. My first concern was that this company used tape backups, and did the person bring the tape? The entire city ended up being evacuated fairly quickly, so it could have been easy to only grab what you needed. (It turns out they had all the backups which made the recovery much easier) 2400 Structures were destroyed in this fire. This led to the question, is the business still standing or was it one of the structures destroyed?
Originally when we set up we suggested using an offsite method in addition to the tape backup they wanted to use such as Veeam (I don’t recall at the time if it was VEEAM, today our go to software is VEEAM). We also put into place a secondary offsite DC, as well as suggested setting up a DFS share, in fact fairly recently when we were doing some work it was discovered that DFS replication was not working properly so we had fixed it. Now the DFS share I would not recommend as a replacement for backups. Nevertheless, it turns out the DFS came in handy in this situation allowing people to access files.
Having the offsite DC allowed us to maintain the user structure and permissions without the need of restoring the backups. This also contained all the necessary information we would need to do a proper restore of exchange.
Overall we were able to get this business back to operational within a 2 day time frame over the weekend. Moving to office 365 allowed us to almost instantly get mail flowing once again. Being in Canada it was a little more work to get the imports of the mail in as we have to use outlook the import tools are not yet enabled in Canada.
So this got me thinking about what we need in a DR plan.
Documentation, is one of the biggest parts of the puzzle. When we talk documentation it may need to be more of how to exactly install the entire infrastructure from scratch. What if you need someone to restore your business as you can’t access your own IT stuck on the other side of a fire. A secure and safe way to access critical usernames and passwords, a list of external IP addresses to internal ips, what ports they use, what they do. This list goes on and on, it should be readable even by someone with only minor IT skills. Today more and more business rely on their infrastructure so much they could lose their entire business without a proper plan.
In this situation some people went North Some People Went South. The people that went North were stuck as they could not get back for a few days.
Offsite Servers and or Backups
Test your plan
Without testing your plan you don’t have DR, you do not know what you are missing, this is another critical part. Know what the steps you need to take will be.