Disaster Recovery

As a small business owner, it is important that you understand the differences between Backup, Disaster Recovery and Business Continuity. Not only do you need to understand those concepts, you need to have a plan if you want to be sure that your small business can survive a disaster, even a minor disaster such as a server crash.

In the not so distant past, solutions that protect businesses from failure in the face of disaster were out of reach for most small business owners. But as technology has advanced, even a small business can afford to put systems in place to be sure that the business will recover quickly from even the most serious types of disasters. In fact, today small businesses cannot afford to be without such a system.

As a small business owner, you should have a three goals - making sure that you NEVER lose critical data, minimizing downtime and recovering as quickly as possible in the event of a disaster.

According to a 2003 study, as many as 43% of all companies who experience a "major loss" of computer records never reopen, 51% close within 2 years and only 6% survive long term. Even if you are able to reopen, how do you regain the trust and confidence of your employees and your customers?

According to the Department of Homeland Security, small businesses account for more than 99% of companies with employees, 50% of all private sector workers and 45% of the nation's payroll. Our local and national economies clearly depend on small businesses being prepared for disasters. Commitment to planning today will help support employees, customers, the community, the local economy and the country.

Let's look first at business continuity, which is the most comprehensive of these three areas. Business continuity requires the creation and validation of a practiced plan for how your organization will recover and restore partially or completely interrupted critical function(s) within a predetermined time after a disaster or extended disruption. This disruption could be any number of events from employee sabotage to natural disaster to terrorism to equipment failure.

Now, this is much, much more than a just a discussion about IT issues. Not only do you need to consider how you protect your IT infrastructure and data, but you should also consider what you and your employees should do if a disaster should occur while you are at work.

Much like the plan you may have developed at home when your children were young, you should have a place to meet in the event of a building evacuation and a way to contact all employees and their emergency contacts. Do your employees know where to take shelter if there is a tornado? Should you have employees trained in first aid and CPR? If you have employees who are trained, do the other employees know who they are?

What will your insurance cover in the event of a disaster? What will you do if one of your major suppliers has a disaster? Do you have alternate suppliers for everything that is critical to your operation?

What about planning to avoid disruptions in the first place. I'm sure you all have locks, perhaps an alarm system, a sprinkler system and fire extinquishers. Do your employees know where the fire extinquishers are located? Are any emergency phone numbers that might be needed easily available? Is your network secure and able to withstand attacks by hackers?

You can find a complete discussion of steps to take to develop your business continuity plan at the Department of Homeland Security, as well as a sample emergency plan.

So, how is disaster recovery different from business continuity? Well, disaster recovery refers to the process, policies and procedures for restoring operations critical to resumption of the business after a natural or human-induced disaster.

Now, by natural disaster, I'm not talking just about floods, hurricanes, earthquakes, tornados and the like, but also things like fire or equipment failures. Then there are human induced disasters - disgruntled employees who may delete critical information from your system, an employee who just makes a mistake and deletes the wrong thing, hackers, theft or acts of terrorism.

So what would you do if you suddenly realized that you have just experienced a major disaster? Over my 22 years at KI, we have had clients that have experienced fire, flood, theft, and of course, equipment failure. I am happy to report that all but one of them did survive, but I also believe that all of them would tell you that it wasn't easy, even the two who experienced rather minor fires.

We did have three clients experience major fires where their buildings burned to the ground. While all three of them did reopen, they are also all three labor unions, which have a very different business model than you probably do. I don't know that my own business could have survived under the same circumstances - complete loss of all electronic and physical records.

That brings us to backup. Very simply a backup is the process of copying your data to a safe medium for recovery in the event of data loss. Of course, if you want to be protected in the event of a disaster, you will need to have a two-step process. The first step copies the data creating the on-site backup; the second step gets a copy off-site.

Perhaps you have a backup system is place. Maybe you even have a disaster recovery plan. But any plan needs to be re-evaluated from time to time to make sure it is still meeting all of the goals.

You can't possibly have a disaster recovery plan or a business continuity plan until you first have a backup solution in place. So, let's take a look at how backups have traditionally been done and how they can be done better.

Traditionally, the backup process has been scheduled to occur nightly. Someone must be responsible for making sure that the correct media is in the drive at the end of the day. The two-step backup is accomplished by assigning someone the task of manually taking a recent backup off-site. Using this methodology, you are relying on someone to make sure that the media is rotated and handled appropriately.

If you want to have any assurance that the backups are working properly, that person needs to be trained to look at the results of the backup checking for errors and they should be performing periodic test restores. We do have clients who are able to perform these tasks, we have many more that have no idea if their backups really worked, would not have any idea how to do a test restore, and others who make no attempt to rotate the media or take media offsite. And even those who do make this process work, may sometimes find that the backup media they need is offsite when they need it to restore a file.

Recently, we have worked with three companies who thought that they had good backups, when in fact they had no backup at all, because the backup jobs had never been created, or had backups that were failing. One company had even had their previous company tell them that they shouldn't worry about the error messages that were being generated by the backup software. My techs will tell you that I don't like to see ANY errors on backup jobs!

Another problem with both of these systems is that they capture only one snapshot per day. If your backup is scheduled at 10pm - you have a picture of 10pm. If you work hard all day and a fire starts at 4:30pm, you risk losing the entire day's work. And if when that fire starts, last night's backup is in someone's purse and they are forced to evacuate without it - you will most certainly lose more than that one day.

Historically backups have been done to tape. Tape has a number of limitations, they are slow to backup and slow to restore. We have had clients with large amounts of data that were unable to backup all of their data between the time the employees left at night and when they arrived the next morning.

Tape media degrades over time and is greatly affected by the environment in which it is stored. So if you have been putting the backup tape in your pocket and leaving it in your car during the summer and winter, you probably have tapes that cannot be relied on. In fact, estimates are that anywhere from 42 to 71% of tape restores fail.

Tape drives are expensive, particularly as the capacity increases. And increasing the capacity is difficult. You can't just buy a bigger tape; you need a tape drive that will accommodate a tape with more capacity. So typically, people make the decision to do only a partial backup. This decision leads to the possibility that some critical information is not available when you need it because it was inadvertently left out of the list of files and folders to be backed up.

Now maybe you are wondering why it is a bad thing to just backup your data. You are 100% sure that you know where your data is and you just want to back that up. If you have a server and the server fails in a way that requires that the system be completely reloaded, it will take about 4 hours for the operating system to be reloaded. Then you have to restore the data, which won't be useful until you restore applications. Then we need to be sure that all of the users are set up and set up properly. Oh, now you want to print - we have to reinstall the printers first. It seems simple, but in fact it will take at least 2 days before the system is running acceptably and probably longer before it is running smoothly.

Tape formats are usually proprietary - you need the same type of drive and the same software to restore the data. While that isn't an issue when your system is new, it may be very difficult and expensive to get your data back if your server and tape drive are destroyed. Data backed up to tape is rarely encrypted because that increases the amount of time that the backup takes, so if your tape falls into the wrong hands, your data is easily compromised. On the plus side, tape media is cheap and small - so it is easy to toss in your purse or pocket and leave in your car.

Over the last 3 to 5 years, there has been a transition to hard disk based backup solutions. There are many, many products to choose from and the features of each are a bit different. But generally, this type of solution offers much faster backup and restore times and the ability to increase the capacity easily. Simply buy a bigger hard drive and slide it into the drive tray. Many solutions use standard Windows file systems, and I would recommend that if you are looking at this type of solution that you insist on that.

Like tape, you still need to be sure that someone removes last night's disk and replaces it with tonight's. You also need to be sure that someone is responsible for keeping a current copy offsite. And if the drive falls into the wrong hands, it is even even easier than tape to get the data off. Most solutions are also much larger than tape media, so it may not fit in your pocket or purse, which may reduce the likelihood that you will actually take it offsite.

So here is the problem. Some of you are thinking that what I have just described is a pretty good backup system. You, or someone in your office, are very disciplined and you know that your backup is taken offsite every day. And you firmly believe that a good backup system is good enough. And maybe for you it is. But, let's look at the criteria for a backup system that is a great backup system.

First, the human element should be completely eliminated from the equation. I need to know that my backups are going to happen just like clockwork and it won't matter if half of the office is off work or out of the office.

Next, I need to be absolutely sure that every file that is part of my system gets backed up - every time. I didn't mention this as a problem with the other solutions, but frequently businesses have some database applications, or they are running Exchange for calendaring or email and certain files are always open and never get backed up. This creates problems when the data is restored. I don't want to restore the data and then spend several more hours dealing with the problems created by the fact that a file was open and did not get properly backed up.

The backup should be automated and easy. This insures that the backup always happens!

Backups taken during the day are a huge benefit. I am sure that most of us have worked on some project for significant time and then made some terrible mistake and wiped out hours of work!

Early in my career at KI, I worked all day on a data conversion and program upgrade for a client in Kalamazoo. Everything was complete and working properly. Because the client had limited disk space, it was very expensive in those days, I needed to clean up the files that were left by the conversion process so that they would have enough space to work. Unfortunately, I put a space where it didn't belong and promptly began deleting every file on their system. I remember thinking that if the Lord had any plans to take me off of the earth any time soon, that was the perfect moment! Fortunately, I had a backup from before I began and I was able to stop the process before all of the files were gone, but being able to go back 15 minutes would have been a big improvement!

The impact on your operation should be minimized as much as possible. Ideally, you should be completely unaware that a backup is in progress. If you need to work late one night, you shouldn't be thinking that you have to be sure to be off the system by 9pm because that's when the backup starts.

Flexible restores are a must. You need to be able to restore a partition, folder, file, database, mailbox or message quickly and easily. And should you have a server that is completely destroyed, you should be able to restore the backup to the new server, even if it is drastically different from your old server. Without this important feature, your backup is not very useful a year or two after your server is install. There is little chance that you can replace your old server with an identical one - and why would you even want to?

Better yet, while you are waiting for that new server, couldn't your backup device function as a server? Many solutions available today offer the ability to virtualize your server and have you back up and running in under an hour rather than the hours or days you might wait for repairs or a new server.

When you evaluate an option for off-site backup, there are a number of additional criteria that should be met. For many of us, the idea that our data is going to be sent to an offsite location somewhere across the country is scary. We want to be sure that no one else has access to our data. In fact, for some it is a regulatory requirement that the data be kept secure. So perhaps the most important question to ask is if the data transfer is secure. Can your information be intercepted and stolen?

Then you need to understand how your data will be stored. You should not only be concerned about whether or not others have access to your data, but also is the facility safe - able to withstand disasters or loss of power.

You need to understand what will happen in the event of a disaster. If you have a disaster you will need to be able to get the data back quickly, and if you have even 10GB of data, which is a relatively small amount, that cannot be transmitted overnight. So, does the provider have a plan in place to get your data back to you, overnight, without use of the Internet?

And if your initial backup is large, do you have the ability to copy it to a hard drive and send it to the data center for loading.

You should also consider where the offsite location is relative to where you are. If you are just sending the data across town and there is an event that causes widespread damage to our area, your data may not be as safe as you had hoped. Utilizing data centers in geographically separated areas provides additional protection from disaster.

Offsite storage must be reasonably priced - and believe me, there are some solutions that are decidedly not reasonably priced.

Lastly - depending on your situation - you may also need to be concerned with various regulatory requirements, such as HIPAA, Sarbanes-Oxley and Graham-Leach-Bliley.

Perhaps you are thinking that anything that will do what I have described is likely to cost much more than you can afford. But before you come to that conclusion, just how much can you afford?

I've come up with a list of questions that will help you think about the value of such a system in a way that you probably aren't accustomed to.

First, how much revenue, gross AND net do you generate in year, quarter, month or day? If you are an attorney, CPA or other professional, how much can you bill per hour?

How much do your employees cost per hour and how many of them do you have? Don't forget the cost of benefits such as vacation and sick time, insurance and social security.

How much of your revenue and how much of the work done by your employees is dependent on your IT infrastructure?

How will a failure - even a short lived failure - be perceived by your customers? By your employees?

How quickly can you recover lost files?

If a server fails, how long will it be before you are back up and running and how much opportunity cost does that represent?

It isn't too difficult for even the smallest business to quickly get to a cost of $1000 per day or more.

By contrast, you can protect your data, implement a solid disaster recovery process and be well on your way to a business continuity plan for as little as $250 per month. Your local technology provider should be able to help you sort through the options and develop a plan that fits your situation.

KI Technology Group has a Backup and Disaster Recovery plan that meets all of these criteria and we would be happy to work with you so that you can sleep soundly at night.