The old saying goes that if you plan for disaster, it’ll never happen. If you don’t, it will. Plan for the worst, hope for the best.This is especially true if you’re talking about your technology and your critical business data.

But if does happen, you certainly want to be ready. The consequences to your business can be catastrophic otherwise.

This story hit home with us. In early June, we had a hardware failure in our main server in our office. We were already a bit on-edge, as our office was just across the highway from an area that was on a “get ready to evacuate” evacuation alert because of the Two Bulls Wildfire in Central Oregon. The server that kicked the bucket is what we like to call a Virtual Host, meaning it held several virtual servers running on it. What that means is that it brought down our ticketing system and other internal services that we use to help our customers. We worked as a team to minimize the impact on customers, and manually tracked tickets and time via email. Some of our systems (including our remote access and monitoring tools, email and phone system) are hosted in the cloud. And while the cloud certainly isn’t perfect, it did help us our here to not have all our eggs in one basket.

We always preach the importance of having good backups, and making sure you are monitoring and testing your backups. We do this as part of our maintenance agreements for many of our clients. And being that we try to practice what we preach, we did have good backups (including good offsite backups), and we got our main services completely back up later that day on new hardware.

The whole process made us revisit our disaster recovery plan and revise our “What needs to be up and running first” list. Our ticketing and scheduling system – a crucial part of our operation – was down for the entire day while we watched progress bars and continued to assist our clients. But sometimes one service requires another to be operational. For example, our ticketing and scheduling server requires our main Active Directory domain controller to be up and fully operational as a prerequisite (which we had forgotten when we tried to fire it up and nobody could log in).

We had a pretty good disaster recovery plan, but it hadn’t been updated in quite a while. After this incident, we worked together as a team to update that plan to come up with what services will need to get up and going first, how fast they realistically need to be back up, and put plans in-place to make sure we could meet those goals moving forward. Some of the plans involved additional hardware, and some of those plans involved moving services offsite (or at least having them mirrored offsite).

We even made sure that we had a plan on which department was going to get the coffee pot filled up (a very important thing when you have a bunch of technicians who are about to have a very long day).

These kinds of things are issues you are facing with your business as well. Think about it for a minute: If your server kicked the bucket, what services would you need up and operational sooner than others and what could you live with being offline for longer if need be? How much would that downtime cost your company in lost revenue and expenses because your employees can’t work and your clients cannot be serviced? It’s a scary number.

Disaster recovery is something you never want to have to deal with, but it’s something you really need to plan for. Every business is going to be different, and is going to have different needs, retention policies, offsite requirements, and services that are critical to their environment. And every one of those critical services has different options for protection in case of worst-case-scenario. From full system backups, offsite replication, email continuity protection, and more, we can help you plan for when things go bad.

Hopefully they never will, but it’s best to be ready if they do.

 

 

 

Photo Credit: jseliger2