We know as well as anybody that you need to have a good disaster plan with your IT infrastructure. We know because we had disaster hit us years ago and we learned a lot from it and how we can better protect our clients if the situation arises for them. We’ve implemented a lot of those practices with our clients, and while we’re never happy when a disaster hits, we are happy that we get to prove our solutions work – like a recent incident with a server that got a little wet.
We have a client that followed all our best practices in their network room, from proper ventilation, battery backups, proper rack and cable management, etc… .
But as always is the case, you can’t plan for everything. This client is a manufacturing client who recently had to implement their disaster plan because of a server that died because of water damage. In this case, what we figure happened was an air-conditioning unit on the roof of the building had a condensation drain line either clog or break, leaking overnight. The leak wasn’t fast enough to flood and immediately shut down the server or to be obvious that it was happening. It was slowly running onto the power cables and onto the vents of the system, finally getting it wet enough to short the motherboard and power supplies of the system and killing it completely.
Like every business, this client relies heavily on their server infrastructure being up and online as much as possible. Once we realized what was going on, we immediately went into disaster recovery mode. They followed our best practices and had a disaster recovery appliance in-place that we setup that was spared the water damage. We verified backups ran less than an hour ago, and spun up their servers on the appliance as virtual servers. Only a couple hours of downtime (the client is in a remote location so much of that was spent driving there) and minimal data loss when it was all said and done.
While we’re certainly happy the disaster recovery device was spared water damage if for some reason it was wet, too, we could have spun up their servers in the cloud as well. But local servers are always faster than remote servers, so we spun it up locally
When we got them up and running on the BDR device, we took their server out of the rack and assessed the situation: it was toast. We worked with our vendors to get parts for a new server in our hands as soon as possible (next business day after verifying damage). Since they were running on the backup appliance, they were fine for now. We got the new hardware delivered and then scheduled some planned downtime to time to swap back from the BDR device to the replacement server and restore the updated data back to the new hardware.
The transition, recovery was as smooth and seamless as possible and downtime for the client was minimal. Their workers were able to continue working with minimal interruption, allowing them to continue to be profitable to the company.
Do you have a disaster recovery plan with proven, tested results and minimal downtime? Call us today to discuss the best options for your network.