In the fall of 2007, I had the opportunity to interview Mukul Chopra, National Association of Children’s Hospitals and Related Institutions (NACHRI), and Mark Rogers, American Alliance for Health, Physical Education, Recreation, and Dance (AAHPERD) about their innovative plan for creating hot sites without spending a fortune.
With growing concerns about business continuity, many not-for-profit organizations are investing in hot sites. According to Wikipedia, a hot site is
“…a duplicate of the original site of the business, with full computer systems as well as near-complete backups of user data. Following a disaster, the hot site exists so that the business can relocate with minimal losses to normal operations. Ideally, a hot site will be up and running within a matter of hours. This type of backup site is the most expensive to operate. Hot sites are popular with stock exchanges and other financial institutions who may need to evacuate due to potential bomb threats and must resume normal operations as soon as possible.”
In the summer of 2007, NACHRI retained the RSM McGladrey not-for-profit consulting team to perform an information technology assessment. As we were reviewing their network map and documentation, we noticed something a little unusual: information about a hot site reciprocal agreement with AAHPERD. I interviewed Mark Rogers, Director of Information Systems at AAHPERD, and Mukul Chopra, Director – Information Technology at NACHRI, about this agreement after the engagement concluded.
Mark Rogers at AAHPERD posted the idea of doing a rack swap to the ASAE & the Center for Association Leadership’s Technology Listserv in the spring of 2005. “AAHPERD was becoming increasingly reliant on our technology, and I just didn’t have a solid business continuity plan for my servers,” noted Rogers.
“I had been thinking about the same thing, because the cost of setting up a hot site with a commercial provider was just astronomical, so I contacted Mark right away,” said Chopra.
“The basic concept is pretty simple, but there are a lot of details to be worked out in implementation. The key question you have to answer is, ‘Do we have the same vision for how the pieces will fit together?’ Fortunately, for AAHPERD and NACHRI, the answer was yes,” remarked Rogers.
Getting Started
Both men agreed that the toughest part of setting this all up, other than making sure the actual technology functions the way it’s supposed to, was getting senior management on board. Even though the two organizations are similar in size, infrastructure, and deployed applications, “it took patience and lots of discussions on both sides to help everyone get comfortable with what we were trying to accomplish,” remarked Chopra. “One of the first things we agreed on was that, at each stage, we would only move forward if the benefits were clear for both organizations. So far, they have been.”
Rogers added, “You have to be careful to be very clear about your goals and objectives. You have think about where things can go wrong and test, test, test. And you have to do site visits. Calls, discussions, and ideas can get you started, but you need to see the physical space.”
Both IT Directors had initially been focused on creating a disaster recovery plan for their servers, but over time, the relationship has expanded to encompass true business continuity. What started as a swap of 14 units of rack space that could be filled with servers loaded up with all the latest patches has turned into a secure LAN with VPN between the locations and includes DNS, DHCP, domain control, Active Directory replication, terminal services, a SharePoint site, virtual directories – even an agreement to provide a limited amount of physical office space complete with phones, fax, and copier access. “It’s not intended to be a complete relocation facility. Neither organization has the space for that. We provide each other with space to ‘tide you over’ on mission critical functions until something more permanent can be arranged. It’s like an insurance policy,” explained Chopra.
Why don’t more organizations do this?
“Historically, hot reciprocal sites are the least preferred of the various hot/warm/cold/reciprocal site options because it can be tough to enforce a stringent SLA [Service Level Agreement]. Sure, that’s a problem for a Fortune 500 company with a large BCP budget, but associations have to be more cost-conscious, which makes this a much more feasible option,” explained Chopra.
Do they have any tips for other organizations that might be looking to imitate them?
Both Chopra and Rogers agreed that the single most important element of a successful reciprocal arrangement is trust. Although the right geographical distance, and organization that’s similar in size, deployment, infrastructure, technology, applications, are helpful, this cannot be done without identifying a partner organization that has similar values, standards, and vision – all things that support a trust relationship.
Rogers pointed out that “Comfort level with your partner organization is key. When you have to activate your BCP for real, you’re not going to be operating under normal conditions. The situation could be bad, and your staff could be very distracted and upset when the plan is activated. The pieces need to be in place ahead of time, and it needs to be simple to run.”
Rogers noted a few additional items. “It’s easier to ‘sell’ to your senior management if you have a clear exit strategy in case it doesn’t work. If this arrangement stops meeting the needs of either of our organizations, we can just come collect our servers. I also recommend that you start small and build out additional capabilities in later phases of the project. Score some successes up front before you try to get too complex. And remember that this is a living system that has to be monitored, tested, and maintained.”
Final Thoughts
Both men agreed that the best surprise in the process has been the friendship that’s sprung up between them, which has provided a valuable opportunity to network, share ideas, and act as sounding boards for each other.
And finally, “People need to understand the difference between disaster recovery and business continuity and be clear about which one they’re trying to accomplish,” remarked Chopra. “And remember that your goal is something that works, not necessarily the perfect BCP solution. Don’t let the perfect be the enemy of the good.”