While writing this blog, Hurricane Harvey has been dropping several feet, not inches, of rain on the Southern Texas and the surrounding Houston areas. As I sit and watch the flooding, the rescues going on, and as I reach out to friends and family in the Houston area, I pray for their safety and their recovery. I urge you to prepare for both your immediate safety and the recovery through these events. Don’t know where to begin? As someone who has experienced five catastrophic events as AAA Ambulance Service’s chief of information technology and communications, I can’t stress enough how important it is to plan, test and test again.

Hurricane Katrina Provides the Opportunity to Rebuild our Communications System

In 2005, Hurricane Katrina hit the south with winds exceeding 100 miles per hour (mph) as far inland as our city of Hattiesburg, Mississippi. Unfortunately, there was loss of life and damage to property that is still visible as you drive around the coastal areas today. This catastrophe led to the rebuild of our communications system from a clean slate. Katrina gave us the opportunity to redesign our antiquated circuit line connections to more robust RF connections to our towers spread across our service area to building a control tower near our office for redundancy and control features. We also had the opportunity to start rebuilding our networking to include servers that are more robust and switches to run our CAD and Billing Pinpoint Software (we purchased the software in 1997, and ZOLL bought PinPoint in 1999). We have continued to grow the software over the years to include the many products from ZOLL.

Lightning Strike Impacts Computers & Switches

The second event that occurred was less costly to rebuild from, but it was a different type of event. Because we now had the control tower near our building with many connections, it became a 150-foot lightning rod. Yes, we encountered a direct lightning strike to the tower and thus power surges affected several components. Fortunately, this one event didn’t take down the system nor did it affect too many systems. The isolated event did impact a few computers and switches. All of which were replaced quickly with minimal down time.

Software Integration Helps AAA Operate Smoothly After Tornados & a Fire

The third event did have significant impact to everything. We had a fire at our office near the server room and had to evacuate the entire dispatch, business office and administration. The fire caused us to relocate to four different places around the city. Fortunately, no one was injured. The total integration of our ePCR and CAD/Billing allowed our crews to continue to function on calls as if a fire hadn’t happened. By staying on the ePCR, the crews were able to document calls, signatures, etc.; and when the servers were restored, the CAD records updated, and the calls on the tablets could be merged and submitted to the server as normal. Next, we restored the billing systems and payments continued. The CAD was down 36 hours, and the billing office was out for five days without a single loss of ePCR or any data. The integration of the systems made it work.

The fourth event and the fifth event are tied together by type; however, they were different by comparison of destruction. We had a tornado in February of 2013 and another in January of 2017. During both, we were working on our CAD/Billing and ePCR software without incidents. We never missed a single trip or ePCR record thanks to the integration of the products that make it all work during adverse situations.

We have since transitioned to a virtual environment with on-site disaster recovery (DR). The on-site is a real-time replication located in a hardened building that is also environmental controlled with emergency power. Testing your emergency power to test the limits and loads is a necessity; we test this system frequently with recover and restore functionality in the VM products. While you take the time to adequately test, make sure you have enough fuel to last the duration of a catastrophic event.

Lastly, having the offsite backups is great, but let me ask you a few questions: If you had to restore them, what hardware would you use if you could not get to your existing equipment? If you had to order it, how long would it take to arrive? What would your downtime be? Where would you relocate to plug in the new servers? These are questions to ask now before they become the priority and before the event happens. Develop your internal checklist to verify your redundancy and your contingency plans.