The company: TFI Resources, Inc. (now known as “People 2.0 Recruiter Services”)
What they do: Service provider for companies that choose to outsource their “employer of record, payroll funding, and payroll processing” [2] responsibilities
Location: Texas
Disaster: Hurricane Ike
Date: September 1, 2008 to September 15, 2015
Category: Category 4 Hurricane
Areas affected: Cuba, Dominican Republic, Texas, Haiti, Bahamas, amongst others
Damage caused: $37.5 billion – “the third costliest Atlantic hurricane” [3]
DR infrastructure: a 10 x 10 leased office in Austin with a server tower, Internet connection (shared with the building services), a communication cabinet that configured a VPN connection to the collocation facility, a server tower, 6 laptops and two printers, and backup tapes
Response to the disaster: [1]
September 3 (Wed): Hurricane announcements
September 4 (Thurs): Notification to employees about closing of offices; executed the “office closure preparedness plan”; prepared the physical office for hurricane;
September 5 (Fri): IT Manager loaded car with the last set of backup tapes; network connection lost
September 6 (Sat): Hurricane hit early in the morning
September 7(Sun): IT Manager went to the Austin office; took stock of IT staff without property damage, team of 10 deployed to Austin
September 8(Mon): Team arrived at Austin office; Successful in connecting to databases and files; no connection to email servers;
Email replication system had failed; fell back on Plan B for communication-
> Used company emergency website for mass notifications; used fax server and a temporary Hotmail account to reach out to clients
10 people worked for 16 hour days for five days to get everything finally back in order.
After the disaster:
> Leased a bigger space
> Two colocation centers (in Austin and Houston) with sets of backup
> replaced replication service for Exchange – and went with a one with “High Availability”
> Put the primary payroll service for their clients into the cloud and migrated services to a SaaS vendor
> Drill twice with staff, before June each year, at both DR venues
> all major IT changes are backed up and tested for effectiveness and availability
Remarks:
The company had a minimal DR plan in place when the hurricane hit them. But the resourcefulness of the staff and some bit of good luck (that their backup, untested servers were not affected) saw them through.
They have done well in learning from their lessons and having another DR office, using a SaaS vendor and performing tests and drills
They could do better by either getting all their critical data and services up in the cloud (US-based, out-of-state with high availability) or by having one of the DR offices outside of Texas and away from the path of the hurricane.
References:
1. Real Life Disaster Recovery Stories. (2014, October 22). Retrieved from http://www.enterprisefeatures.com/real-life-disaster-recovery-stories/
2. About Us. (n.d.). Retrieved from http://www.tfiresources.com/about/
3. Hurricane Ike. (2017, May 14). Retrieved from https://en.wikipedia.org/wiki/Hurricane_Ike