As Mardi Gras was celebrated in a still-recovering New Orleans last week, more than a dozen IT managers in Louisiana and Mississippi said they are rushing to build more resilient systems, improve their companies' communications capabilities and apply the many lessons they learned from Katrina.
There's a particularly strong push to increase data backup capabilities, especially via online replication to other sites, and to set up wireless and satellite communications systems, the IT managers said. And the work is being done with a sense of urgency because of worries about this year's hurricane season, even while many employers continue to struggle to find IT staffers to replace workers who left the region and haven't returned.
"I have a great concern for this coming hurricane season," said James Mehaffey, manager of the business continuity program at Blue Cross and Blue Shield of Louisiana. "In the Gulf Coast, our infrastructure is severely weakened."
Many of the people leading the efforts to reinforce IT systems and networks have already been through a lot, both professionally and personally.
For instance, the home of David Scripter, regional IT manager of URS Corp.'s New Orleans operations, was located near a breached canal in the city. "I lost my house and pretty much everything I own," Scripter said, although he added that at least he, his wife and their three young children were all safe and sound.
From a business standpoint, the engineering services firm's offices were inaccessible for about a month after Katrina struck on Aug. 29 and caused devastating flooding the following day. Because its systems were out of reach, URS had to buy new hardware, especially PCs for its end users, Scripter said. The company set up remote offices in Baton Rouge, La., as well as Houston and Austin, and it took URS two weeks after the storm to restore its IT operations.
Now Scripter is back in New Orleans, working to make certain that the company could recover from another catastrophic storm within five days. He said his staff has installed a storage-area network in New Orleans and is investigating how to mirror the data on the SAN to an off-site location in another state. URS is also installing a system that will replicate separate Oracle databases used in 3-D engineering applications to remote locations. "We're trying to get a lot of this stuff in place by June 1," Scripter said.
Familiar Ground
Larry Mayo, vice president of information technologies at Keesler Federal Credit Union in Biloxi, Miss., is working under the same deadline.
Keesler's headquarters building lost its roof and most of its windows to Katrina, but the credit union's data center was spared because it was located between two cement floors in the middle of the building, Mayo said. A generator worked throughout the seven days that the company was without electricity, and Mayo took steps to activate Keesler's disaster recovery hot site in Scottsdale, Ariz.
But the credit union decided not to go live with the hot site because of the widespread failure of voice and data networks in the Gulf Coast region. "Without the communications leg of it, no matter which data center we worked out of, we were going to be dead in the water," Mayo said.
As a result, Keesler is installing satellite communications links as network backups at its 12 locations in the U.S. and three offices in the U.K., he said. The installations have been completed at three facilities, and the remainder are due to be finished by June 1. By that date, the credit union also expects to have finished deploying a converged voice and data network that will replace an aging Cisco data backbone and a private branch exchange phone system. The converged network, which was expedited after Katrina, will be able to work over the satellite links, Mayo said.
The day after Katrina struck, Neal Hennegan, director of technology at insurer Gilsbar Inc., had to drive 60 miles from his office in Covington, La., to find a working phone so he could declare a disaster and authorize hot-site operator SunGard Data Systems Inc. to switch his IT operations to a Chicago data center.
Like many IT managers who experienced the storm and its aftermath, Hennegan can quickly list the things he did right and the things he has since addressed. For instance, he was especially pleased that Gilsbar had tested its SunGard backup capabilities before Katrina struck, because that helped avoid start-up delays at the hot site. But Hennegan said he plans to send IT employees to Chicago in advance of any future storms "and empower them to declare a disaster without talking to us."
Even at a company like Blue Cross and Blue Shield of Louisiana, which had invested heavily in business continuity capabilities, Katrina managed to expose issues. For instance, the health insurer used diverse routing techniques on its networks but wasn't aware that at one point they all went to a common physical site in New Orleans, Mehaffey said.
At the University of New Orleans, IT workers couldn't begin returning to the school's campus until mid-November, said Jim Burgard, assistant vice chancellor for university computing and communications. And it wasn't until the start of January that all of those who remained on the IT staff were back working on campus, he said.
Even now, the university is still running its mission-critical applications at Louisiana State University in Baton Rouge on 25 servers -- some taken from its own data center, and some donated by LSU. But the New Orleans school is installing an uninterruptible power supply and a new generator powered by natural gas at its own facilities, Burgard said. When that work is completed in about six to eight weeks, the applications at LSU will be shifted back to the campus in New Orleans.
But, Burgard added, the university plans to leave the servers in place in Baton Rouge and mirror data to them so that the LSU location can function as a hot site if another disaster occurs. That strategy is also geared toward eliminating some of the problems the university had retrieving its backup tapes after Katrina, he said. Although the school didn't lose any data, it had to wait three weeks to access the tapes, which were stored on the sixth floor of a building in downtown New Orleans
Meanwhile, Burgard is still trying to replace eight IT workers -- nearly one-fifth of his 45-person staff. A recent advertisement for a senior networking position that before the storm would likely have garnered about 20 resumes netted just four, he said. In addition, he received only two resumes for a senior database administrator slot.
Also working to fill IT positions is Ochsner Clinic Foundation, which operates a 600-bed hospital on its main campus in New Orleans and 35 clinics throughout Louisiana. Ochsner lost 23 workers from its 150-member IT staff after the storm, said Bill Saussaye, director of information services operations. The health care provider has managed to fill some application development jobs with local people but is using contractors to temporarily fill positions in networking and communications, he said.
Ochsner is aiming to complete several disaster recovery projects before June. One involves the installation of redundant Freon and chilled-water cooling systems in the company's primary data center -- a plan prompted by its inability at times to generate enough power to chill the water in the existing system after Katrina hit. Saussaye said he is also looking to set up a remote data center at least 100 miles from New Orleans and hot sites for Ochsner's distributed systems.
Jan Rideout, CIO at North-rop Grumman Corp. 's Ship Systems unit, lost her Mississippi home to Katrina's fury. She said that if there's a bright side to the storm for her, it's that the disaster prompted the Ship Systems operation -- which has facilities in New Orleans -- to accelerate a plan to replace networking cables with wireless technology.
Disaster recovery plans have to be flexible and must take an organization's ability to respond into account, Rideout said. For instance, many plans assume that employees will be available after a disaster occurs. But when the air conditioners in Northrop Grumman's data center stopped working after Katrina, Rideout said, there weren't any employees on hand who knew how to gracefully shut down the servers.
That points to the challenges that Rideout and other Gulf Coast IT managers are facing as they scramble to shore up their systems, networks and data-protection mechanisms so their companies can withstand another major storm.
"There is not ever going to be the perfect disaster-recovery plan," Rideout said. "You can't write the perfect script, and the next time will be different. And I don't think we can forecast and predict every combination."
SIDEBAR
Sidebar: Disaster sharpens views of IT's value
One positive legacy of Hurricane Katrina, IT managers said last week, is many business executives' increased appreciation of the value of technology. That might make it easier to take IT in new directions, they added.
That has been the case for Nancy Kuo, IT director at M.S. Rau Antiques LLC, a large retailer in the French Quarter of New Orleans. Kuo said business continuity issues have moved to the forefront at M.S. Rau in Katrina's wake. For instance, the retailer recently decided to move its Web servers to a hosting facility operated by Rackspace Ltd. in Austin.
Kuo said that before the storm, businesses such as hers typically viewed IT as a secondary concern. Often, the technology focus was simply on whether computers were working properly, she said. But now IT is seen in an entirely new light, according to Kuo. "It's way beyond that -- it's about the survival of the business," she said.
Rizwan Ahmed, CIO at the Louisiana state government's Office of Group Benefits in Baton Rouge, said Katrina was "more of an eye-opener for the folks who were in the business area" than it was for many IT staffs. "They always thought that somehow IT people will get things magically done," Ahmed said.
SIDEBAR
Sidebar: New Orleans IT vet: Workers come first
IT employees throughout the University of California's 10 campuses and administrative offices know firsthand what earthquakes can do.
But the university system's IT leaders, whose units support some 121,000 faculty members and employees, still felt they could learn something more about natural disasters from New Orleans' experience last summer.
So at the end of January, John Lawson, formerly CIO at Tulane University in New Orleans, was invited to speak before the system's IT leadership council, which includes CIOs from each of the schools, about his experience dealing with Hurricane Katrina.
"I thought it was just riveting, compelling, and if we don't listen to this man, we're idiots," said Paul Weiss, who heads technology support services for the 2,000 employees of the university system's office of the president in Oakland, Calif. "He's lived reality."
Lawson's most important point, according to Weiss, was that IT operations must first restore contact with employees after a disaster hits.
The University of California system, with its statewide high-speed networks, can easily build alternative systems and create Web sites that can be used as points of contact, Weiss said. Most employees will be able to find some means to access e-mail, he said.
Nonetheless, Weiss said Lawson's remarks prompted a review of the employee communications plan to make sure it's up to speed.
Several IT managers interviewed last week about the state of IT operations in New Orleans agreed that the first action in a disaster should be to check in with employees, because nothing happens without them.
Lawson said in an interview that Tulane quickly created an employee registration process after Katrina hit because "we cared about our employees -- we wanted to know where they were, and were they safe."
Rick Omartian, IT chief financial officer and chief of staff at The Guardian Life Insurance Company of America in New York, said that the most important asset of any organization is its workers, whose safety should be management's top priority.
"The business will quickly collapse without its people, even if the company deploys the most advanced technology, alternative workspace plans and disaster recovery data centers," Omartian said.
The law firm of Adams and Reese LLP, which had offices and a data center in New Orleans, created a disaster relief fund of $200,000 to assist affected employees, said CIO David Erwin.
"There was even a team of people assisting employees with locating housing, schools, churches, banks and whatever was needed," he said.
SIDEBAR
Temporary locations are still home to some IT ops
The physical dislocations that were caused by Hurricane Katrina are forcing IT managers such as Jim Burgard to address problems as basic as how to get their staffers from one place to another.
Burgard, assistant vice chancellor for university computing and communications at the University of New Orleans, struggled after the storm to find a way to get IT workers to Baton Rouge, where the school set up a makeshift data center at Louisiana State University that it is still using. Finding places for them to live there was also a challenge, he said. Several employees who owned RVs used them as their temporary residences in Baton Rouge. Burgard himself commuted four hours a day round-trip between his temporary home in Crowley, La., and Baton Rouge. Now he's grappling with the transportation and housing issues as part of developing a new disaster recovery plan for the university.
Hancock Bank has been running its IT operations from a building in Gulfport, Miss., that housed its training staff before the hurricane, said Rodney Sandoz, senior technology officer at the bank.
The bank's headquarters building, which includes its data center, was rendered uninhabitable by the hurricane. Until Nov. 11, Hancock used hot sites in Chicago and Atlanta to keep its systems running. None of the bank's hardware was damaged in the storm, and it recovered all of its data, but Sandoz said officials are trying to better prepare for future emergencies.
"We are in the process of reviewing what we did and making plans to have a smoother transition," Sandoz said. He added that the strategy will include replicating critical files and information to lessen the downtime associated with data backup-and-restore operations.
In addition, the bank is eyeing satellite communications to augment its voice-over-IP network as part of an effort to avoid losing telecommunications capabilities if carrier networks are damaged. Sandoz said that as recently as late November, some of Hancock's 110 branches were still experiencing telecommunications problems. "When those circuits went down, we couldn't communicate except by radio phone," he said.