Many have replaced tape archiving with electronic data backup and added redundant voice and data lines or satellite communications systems. Power generation capabilities have been improved, and some companies have even dug wells in an attempt to ensure that they have a reliable water supply. New contracts have been signed with disaster recovery providers.
But Katrina's lessons have exacted a financial and emotional price on many of the people who have strived to reassemble and improve their IT operations over the past year.
"The hardest part of this is fatigue," said Kevin Bassett, who manages IT at Kirschner & Co., a New Orleans-based furniture retailer. Bassett added that it has taken an unrelenting effort to deal with the professional and personal challenges caused by the storm. "We've been doing this every single day for a year now," he said.
Bassett is a member of the National Guard who was called to duty just before Katrina struck and assigned to work in the Louisiana Superdome, where thousands of people took shelter from Katrina and then waited for days to be evacuated. In a car, Bassett provides a tour of once-flooded neighborhoods, where the devastation extends mile after mile. In St. Bernard Parish, just outside of New Orleans, he points out his uncle's house, which was covered with water and now stands empty and ruined in a deserted subdivision. Not too far away, where the waters didn't rise as high, his grandparents are rebuilding their home.
Kirschner runs the IT operations that support its retail business from a 250,000-square-foot warehouse in New Orleans. After the levees along some of the city's canals failed, the warehouse became an island in a six-foot-deep lake and was completely cut off, Bassett said. The steps he has taken since Katrina include installing redundant communications lines and satellite capabilities.
"People are under a lot of personal stress," said Paul Barron, CIO at Tulane University in New Orleans. "There are people working much harder in their jobs here because there are just fewer people, even though we're advertising for [more workers]."
Tulane, which runs one of the larger data centers in the New Orleans area, has about 70 IT staffers but is 10 people short of what Barron said he needs. And things have changed drastically for many of the school's employees. "A lot of people in the Tulane community and technology services lost everything," Barron said. Some suffered deaths in their families, he added.
The situation in New Orleans remains far from normal, according to Barron. The population is only about half of what it was, and the available housing that remains intact is expensive. "The city is not so easy still," Barron said. That makes recruiting employees from outside the area a difficult prospect. "I desperately need some DBAs but can't find them," he said, referring to Oracle database administrators.
In addition to dealing with on-the-job recovery issues, many IT workers still have to contend with ongoing personal concerns, such as working with contractors to rebuild their homes and trying to resolve insurance issues. Some continue to live in temporary housing, while many others have simply left the region, according to IT executives.
"There are more jobs than people," said David Erwin, CIO at Adams and Reese LLP, a New Orleans-based law firm that also has offices in Washington, Houston and six other cities. Erwin said companies in the New Orleans area are competing with one another for the available IT workers.
"We are in this situation where we are hiring each other's folks," he said. "It's a tough job market."
Katrina also has made it more expensive to operate IT facilities in the region because of decisions that executives have made to fund improvements in disaster recovery.
For instance, Tulane has been spending about US$12 million annually on IT, but added disaster recovery costs will increase annual spending by $500,000, Barron said. In part, the increase will help pay for a new contract with SunGard Data Systems Inc.'s Availability Services unit, which is providing the school with a mirrored Web site, e-mail fail-over and resumption of data processing.
Tulane could easily spend more on disaster recovery if it moved from backing up data on tapes to online backup, Barron said. But, he added, the cost of the bandwidth that would be needed to support the electronic data transfers is too expensive.
Barron said that at some point, Tulane will study outsourcing, but it isn't on the horizon now. "What's going to push us toward outsourcing is going to be our inability to recruit people with the skill sets necessary to do the work," he said, adding that a job-market turnaround will depend on New Orleans' overall recovery.
IT managers in New Orleans said that many of their supply chain partners outside the Gulf Coast region now are asking them about their disaster recovery plans, since Katrina has raised awareness about that issue.
Sam Canatella, operations manager at Louisiana Steam Equipment Inc., an 80-year-old maker of industrial equipment that is based a short walk away from the Mississippi River in New Orleans, said the large energy firms that make up his company's customer base want more information from him about how to communicate and reach people in the event of another major storm.
Prior to Katrina, Canatella had installed a remote data backup system for accounting information at a facility in Houston, which is connected to the company's main data center via a virtual private network. But he plans to put in a new system over the next year that will allow data replication between three sites: company headquarters, the facility in Houston and another one in Mississippi.
Disaster recovery was always "a side item," prior to Katrina, Canatella said. But the hurricane made him realize that his company's data may be its most important asset. If Louisiana Steam Equipment loses products stored in a warehouse, "I can just call the factory in Michigan and order more if I need to," he said. "But I can't replace the information in the computer."
When it's suggested to IT managers that New Orleans may have unique disaster recovery needs because of its low elevation and location, the response is typically a weary smile. A variety of events can replicate what they experienced -- tornadoes, earthquakes, acts of terrorism -- and they said Katrina's most important lesson is to have a plan that takes into account the utter failure of everything.
"What nobody anticipated was complete isolation for a week," said Don Chenoweth, CIO at East Jefferson General Hospital in Metairie, La., which borders New Orleans.
East Jefferson, one of three hospitals to stay open during Katrina and its aftermath, was built to survive. The hospital opened in 1971 and incorporated the lessons learned from Hurricane Betsy, which had torn up the region six years earlier. The hospital was built at sea level, which is relatively high ground considering that there are adjacent areas six feet below sea level. The facility has 15 generators, redundant communications lines and its own water well.
On his computer, Chenoweth pulls up photos of Katrina showing the hospital completely surrounded by water. Water literally came up to some doors but didn't make it into the building. When the storm cut external power, East Jefferson's generators kicked in, but then wind and water damage knocked out the electronics of the generator that supplied its data center. Battery supplies gave IT workers enough time for an orderly shutdown, Chenoweth said. Another generator failure knocked out the data center's air conditioning. Fortunately, the hospital's well continued to work, enabling people to take showers and flush toilets.
East Jefferson's data center was out of commission for four and a half days, Chenoweth said. External voice communications were knocked out, but with the exception of a 12-hour period, the hospital had Internet access.
One thing Chenoweth discovered -- and other IT managers said they found as well -- was that it was easier to make a call on cell phones that didn't use a local area code because calls are routed differently on the nonlocal cell phones. Consequently, the hospital has purchased a number of cell phones with Baton Rouge area codes for emergency use.
In addition, East Jefferson now has its communications lines connected to three separate BellSouth Corp. access points to provide triple redundancy. It has also installed a satellite connection, and Chenoweth said he is continuing to look at ways to strengthen communications, such as possibly using carriers that don't route lines through local connections.
"What we're trying to do is create a situation here where we have five or six ways we can communicate instead of just a couple," he said. "That was just a huge lesson, I think, for just about anybody out here."
Children's Hospital in New Orleans also has installed satellite communications capabilities since Katrina, and it has sunk a well to boost its water supply, said Mike McSweeney, the hospital's IT manager.
In many cases, technology issues will be resolved long before the city itself is fully rebuilt. For the region as a whole, the IT managers interviewed last week generally talked in terms of five-year recovery time frame. But many said they're uncertain what New Orleans will look like in the years ahead.
"I think there is going to be a lot of change in the city," said McSweeney, who added that things are still difficult for people in the city. "It is tense, and a lot of people are frustrated with insurance companies," he said. "That has been so stressful, for so many people."
Not too far away from Children's Hospital is Loyola University. Bret Jacobs, Loyola's CIO, regularly checks the progress of the neighborhoods that adjoin the university's campus.
In one neighborhood that Jacobs drives through, there are contractors' signs at almost every house, and workmen everywhere. "Everywhere there's rehab work," he said.
As Jacobs drives farther away from the campus, keeping an eye out for potholes that can appear overnight, the brown, rust-colored lines that show floodwater levels are still visible on many buildings, gradually rising higher. In areas where the flood waters were highest, fewer contractors are at work. But Jacobs points to a house with a sign that says "We will rebuild." Looking at the devastation around him, he said, "there is a real sense of 'we have to.'"