In case of emergency: new technologyand new threatshave businesses reexamining how they cope with disaster
John Goff
BY ALMOST ANY YARDSTICK, PRAIRIE STATE BANK is not what you’d call a major financial institution. With a handful of branches scattered in south-central Kansas, the bank maintains a small retail business in the GWMA (Greater Wichita Metropolitan Area). How small is small? On its corporate Website, the company’s management proudly proclaims that Prairie State “is the 24th largest bank in the state of Kansas.”
Still, the concerns of the top executives at tiny Prairie State are probably not unlike those of high-powered bankers who run global banking giants. High on that list: how to keep computer systems up and running in case of an emergency. While it’s not especially likely that terrorists will strike Augusta, Kansas (site of the bank’s home office), the Sunflower State does get its fair share of tornadoes. And in 1999, an overflowing Whitewater River swelled clear up to the steps of the main office. “We had to sandbag the front doors,” recalls Chip DuFriend, network administrator at the bank.
Until recently, Prairie State backed up its 16 servers to individual, onsite tape drives. But last year, the bank’s Microsoft Exchange server went down and DuFriend was unable to restore the system with a tape backup. Unsettled by the experience, management at Prairie State decided to try something different, eventually signing up for a subscription service provided by StorServer Inc., in Colorado Springs, Colorado. The service enables the bank to store the data from its servers on one server at an offsite location. DuFriend says he can go online and easily recover lost or deleted data files–a revelation for managers used to working straight from tape backups. Says DuFriend: “This system is a paradigm shift for us.”
Paradigm shift aptly describes what’s going on in the world of disaster recovery these days. Spurred on initially by Y2K and, more recently, 9/11 and the great blackout of 2003, corporate executives are focusing on data protection like never before. According to Stamford, Connecticut-based research firm Meta Group Inc., companies spent just 3.2 percent of their IT budgets on security (employee education, business continuity, and disaster recovery) in 2001. Last year, the outlay was more like 8.2 percent–a dramatic increase.
This newfound interest in security goes beyond increased spending. Advances in technology–and a wider array of threats–have corporate executives rethinking their whole approach to disaster recovery. The days of the onsite, raised-floor room, with rows of clunky tape machines and droning cooling units, are fast disappearing. In their place: remote hot-sites, fail-over systems (backup networks that can be brought online instantly), and Web-based file storage and retrieval. Says Gregg Therkalsen, vice president of business continuity at Hopkinton, Massachusetts-based vendor EMC Corp.: “The idea of backing up info on tape, having human beings put that in a truck, and driving it away…. Well, every customer wants that to go away.”
NOT EVERY ONE GETS CORRUPTED
Ellen Christy can attest to that. Christy, director of information technology at Boston-based private-equity specialist HarbourVest Partners, says the firm used to back up its data to tape onsite. Then, at the end of each day, an employee would lug the tape home and store it on some high shelf. “But small companies grow,” she notes, “and one tape becomes two, two become three….”
In early September 2001, after a six-month process, management at HarbourVest ditched its in-house tape backup, choosing instead to send the data from its 15 servers over the Internet to a remote site. The company stores 100 gigabytes of data at the site, and about half of that is base data; that is, financial records, agreements, and the like. The information, which is backed up nightly, is retained for several months. In case of an outage necessitating a massive restore, Christy says vendor AmeriVault could cut the data to tape and get it to her in around two hours.
The real selling point of the Web-based service, however, is that employees can retrieve lost or zapped data simply by going online. “The most common problem we have is people deleting files,” explains Christy. “It takes 50 percent longer to restore a file using the tape-backup method.” Scrambling through reams of old tape can certainly be a laborious process. Worse, tapes and other backup media are notoriously unreliable. Experts say data gets easily corrupted, and often tape backups just plan fail. “Half the time, zip drives and tapes don’t restore,” insists Wally Beddoe, vice president of operations in the Stamford, Connecticut office of Telekurs Financial. “They can be a big waste of time.”
To address that issue, management at the Swiss-owned supplier of financial data hired a Framingham, Massachusetts-based vendor called Connected Corp. It provides a back-up service that safeguards Telekurs Financial’s distributed data–information not stored on network servers. While Telekurs does back up its commercial data to a remote site in Hartford, employees rely on their PC hard drives to store tons of information–contracts, E-mail, even application code. “All the stuff to support our business is on PCs,” says CFO Mike Stisi. Moreover, Telekurs has an increasing number of employees, including programmers, who work remotely. “The stuff they have on their PCs is scary,” notes Stisi. “It’s hundreds of man-hours’ worth of work.”
The finance chief can attest to just how valuable the company’s new retrieval system is. In December 2002, Stisi came to the office only to discover that the hard drive on his computer had failed. “When my hard drive died, I almost had a heart attack,” he recalls. “Duplicating the information, including customers and contracts, would have been a huge headache.”
Using the retrieval service, Stisi recovered his files in a matter of minutes. Since signing on with Connected, he says he hasn’t had to worry about failed hard drives and flat-line laptops. This, of course, raises the obvious question: Why is the company finance chief involved in such mundane matters as lost Excel files–matters usually left to CIOs and system administrators? “As the CFO, I’m responsible for protecting our assets,” explains Stisi. “My neck and the CEO’s are on the line.”
DIDN’T THINK OF THAT ONE
Before Y2K and 9/11, most finance chiefs were woefully ignorant about the value of digital assets. Even today, few know the difference between a Bernoulli Box and a Bento Box. But with new threats–including terrorist attacks, computer viruses, and infrastructure failures–many CFOs are beginning to at least sit in on disaster-recovery meetings. “Some CFOs perceive disaster recovery as a sunk cost,” says Gary Foster, CTO at Boston-based trade-management services provider Omgeo LLC. “But you have to think worst-case once in a while.”
At Edgar Online, a $15 million (in revenues) supplier of public-company data, CFO and COO Greg Adams is doing more than that. He reviews his company’s written disaster-recovery plan in detail each year. Adams is also apprised of changes in the plan before he files the company’s 10-Qs. “Disaster recovery is critical for us,” notes Adams. “If we’re down, a lot of money is lost.”
After the events of September 11, management at the South Norwalk, Connecticut-based company decided to construct a remote hot-site in Rockville Center, Maryland. The site, which has a backup generator, can restore the company’s main systems in a matter of hours. Edgar Online also maintains a New York-based fail-over for its Website (as the name implies, the fail-over immediately kicks in if the Website fails).
The system was put to the test last August, when the huge power outage knocked out the electricity at Edgar Online’s Rockville office. “During the blackout,” recalls Adams, “we had no downtime.”
Other companies were not as fortunate. Atlanta-based Delta Air Lines, which maintains an extensive disaster-recovery and business-continuity plan (including backup generators for its main and remote sites), was able to keep its planes running and its ticket systems operational during the power outage. But according to Keith Hansen, manager of emergency-response and business-continuity planning at the airline, Delta passengers at a number of airports couldn’t board their flights after the power went out. The reason? Unlike the well-prepared Delta, some airport security systems didn’t have backup generators. “We’re now looking at hub and major airports,” notes Hansen. “If they don’t have a backup [power] system for security, we try to convince them to get one.”
The summer blackout exposed shortcomings in other disaster-recovery plans, as well. Many businesses, for example, discovered that their remote sites simply weren’t remote enough. “It’s all right to have a backup center,” says Lance Travis, vice president of core research at Boston-based consultancy AMR Research. “But if you’re in the same power grid, it doesn’t do you any good.” Moreover, a fair number of companies found that their uninterrupted power sources were designed to run for only a few hours. Now, says Travis, some corporations are looking for remote sites that are so far away they can avoid almost any blackout.
Such a strategy, while prudent, can constrain the amount of data that gets backed up. Delta, for one, performs synchronous backups from a mainframe to a remote site. That’s a massive dumping of data–and one that limits the distance between the company’s remote site and its main data center. As Ray Shepherd, coordinator for business-continuity planning at Delta, explains: “You can push that amount of data only so far.”
COMING: MORE BAD STUFF
Experts believe that increased bandwidth and better compression technology will ease the problem. Already, Connected can shoehorn the information from 15,000 PCs onto one NT server, a fairly remarkable achievement. But supply is barely keeping up with demand. The fact is, companies are producing prodigious amounts of data these days, a trend that shows no sign of abating. “Ten years ago, people were running businesses off what you can get in a laptop today,” says Omgeo’s Foster. “Now we’ve got terabytes of data.”
And while the price of storage technology has come down in recent years, backing up mountains of data can be an expensive proposition. Some companies, in fact, are choosing to discard data after a short time. Haynsworth Baldwin Johnson & Greaves LLP, a Greenville, South Carolina-based law firm, sends new of changed files to a co-location site each night. After 14 days, earlier versions of files get deleted. Skip Lohmeyer, information systems director at the firm, says he’s able to retrieve files, which come across in an encrypted format, using a password and a built-in decryption code.
The cost: $4,400 a month to store 140 gigabytes of compressed data. “When you look at it from a mid- to small-company perspective, it may seem expensive,” grants Lohmeyer. “But [the reality is], you’re going to have a disaster.”
The growing number of nasty computer viruses almost guarantees it. Ten years ago, few corporate risk managers and contingency planners worried much about malicious code. “People thinking about disaster recovery were thinking about a catastrophic fire,” says AMR’s Travis. “They weren’t thinking about viruses.”
They are now. Peter Tippett, chief technology officer at Herndon, Virginia-based TruSecure Corp.’s ICSA Labs, reckons that the average cost to corporations from viruses and worms has been growing at a 70 percent annual clip for the past seven years.
In fact, Tippett claims that well over a third of U.S. companies had moderate to major damage from last year’s Slammer attack. The worm, which essentially swamped corporate networks with traffic, knocked out a portion of the ATM machines in the United States. Continental Airlines went to manual reservations and check-in, and flights were canceled or delayed. All told, Trend Micro Inc. estimates that malicious code cost global businesses $55 billion in damages in 2003.
That number is likely to go up in coming years. It’s not overly surprising, therefore, that CFOs are giving the OK for more spending on network security. And while few finance chiefs are going down into the trenches, some are at least helping draw up the battle plans to stave off disaster. Adams of Edgar Online says he now has periodic discussions about viruses, Trojan horses, and worms with the company’s operational staff. “It’s the CEO’s job to run the company,” Adams says. “It’s the CFO’s job to make sure there’s a company to run.”
FLIRTING WITH DISASTER Average
spending on business continuity, 2002.
In $ millions
General Manufacturing $0.8
Automotive 0.9
Wholesale Distribution 1.0
CPG 1.3
Health Care 1.6
Pharmaceutical 1.7
Transportation 1.7
Retail 1.7
High Tech 2.2
Aerospace & Defense 2.7
Oil & Gas 2.9
Chemicals 3.4
Financial Services 4.1
Telco Services 4.6
Source: AMR Research
Note: Table made from bar graph.
RELATED ARTICLE: Beyond the raised floor.
HOW COMPANIES GET THEIR SYSTEMS UP AND RUNNING AFTER A DISASTER STRIKES.
COLD BACKUP. Basically, an empty room in a building. Once a disaster hits, computers, routers, and telephones are moved into the room. Cold backups, while cheap, require a fair amount of time–often days–to restore full operations.
WARM BACKUP. A room with computers that replicate a company’s existing data center or network. After a disaster, an offsite tape backup is used to boot the computers. Then the hard work of recovery–a process that can take up to 24 hours–begins.
HOT BACKUP. A mirror image of an existing data center or network, with preconfigured systems. Like a warm backup, a tape backup from an offsite is delivered to the data center in case of an emergency. Unlike a warm backup, it takes only a few hours to get these preconfigured systems up and running.
FAIL-OVER. The fastest–and most expensive–backup option. If a primary system fails, a fail-over automatically switches to a standby database, server, or network. A fail-over site redirects requests from the failed system to the backup system. Websites are big users of fail-over.
MOBILE BACKUP. A trailer, replete with computers, routers, and telephones, that can be rolled up right next to a company’s main building or offsite backup. A mobile backup provides additional capabilities in times of emergency, but frees companies from having to invest in permanent office space.
ONLINE/OFFSITE BACKUP. Backing up data or systems to an offsite location via the Internet. With increases in bandwidth, and advances in technology, online backup could be the wave of the future for disaster recovery.
REMOTE BACKUP. Offsite backup locations that are typically 50 to 75 miles distant from a company’s main data center or operation. Since the blackout of 2003, the definition of remote has changed, with some companies investing in backup sites on different power grids. * J.G.
Sources: AMR Research, Webopedia
RELATED ARTICLE: The long, long trailer.
MENTION CALGARY, the city of 400,000 in western Canada, and civil unrest doesn’t spring to mind. This quiet outpost in Alberta is better known for the Saddledome, all-season skiing, and the annual rodeo roundup called the Stampede.
But in 2002, the prospect of civil unrest was worrying managers at Calgary-based Canadian Pacific Railway. At the time, Calgary was getting ready to host a Group of 8 summit, and reports began circulating that protesters were going to try to shut down parts of the city. Recalls Paul Cammack, a manager of the railroad’s contingency-planning management group. “We were concerned employees might not be able to get into the building.”
The civil disobedience never materialized, and the railroad kept running. But other dangers remain, including fires and car or truck accidents in the city center. “A main line runs right by the office,” explains Cammack.
To keep its operations center operating, the $3.7 billion company has invested considerable resources in disaster recovery. In 1999, it constructed a state-of-the-art, remote hot site. Interestingly, the company has also poured money into an empty lot that abuts the site.
Rocky Mountain fever? Hardly. In case of a catastrophe, the company plans to park two large trailers on the lot. The trailers are deployed complete with computers, desks, and telephones, courtesy of Agility Recovery Solutions, in Mississauga, Ontario. They are connected via “hitching post” to the hot site for instant connectivity. During an emergency, the railroad’s management plans to house up to 80 additional workers in the trailers, mostly to handle customer inquiries.
Until the recent train bombing in Spain, a mobile site next to a hot site might have qualified as disaster-recovery overkill. But such a view ignores the herculean coordination necessary to run a transcontinental railroad. “If we can’t throw switches,” says Cammack matter-of-factly, “we’re out of business.” * J.G.
JOHN GOFF (JOHNGOFF@CFO.COM) IS TECHNOLOGY EDITOR AT CFO.
COPYRIGHT 2004 CFO Publishing Corp.
COPYRIGHT 2004 Gale Group