Land O’Lakes solves disaster recovery puzzle

Land O’Lakes tried for years to come up with an appropriate plan and infrastructure for disaster recovery and business continuity.

The $6 billion food and agricultural cooperative worked with a variety of vendors, none of which provided it with services and plans that Land O’Lakes IT management found sufficiently broad, flexible and effective. Meanwhile, as IT projects and priorities went, disaster recovery wasn’t something that necessarily registered on upper management’s radar screen.

Solving the disaster recovery riddle, however, became a corporate priority after the terrorist attacks of Sept. 11, 2001 and the increasing number of U.S. government regulations, such as the Sarbanes-Oxley Act, which require companies to have contingency plans in the event of a disaster, for example, to protect business data.

“We struggled with disaster recovery as many organizations have because disaster recovery is one of those things the business leadership doesn’t want to focus on unless they have to, because of the time and money it requires,” said Land O’Lakes Chief Information Officer Mark Wilberts. “It’s kind of like insurance: if you need it, you really do need it, but if you don’t need it, you don’t like to think about it and you certainly don’t like spending the money and resources.”

Land O’Lakes’ emphasis on disaster recovery isn’t the norm, because these services remain prohibitively expensive for all but large companies, said David Tapper, an IDC analyst. “Disaster recovery is a priority and a concern for enterprises, but it’s not among their top three priorities in general, because only the largest companies can afford this type of service,” he said. “After 9/11 there was a belief everyone would start doing this, but it didn’t happen.”

But it happened at Land O’Lakes, where the IT management once again went looking for a disaster-recovery vendor that could develop a plan meeting its needs and requirements, including:

— An office recovery center located reasonably close to its main Arden Hills, Minnesota, campus (about 8 miles north of the St. Paul/Minneapolis downtown areas) where the company could relocate workers in the event one or more of its buildings were damaged;

— A data center hot site located out of Minnesota equipped with the necessary network and server infrastructure for Land O’Lakes to load its data and applications and restart its operations;

— Consulting services to help Land O’Lakes’ different business units draft business-continuity plans that would be put into action in the event of a disaster.

“We had for a number of years relationships with other disaster recovery suppliers, especially hot sites kind of suppliers, and we were never really able to get where we needed to get, for a variety of reasons: performance, (business) relationship, flexibility. And there was also a missing component in the whole disaster recovery situation: If we had an office failure where would our people go?,” Wilberts said. He declined to name the vendors the company has worked with in the past.

The solution proposed by most vendors was for Land O’Lakes to simply move its workers to a hotel or to a similar facility, but setting up the necessary working conditions there, including networked PCs, wouldn’t be practical, Wilberts said. “Logistically, it would be almost impossible,” he said.

Land O’Lakes finally found a vendor able to provide a satisfactory disaster-recovery plan in Hewlett-Packard Co. (HP) that included the officer recovery center component. HP’s solution? Build such a center based in Minneapolis on Land O’Lakes’ request and then market it to other area companies as well.

“HP came forward and has been a very flexible partner and responded to our needs,” Wilberts said. “The reason there is an office recovery center here in town is because we expressed an interest and they initiated a marketing assessment to determine whether that was a valid business proposition for them and they concluded it was. They built the center and we’re the first committed customer. That’s the type of partner we need to have,” he said.

“Between their hot site, the office recovery center and the network we were able to put together a complete realistic solution to our disaster recovery needs at a cost effective price and they’ve been very flexible at making that solution package feasible,” he added. The office recovery center is being inaugurated Thursday.

This is the 11th disaster recovery facility HP has in the U.S. — some are full blown data center hot sites and others are office recovery centers — and the 19th in the Americas, said a spokesman for HP. HP has plans to open two or three more this year in the region, said Belinda Wilson, HP’s director of business continuity services for the Americas.

HP also has plans to expand some of its current facilities in the Americas, she said. For example, a facility in Colorado Springs, Colorado, will be extended as a result of demand from West Coast clients and HP is considering building new facilities in Phoenix, Las Vegas and New York City, she said. It has a total of about 50 such centers in 38 countries worldwide, Wilson said. “We can’t keep up with the demand for this type of service. It’s been phenomenal,” she said.

Still, in the overall plan of things, HP is a tier-2 player in terms of size, IDC’s Tapper said. “HP is providing an option to clients. The two major players continue to be SunGard and IBM, which have the predominant part of the market and the most experience in this business,” Tapper said.

Nonetheless, choosing HP was a good fit for Land O’Lakes in another way, Wilberts said. It is consistent with its strategy to work with fewer IT vendors, since Land O’Lakes’s Unix and Intel Corp.-based server infrastructure is based on HP hardware, he said.

If a disaster occurred, Land O’Lakes would use HP’s Atlanta area hot site. It has the necessary network hookups for Land O’Lakes network services providers to re-route WAN traffic to it, said Michael McKeown, Land O’Lakes director of production services. It also has the necessary server infrastructure for Land O’Lakes to set up its systems and load its data onto them. Land O’Lakes backs up its data onto tape and stores it outside of its corporate campus with a third-party provider. If the data center was damaged, those tapes would be flown to the HP hot site in Atlanta. Land O’Lakes staff would also fly to Atlanta to set up the server operations, McKeown said.

In addition to this, HP is also providing business-continuity consulting and advisory services. Each business unit needs to have a plan in the event of a disaster to know what they do until the computers are up, how they communicate with customers, suppliers and employees and, once the company is operating in recovery mode, how they manage the business, Wilberts said. “We’ve partnered with HP as well on this side,” he said.

Land O’Lakes signed HP in mid-2003, for a five-year disaster-recovery engagement, the financial details of which aren’t being disclosed. Overall, Wilberts estimates the Land O’Lakes disaster recovery project is about 30 percent done, and should be completed before the year is over. “We’re not through the process yet, but everything is going as planned and looks good and hopefully as we get towards the end of year we’ll have completed our work,” he said.

A key to the success of the project is the involvement of IT staffers from every part of the Land O’Lakes organization, Wilberts said. “We have two or three IT people working full time on this, but you have to recover your entire operation so we have people from every part of our technology environment, from desktop to processing to network to support services, involved in some capacity. The plan has a very broad impact, touching a lot of people, but we’re involving them as they need to be involved.”

As it sets up its disaster recovery plan, Land O’Lakes is also fine-tuning aspects of its IT operations, Wilberts said. “We’ve made some small incremental improvements to date. We have identified a number of things we must change, which we’ll be addressing over the next six months, and fully anticipate when we complete this we’ll have improved, streamlined and bullet-proofed a number of our processes here,” he said.

Meanwhile, upper management is firmly behind the IT department in this project. “I work for the CFO (chief financial officer) and it’s in his objectives this year to complete this work. It is important for fulfilling our Sarbanes-Oxley requirements. And it’s clearly an expectation of our senior management and our board that we’re able to operate our business in the event of a disaster,” Wilberts said.