All systems go

On Sept. 11, Lehman Brothers’ CTO and Managing Director Robert Schwartz triggered the financial company’s IT contingency plans via his BlackBerry from a stairwell of the World Trade Center. The company had three floors of IT developers at 1 WTC, the North Tower, and a datacenter and trading floors at 3 World Financial Center.

Under Schwartz’s leadership, Lehman Brothers already had in place a resilient technical infrastructure, including interapplication multicast infrastructure and geographically dispersed datacenters linked by dark fiber with a redundant private WAN. As a result, the company was able to relocate more than 6,000 employees, recover from the loss of more than 1,000 servers and 50TB of storage, and keep systems running.

“The proof is in the pudding,” says Schwartz. “We didn’t miss any time trading bonds or stocks once the market reopened. We built two replacement trading floors in under a week. I credit the hundreds of people involved.”

Even with these back-up plans in place some 12 to 18 months prior to Sept. 11, Schwartz and the company’s 700 IT staffers had their work cut out for them. The company, with $22.3 billion in revenues last year, had to move 400 traders, recover mostly proprietary trading and financial risk applications in new environments, re-establish market data and other real-time links via a “completely redundant network infrastructure” of dark and DWDM (Dense Wavelength Division Multiplexing) fiber links, and find office space for its investment bankers.

By maintaining geographically dispersed datacenters, linked by a high-capacity dark fiber network with a completely redundant private line-based WAN, the firm was able to maintain complete online access to all of its branches around the world when the New York City datacenter became inaccessible.

“We were still online with all of our branches from our New Jersey location,” Schwartz explains. “The basic infrastructure was already in place to support trading out of either location.”

Schwartz also oversaw implementation of an interapplication multicast infrastructure that operated through the disaster. This messaging platform transmits Lehman Brothers’ transactions and market and risk data through their WAN, transparent to the company’s proprietary applications — thus providing a resilient straight-through processing environment linking back office and front office activities as well as increasing programmer productivity and infrastructure efficiency.

“One benefit that we received is that our programmers could pass information from one end of the network to the other — it’s sufficiently resilient so [information] can be routed around gaping holes,” the CTO says.

Web services also played a role in Lehman Brothers’ recovery from the Sept. 11 events. “We increased the service offerings to our employees over an extranet portal. For instance, it made it possible to update their phone numbers as it became more difficult to figure out where they were located. We also used Web services as a general broadcast mechanism in advising people how to report and where to go to work,” he says.

Even using “disruptive technologies,” Schwartz found that one of the least technically-challenging moves the company made turned out to be one of the most useful: a limited deployment of some 1,000 IP telephones. “We lost our phone systems servicing 7,000 employees. When we moved into temporary buildings, as soon as we had a network connection, we had phones implemented immediately. Now we have 4,000 IP phones [in use].”

As Lehman Brothers relocates and moves thousands of employees into new offices in Manhattan, Schwartz again sees the benefits of using plug-and-run IP phones and is also outfitting some conference rooms with 802.11b connections.

As Schwartz looks back on last year, he encourages other CTOs to take a broad view when thinking about disaster planning.

“You need to worry about the long lead time items, to implement a second WAN can take months and months. If we hadn’t had that, we would have lost connectivity to all offices.”

— Loretta W. Prencipe