The IT lab evolution

The image of laboratory technicians dressed in white coats conducting performance tests and monitoring arcane experiments is antiquated at best. The traditional testing and experimenting functions of an enterprise IT lab have been supplanted by the fast-paced, real-time needs of daily business operations, chief technologists say. Labs have been reinvented to fit today’s “new normal,” and testing is now done on the fly.

When the need arises to verify performance of a system or a new product, enterprise chief technologists need staff to quickly implement testing procedures that can enhance IT strategies and protect enterprise systems.

Larry Biagini, CTO of General Electric in Fairfield, Conn., says he relies on key staff members to test their systems. “What we have is an environment where we can replicate changes that we need to [make] with the same people that are going to make the change,” Biagini explains. “We don’t have a building somewhere with a bunch of people sitting around doing nothing, waiting for something to implement.”

“Calling something a ‘lab’ raises red flags in this era of tight IT budgets, as well it should,” says John M. Jordan, a principal in the Office of the Chief Technologist at IT-consulting company Cap Gemini Ernst & Young in New York .

“Separate, stand-alone environments can’t pay back along the conventional ROI axis, and there’s a tendency to isolate technology from business practice at a time when this is particularly risky,” Jordan adds. “That said, CTOs definitely need environments in which to explore technology alternatives.”

Testing is part of the mix

The need to test before implementing new systems, upgrades, and products remains essential and the enterprise continues the practice, Biagini and others say.

“We use labs to simulate volume, and we use labs to simulate configuration changes and the effect on volume response time and things like that,” Biagini says. “Because of our size, from a technology standpoint, people give us what they intend or want to sell us to have us try it out.  They know we wouldn’t buy anything without knowing it’s going to work first. So [the testing is] worth the investment because if we don’t test it ourselves and we can’t predict the effect any change it will have on an environment, what we’re going to do is introduce variability that we can’t live with.

“Let’s say that I am going to upgrade our instant messaging environment from depending on Directory A to depending on Directory B,” Biagini says. “We change our directory environment, and we want to see the impact on instant messaging. We would run a test with the old environment, simulate it with a testing suite that we have of robots that can simulate volume.”

Biagini’s team would then gather testing results and change out those pieces that to be replaced in production and then rerun the same test. “Did we see response time degradation? Did we see any changes in the way the data displays as it would do from a name perspective?” asks Biagini. “When we verify those results, then we’ll figure out how to implement it in production. We’ll also use it to figure out how long it will take to do a change.”

Speedy communication of testing results becomes important in IT enterprise environments. “In such a large company, something simple like touching every entry in your [directory] address book has a ripple effect because all the changes get replicated down,” Biagini says. “We’ll figure out, before we make that change, what it’s going to take for that change to propagate through GE. We can give the businesses some view of what the next couple of days or hours are going to look like.”

Testing occurs continually, as organizations experience constant need to tinker and plan, says Anthony Hill, CTO of Golden Gate University in San Francisco .

“We’ll test upgrades and we’ll stage desktop migration to Windows XP,” Hill says. “On the applications side, we have test and development environments. We have Novell companywide, so we run tests on a smaller Novell network. We have some desktop computers and we test OS migrations strategies and application compatibility.”

Such activities remain a strategic component for an enterprise in order to implement a technology plan, says Dan Kusnetzky, director of system software research at Framingham, Mass.-based IDC.

“If all your organization does is run prepackaged software, then the testing may just involve making sure the technical people and the end-users know how to use the software. The tests may have a training function,” Kusnetzky says.

“But if it’s an organization doing more in the way of integration of technology and [diversification] of technologies, then [the organization’s CTO is] probably doing a lot of work to construct [software] and put it together,” Kusnetzky adds. “They may have built software and put it into packaged software. When you integrate any technology you need to test to make sure it’s working as intended.”

Virtual labs

The notion of laboratories has been modified to fit evolving needs of modern business plans and the IT shop.

“Large organizations naturally subdivide into smaller units, and some of these will have attractive cultural, business-process, and technical characteristics that lend themselves to pilot implementations of new technologies,” Cap Gemini’s Jordan says. “Seeing how new hardware or software integrates with and alters existing practices is usually more important than pure laboratory benchmarking.”

At GE, Biagini says that the company’s research and product development labs are advanced, on the IT side. “It really becomes more of a testing and Q&A process than a separate stand-alone,” he explains. “The idea is, ‘Let’s test this out and see what it does.’ It’s really the same group of people and I think that’s applicable to business of any size. They have to be able to have some process in place where they can get some predictability and repeatability out of any change they’re going to make.

“In our environment, it’s the same people who have to live with the results,” Biagini adds. “So if you’re in charge of instant messaging or you’re in charge of the mail system for GE or security for GE, it’s your responsibility to make sure that you’ve tested this stuff and have the ability to test it any time you introduce a change.”

Golden Gate University ‘s Hill says he keeps a small space for testing. “You need an isolated network to test equipment and new applications,” he says. “But I think that it’s an expensive proposition to staff people in the lab. So in concept the lab is really a test center with isolated networks to examine things and isolate them from the production enterprise.”

Important exceptions to the rule

There are times when major IT projects need to be tested before decisions can be made about deployment. In such cases, a full-fledged lab has to be sought out.

“Some technologies simply do not work in a given context, and there are times when performance and interoperability must be assessed before selecting platform components,” Jordan says. “Such an evaluation environment needs careful oversight so as not to take on a life of its own, but running offline simulations of new hardware/software combinations and configurations can save CTOs significant pain and cost when they discover whether or not a product can actually scale or perform as required.

For example, DigitalNet in Herndon, Va., which designs, implements, and manages networks, and provides security for information systems for U.S. government agencies and the military, conducts major system testing for its clients, says CTO Mary S. Stassie.

Government agencies can’t always conduct sophisticated testing necessary to ensure that new technology can work with legacy systems, Stassie says.

“For example, in security issues, we are focusing on multilevel network security.” In testing a new firewall, she says, engineers had to achieve a balanced level of interoperability that allows just enough ease-of-use for clients without shutting down access.

DigitalNet made sure the customers system could function with the new firewall, Stassie says. “The major issue is interoperability.”

Make it work

Chief technologists can decide if their organization needs outside testing help, Kusnetzky says. “Small companies are less likely than big companies to worry,” he says. “They rely on partners, resellers, or software suppliers to present them with a complete solution and they usually charge a consulting fee to do so.”

But whether you test in-house or use outsourcers, the bottom line is to make sure it adds value to the business process.

“When you have a test process as part of deployment plan, you can find and fix problems before they get big,” Kusnetzky says. “If you are going to make a change with any critical systems you want to test it and have a strategy [so] that if you run into a snafu, there is a way to gracefully get back to the previous implementation. Otherwise, people get really mad at you.”