Applying patch management

At one time, the concept of patch management merely meant keeping an owlish eye on Microsoft’s download site. But in recent years, Internet security has steadily deteriorated. Anti-virus vendor Sophos reported a staggering 959 new viruses and worms discovered in the month of May alone, and the number of Windows exploits and vulnerabilities seems to have grown exponentially.

No longer a grunt-level headache for systems administrators, keeping abreast of security patches has become an essential business practice for any company, large or small. Although an ad hoc patching policy might once have sufficed, the surge in updates during the past two years demands that IT managers be aware of security at every level. After all, if even one critical system is compromised, the entire network can be exposed.

Unfortunately, the sheer volume of updates has made securing an enterprise network more difficult than ever. Preparing for a major OS upgrade is one thing, but too often, an unexpected fix can blindside IT administrators.

“It’s the security updates that are coming so fast nowadays,” says John Saulz, principal systems engineer for the City of Colorado Springs, Colo. “You’ve got to know when to find them, test them in almost no time, and then decide whether it’s safe to deploy them. And you’ve got to do it in just a few days.”

Navigating this increasingly torturous terrain doesn’t have to be a nightmare, but it does call for a comprehensive security strategy that gives patch management a central role.

Windows on the defense

Every platform is subject to security fixes, but Windows systems are typically the driving force behind most companies’ decision to implement a patch management solution. Not only does the Windows platform account for the bulk enterprise systems, it has also been the source of the greatest number of security vulnerabilities.

For a time it seemed that IT departments’ calls for more secure software were falling on deaf ears in Redmond, but Microsoft has since made security a top priority. Last fall, the company initiated a full-force drive to revamp its patching strategy, beginning with the announcement that it would begin issuing patches each week. These scheduled updates have won acclaim from harried IT administrators, causing even other software vendors to take notice.

“It’s extremely expensive to patch systems, so consolidating and centralizing the process is hugely valuable to customers,” says Mary Ann Davidson, chief security officer at Oracle. “Microsoft has taken some good first steps in this direction with their announcement of scheduled releases last fall. Customers really like that, so we’re looking into it as well.”

In addition, Microsoft offers a number of products aimed at helping administrators keep abreast of new updates and security fixes as they arrive. Windows Update forms the baseline defense, with the forthcoming release of Windows Update Services — previously known as SUS (Software Update Services) — providing additional tools for network administrators. Finally, SMS (Systems Management Server) 2003 offers advanced deployment, reporting, and compliance-enforcement features for environments with more demanding management needs (see Secure Enough for a Bank).

Further complicating matters, a wide variety of third-party patch management solutions provide features above and beyond what Microsoft’s products offer. Which of these packages might be right for you will depend on your organization’s specific requirements.

Taking stock of security

The first step in developing an effective patch management strategy is to compile a complete and accurate inventory of the IT assets in your organization. In today’s networked environments, any device could potentially become the weak link in the security chain, and thus understanding what systems are in place is essential to formulating a strong defense against network threats.

“The key is knowing what you’ve got,” says Saulz, who manages 1,500 nodes and 150 servers at 40 locations for Colorado Springs. “Once you know what you’ve got, you can identify key systems, key patch types, and figure out a strategy for deployment.” With a completed inventory in hand, IT management can use its findings to draft a corporate security policy that clearly defines the company’s stance on patch prioritization, testing, and deployment for systems across the enterprise.

According to Liehlin Chang, senior network administrator for the IT department of Nassau County, N.Y., one of the primary goals of the inventory process should be to identify those mission-critical systems that must take top priority in a crisis. “Once you know what’s got to be protected right away in any vulnerability situation, you can begin to plan the rest of your patch strategy around those critical systems,” Chang says.

Armed with a thorough understanding of the network environment and a prioritized list of critical assets, an organization can begin to evaluate software solutions to help alleviate the headaches of security maintenance. Patch management software is available in a wide variety of forms, each one offering a different way to attack the problem (see The Fix Is In). If you’ve assessed your requirements correctly, selecting the one that’s right for your organization should be a matter of matching key policy requirements to each potential package’s individual strengths and weaknesses.

Averting a crisis

Although a comprehensive patch management solution might be the ideal, many administrators — even those in charge of large networks — have been railroaded by circumstances into an ad hoc approach. Chang’s department currently manages more than 1,000 network nodes, including more than 100 servers, spread across several locations. Unfortunately, ongoing expansion has stymied any plans to implement a networkwide patch management solution.

“Moving to a dedicated patch platform simply isn’t feasible for us right now because there are several initiatives pending that will bump our node count to over 4,500 by the end of the year,” Chang says.

A situation like this calls for an immediate stopgap measure to buy time until a more comprehensive patch strategy can be developed down the line. Chang’s group has begun a test implementation of SMS 2003, but he doesn’t believe that system will be ready until the last third of the year. He chose SUS to fill the gap.

“SUS was something we could implement quickly,” Chang says. “That at least gave us a central patch-processing point even though the process is still way more manual than we’d like.”

SUS allows a network administrator to create a patch repository on the local network rather than having to rely on Microsoft’s own Windows Update site. Ultimately, it is still a manual solution, requiring IT staff to patch each system individually. For some environments, this type of approach is simply not feasible. Such cases call for a more advanced tool, such as SMS or one of the many options available from third-party vendors.

“A third-party patching solution quickly became a no-brainer for us,” says Michael Roberts, CIO of Bank of Alameda. Roberts’ network consists of six separate locations with more than 50 discrete nodes and servers. “The network isn’t that large overall, but the disparate locations make manual patching pretty much impossible,” Roberts says. “We’d need to hire more IT staff than we’re budgeted for, and they’d be spending all their time running between locations.”

Roberts eventually settled on a solution called Hercules from Citadel, an 8-year-old vulnerability-assessment and patch management vendor.

“[Hercules] not only handles all our platforms but lets us centralize management across all our locations,” Roberts says. “It even imports vulnerability information from our [eEye Technologies] Retina scanning tool and then uses that information when building its patch packages.”

Colorado Springs’ Saulz also chose a third-party solution. Rather than invest in a dedicated platform, however, he looked to the tools he already had in-house. “We already had Configuresoft running as a configuration repository,” Saulz says. “Adding the patch management option was not only cost-effective, it was fast. But what makes this platform so perfect for us is that the scanning and inventory pieces are all COM-based, which means we can scan even our older Windows 98 workstations.”

Another significant benefit of dedicated, third-party patch management solutions is that, unlike Configuresoft’s or Microsoft’s products, they can often support platforms other than Windows. Citadel’s Hercules can manage patches for Windows, Red Hat Linux, AIX, HP-UX, and Mac OS X, and the company plans to release agents for networking products from Cisco and other manufacturers later this year. “Making sure Hercules integrates into any network scenario is critical for us,” says Citadel CTO Carl Banzhof.

OS patches are only part of the problem, however. As more and more applications rely on networked environments, vulnerabilities in these systems must also be addressed. “Key servers are important to any network operation, but key applications are important to business processes, and we’ve got all kinds of those,” says Jamie Bernstein, vice president of LifeLine Computers, a network and systems consultancy.

Remote users are yet another problem that some vendors, including Cisco, are beginning to address. Even with a stringent security policy and an automated means of distributing tested patches across the local network, network administrators are helpless if the clients aren’t connected.

Applying proven patches

For many customers, one of the greatest benefits of a third-party patch management solution is that an outside vendor brings a fresh set of eyes to the patching landscape. Microsoft takes pains to ensure that each patch it delivers does no harm to existing environments, but more than one administrator has been burned by a patch that seemed to create conflicts rather than resolve problems.

To help correct this, several vendors offer their own patch repositories as alternatives to Microsoft’s. The advantage is that patches obtained from these vendors have been prescreened by their own testing staffs to identify any potential conflicts that Microsoft’s engineers may have missed.

Although this screening process gives some additional comfort to those with stock Windows environments, no outside vendor can test a patch against an enterprise’s own custom software. Customers with unique requirements will want a patch management solution that allows IT staff to evaluate patches on their own networks before deploying them on a wide scale. For example, LANDesk recently partnered with VMware to create a patch-testing environment based on VMs.

“The testing piece is crucial for us,” says Bernstein, who is evaluating several patch management systems for LifeLine’s clients. “We manage a number of smaller corporate networks, so pushing out a patch that winds up hurting some or even all our customers is something we really need to avoid.”

A measured approach

In the past, pushing patches to individual machines from a central location was the big problem, but most modern systems management products can handle that job with ease. Instead, today’s headaches stem from the sheer volume of nodes that must be serviced, as well as the complexities of heterogeneous environments.

Even for those organizations with the ability to deploy a comprehensive patching solution, the path to a successful strategy is convoluted and highly individual. There are a number of different ways to attack the problem, each with its own strengths and weaknesses.

The one certainty is that no organization can afford to ignore the problem of patch management. Ignoring critical security fixes is not an option. Instead, the goal should be to apply the latest patches on a timely basis while minimizing the risk to the overall IT environment. To this end, each organization must identify its priorities, establish a policy, and implement the software tools that best suit its unique needs.