Simplifying the patch process

Next to random Windows problems and day-to-day systems management, I find patch management the nastiest task for an enterprise Windows systems admin. It’s often frustrating, time-consuming, and complex, and if you don’t watch it, it will wind up adding more dollar signs to the red half of the books than you want.

And the patch management problem is getting worse. Although Microsoft seems to have stabilized Windows 2000, the entire .Net Server family is going through revision. Windows XP and Office 2003 mean more patch problems on the client side, and security vulnerabilities are growing by approximately 50 percent, based on first-quarter incident reports from the Computer Emergency Response Team (CERT). That’s a lot of patching.

You won’t really know whether any given patch will affect your enterprise until a systems administrator has made sure. You can weed through some of this chaff by matching the patch up against features or software your company does or doesn’t use. But within that rather large area, you’re still looking at multiple new patch developments per day for many companies.

While you can face this nightmare on your own, you’re better off doing so with a team and using appropriate tools for the job. Even with this level of preparedness, however, it’s never a foolproof system. I’ve looked at a couple of patch management products recently, and both of them had the same basic doozy of a requirement: a testing environment.

Patch management vendors play down this testing environment requirement simply because it doesn’t need an exact hardware mirror, but that has never been the real problem in my experience anyway. The trouble is, setting up and maintaining even a semi-accurate testing environment means installing the same software suite you use in your production environment, especially back-end server suites and at least one representative of every client configuration.

Regardless of whether you can play the hardware-server consolidation game, that’s still quite a lot of extra software licenses and a good deal of staff time devoted to installation, configuration, and maintenance. And frankly, most of the patch management vendors don’t recommend shortcuts on the hardware side, either.

While setting up a test environment is costly and time-intensive, it’s the only way to make sure any new patches don’t cause more troubles than they solve. The trick in setting up the environment is paying attention to details. Hardware specs, for example, don’t have to match production specifications exactly in terms of resources, but you’ll want to stay on the same page when it comes to system boards, BIOS versions, RAID systems, and similar features.

One tool I’ve found invaluable for maintaining accurate server and client configurations is Symantec’s Ghost server. By keeping an up-to-date image of all basic production server and client configurations, not only can you quickly ensure a clean and accurate testing environment, but you can even rebuild the entire lab in the space of a single day. That’s a fantastic advantage if the patch is a critical one and you need to bring up a testing environment in a hurry.

But with all these resources, you might wonder where patch management tools add an advantage. Mostly, it comes in the form of improved organization and ease of patch deployment. You’ll find that most patch managers will monitor your required list of vendors for new updates. Then they’ll download any new patches and match them to an inventory they take of your patch-targeted systems. This way, you know what’s installed on every machine and have an easy mechanism for pushing out new patches if they pass your testing criteria.

One notable exception to solutions requiring a testing environment is PatchLink Update, which boils down to a patch management service provider. Clients will need to install an update server based on a Windows 2000 box running MSDE (Microsoft SQL Server 2000 Desktop Engine) as well as a client module running on every machine to be patched. PatchLink will monitor a series of vendors, download new patches, and run them through their own basic testing criteria and then push them out to customers. That means small and midsize business with limited IT staffing resources can push out new patches with a reasonable assurance that this won’t cause a crash.

Even with such a service provider, however, patch management will never be a brainless operation. The trick is making sure your patches have been tested, keeping track of what’s been patched and where, and finally, not being too ambitious. Rolling out one patch at a time when you’ve got five or six on your plate may seem tedious, but it’s a heck of a lot easier when you’re faced with tracking down problems.