Patchy years ahead for software users

Dale Sweitzer, a network administrator for Crossville Ceramics in Tennessee, has hit a rough patch — or series of rough patches to be exact.

Sweitzer, who handles security for 160 geographically dispersed PCs running Microsoft Corp. software, says that he spends more time and money applying software patches than he does doing almost anything else on the job, and he’s not alone.

Although vendors like Microsoft have been working recently to simplify software patch delivery, the problem remains a critical one for IT administrators who are struggling to keep up with all the patch work.

From filling security holes to upgrading features, the act of applying patches, or putting a new piece of software code over an old one — has become for many a full time job.

“Patching is a nightmare,” said Alex Bakman, founder and chief executive officer of patch management provider Ecora Corp. The recent onslaught of security threats like Blaster and Slammer, have only aggravated the problems administrators face, he said.

According to Bakman, it takes an average of 30 minutes to apply a patch to each machine in a company’s system, and hundreds of patches have been released by various vendors so far this year. Companies are selecting which patches they need to apply because their IT staff is already spending two to three hours a day patching systems, Bakman said.

“The current frustration level is incredibly high,” Bakman said.

Of course, patching is just a symptom of the wider problem with software, users say, which is the gradual and seemingly endless discovery of new security vulnerabilities which must be patched and fixed throughout the software’s lifetime, requiring an enormous amount of time and money.

“The total cost of ownership of software is incalculable,” Sweitzer noted.

Vendors blame insecure code. However, eradicating patches by creating flawless software is impossible with imperfect humans writing software code.

“Despite all our best efforts, all vendors in this industry still have vulnerabilities,” Steven Adler, senior security strategist for Microsoft in Europe, the Middle East and Africa (EMEA), told an audience of IT administrators at Gartner Security Summit in London last month.

While the patch situation doesn’t look set to improve anytime soon, vendors say they understand the administrators’ frustration and are working to improve the situation.

Oracle Corp. Chief Security Officer Mary Ann Davidson said that her company sees patching as the last phase in its security efforts.

“We try to do things right the first time but to err is human,” Davidson said.

Oracle has a rigorous policy in place for testing and delivering patches, she said, and notifies customers of severe problems which can be exploited.

“Otherwise, we don’t want to yank their chains. People don’t have time to apply a lot of patches,” she said.

Patching is also a big concern for Sun Microsystems Inc., according to Gilles Gravier, the company’s managing director of operations for platform infrastructure and security in EMEA.

Sun combines patches with the upgrades of its Solaris and Orion software, so customers can update and fix their systems on regular schedules. However, for more pressing security issues, the company releases stand-alone patches. Sometimes these are temporary patches — what Sun calls t-patches — that have not gone through a thorough testing program, Gilles said.

“People installing t-patches know that they haven’t gone through full testing and that they could break something,” he said. However, the company feels it is necessary to issue patches for exploitable problems as soon as possible, Gilles said, noting that full testing can sometimes take a few weeks or longer.

Despite all the efforts put into delivering timely and high-quality patches, Davidson added that all vendors think they can do better. Users don’t seem to expect a miracle, but are looking for a lessening of their patching problems.

Microsoft’s recent decision to simplify the process by delivering patches once a month and combining fixes when possible is at least a sign that the industry is taking the problem more seriously, some users say.

Andreas Wuchner-Brühl, head of Global IT Security for Novartis Pharma AG, said that the changes were a “step in the right direction.”

Wuchner-Brühl, who is on Microsoft’s security advisory board, said that the software maker is paying closer attention to customers concerns.

However, the company’s most recent security changes will be of little help to him. Wuchner-Brühl manages a system of over 3,000 mixed servers in a “qualified” environment, meaning that detailed reporting of any changes to the files and systems must be documented. Novartis is a pharmaceutical firm and must comply with detailed healthcare industry regulations.

In addition to the 30 minutes it takes to apply a patch, his staff has to do two to three hours of paperwork to document the patch.

“In a qualified environment there is a lot of work behind the scenes, you don’t just apply a patch,” Wuchner-Brühl said.

The company already collects patches for a monthly update and combining multiple fixes in one patch can actually create more work in qualified systems because administrators have to document all the changes, whether they thought they needed them or not, Wuchner-Brühl said.

Even administrators working in an unqualified environment have to do more work than simply applying a patch implies. Most companies test the patches on an isolated system first to make sure it doesn’t “break” an application, especially if that application is customized.

In fact, fear of breaking applications deters many companies from applying patches that they need, according to Ecora’s Bakman. Companies will put off patching, and certainly won’t go through the process at critical times, like before a big retail or holiday season, he said.

Oracle’s Davidson added that, “people won’t apply patches for anything in the last three weeks of the fiscal year because they don’t want to risk their systems going down.”

Still, patching is just a symptom of underlying problems with software, Wuchner-Brühl noted. To address vulnerability issues, software vendors are increasingly looking to offer more secure products from the outset under “secure computing” initiatives.

Microsoft, for example, has said that it is rolling out technologies to protect customers from problems such as buffer overruns which are often exploited by hackers to takeover computers, and to offer protection against attacks on communications ports.

The Redmond, Washington, software maker has said that its upcoming desktop operating system, code-named Longhorn, will be more secure. Longhorn isn’t due until mid-2005, however.

“That’s at least two more years of patching on the desktop,” Gartner analyst John Pescatore said at the Gartner Security Summit in London. “And there will still be problems.”

While vendors work to deliver more secure products, users are being advised to put a patch management process in place.

In a research note released in March, Gartner analysts advised companies to prioritize patch installations based on how critical the security vulnerability is, and to evaluate the patch installation requirements. Some patches may require other patches to be applied at the same time, Gartner said, and can be superseded by more-current patches or service packs.

Companies should classify server and desktop configurations as standard and nonstandard so they can be patched according to their specific needs and all patches should be tested before deployment, Gartner said. Furthermore, companies should only accept official patches and the patch management infrastructure should be as secured as the company’s outward-facing Web and application servers.

If all that is enough to make the IT department’s head spin, a host of vendors has stepped in to offer patch management tools which, among other things, log system configurations and automate some installation and update functions.

While these tools offer administrators some much needed help with the symptoms of software insecurity, the problem for now, remains.

“Since software was first developed, there have been patches,” Ecora’s Bakman noted. “And that won’t change anytime soon.”