Windows Security Update Roulette

To patch or not to patch — that’s the gamble IT managers have to make time and again with Microsoft security updates. Put your money on the wrong side and your company’s systems could be toast. And the worst part is it’s a game where nobody ever wins – the best you can hope for is that you don’t lose.

One IT manager returned from the Thanksgiving holiday yesterday to discover that he’d come very close to losing the user permissions on his network, as well as his sanity. “Two of my server admins were working this past weekend applying the latest security updates to Windows Server 2003 R2 on our main departmental server,” he wrote. “The upgrade starts performing checkdisks and blowing away all of our ACLs with NTFS permissions. This is a server that is tuned for security with probably several hundred Active Directory groups to allow us to provide the granular application of security permissions necessary in a complex business environment. I am so far beyond stunned on this I don’t even know where to begin. If we’ve avoided utter disaster, it’s no thanks to Microsoft.”

The admins had noticed the server was running Chkdsk.exe on the hard drives after bootup, but assumed it was one of Windows’ standard maintenance routines. Fortunately, after updating some other equipment, they looked again and saw that the server was pouring out line after line of “Replacing invalid security ID with default security ID for file XXXXX.” Realizing that the NTFS permissions were being trashed, one of the admins did a hard power-down. “The only thing that saved us was, as soon as they became suspicious when they saw the Chkdsk messages scrolling by, one of the admins slammed the power button and killed the server,” the IT manager writes. “They booted the server again, skipped the Chkdsk, then they both scrambled late into the night to identify damaged directories and repair permissions. It could have been a lot worse.”

Why that particular server was hit while others they’d updated at the same time weren’t is not yet clear. Staff members found a Microsoft KB article that appears to describe the problem. “I will find it totally unacceptable if someone at Microsoft wants to say, well, there is KB article Number 5,000,077 on this and it’s our problem if we only read up to Number 5,000,022,” the IT manager says. “Why or earth should I expect a standard upgrade with security patches to re-set ACLs across my entire server!? Yes — I am shouting. This is absolute, incompetent madness on the part of Microsoft.”

From the knowledge base article, one factor was probably the fact that the reader’s organization had not applied the full Service Pack 2 for Win2003 due to issues they’d encountered earlier. But the update hadn’t specified that SP2 was required. “Any way you cut it, a security update offered freely without requiring SP2 should not blow away the ACLs of the server. In fact, no should do this. This is just sickening – we are all starting to despise Microsoft and its patching methods.”

What do you think? Post your comments about this story below.