One last kick in the pants

Usually, this week is reserved for the “What We Can Expect in 2005” column. But I think I’ll push that out a week or so in favor of what may become a new tradition: “The Last Kick in the Pants.”

Let’s call 2004’s kickee “Steve,” for lack of a better name. Steve is a typical Microsoft systems administrator: Rising workload, rising complexity, and decreasing pay summarize Steve’s work life accurately. So he looks to Microsoft for back-end administrative tools to help ease these burdens before his boss or his users take his life.

What makes Steve’s life a little extra-harried is that he’s responsible for more than 25 remote locations connected to the main corporate site via a couple of dedicated T1 lines. And that’s where our tale begins, namely when our intrepid hero receives a barrage of user calls claiming they can no longer see the network.

He checks his management tools, all of which tell him the T1 connections are down. But losing two Ts simultaneously is rare even for Verizon, so Steve dials the phone company and politely asks what the hell is going on. Verizon does its magic and helpfully informs him that both Ts are humming along just fine — but that Steve’s company is exceeding its bandwidth by a significant amount, so the lines are simply clogged. Sherlock Steve begins investigating his network traffic, but it isn’t until he checks his router logs and discovers a few hundred simultaneous Windows Update entries that he tracks down the culprit. It seems as though every one of his corporate clients has requested Windows SP2 (Service Pack 2) at the same time.

Sweet!

Steve’s no dummy. All clients have had their Windows Update service disabled. He’s got an SUS (Software Update Services) server installed, which is holding SP2 in reserve as “unapproved.” Therefore, what just happened shouldn’t have been possible. Reluctantly, Steve is forced to call the great tech support giant in the Pacific Northwest and ask, somewhat less politely, just what the hell is going on.

Surprisingly, Microsoft tech support puts in a good deal of effort. They research. They ask for samples of his client disk images — all 40GB of them — to be e-mailed. (Steve refuses that one with some choice adjectives.) And then they call him back repeatedly to report that they still don’t know anything. In the meantime, SP2 has inveigled itself into every client-side nook and cranny with the usual results: Several PCs crash outright (ctrl, shift, esc gets you in far enough to uninstall the Service Pack in these instances, for those in the same boat). Several suffer seemingly random application incompatibilities, and Remote Assistance dies a quick and, so far, permanent death. Regardless of Windows firewall settings or whether the firewall is even on, RA is down for the count in both directions.

After dodging a lynch mob of users, Steve sets about finding out what happened. As it turns out, he approved the BITS (Background Intelligent Transfer Service) update shortly before the incident. It further turns out that buried somewhere in Microsoft technical documentation hell is a statement that you can hold back SP2 for 120 days but no real mention of what happens after that. Apparently, what happens is that XP, BITS, and possibly the SUS server — although we haven’t confirmed that yet — decide enough is enough and simply download the service pack for you. Using all clients simultaneously. In the middle of a workday. Fantastic!

Now I know that QA testing is difficult — and pretty much impossible from a third-party perspective for our poor boys and girls in Redmond. But QA within their own product lines, updates or not, really shouldn’t be impossible. The order in which Steve passed his updates can’t be unique, and he’s running on a fairly new and fully up-to-date network on both the client and server side. The screwup brought Steve’s company down for almost a full working day, and he’s only responsible for 300 or so nodes. I shudder to think what might have happened on some larger networks. I also shudder to think that the only way I can keep my pesky XP clients from grabbing whatever software Microsoft puts in its update queue is to hard-block them at the firewall.

“But at least it’s a solution,” mutters Steve, as he cleans the big Microsoft-logo boot print from the seat of his pants. Happy New Year.