The second tech team makes life worse for the first

At the time of this story, I worked as a sys admin at a large company. My team managed a suite of critical tools on our customers’ machines. One day, our company signed a multi-million-dollar contract with a new customer, a very large corporation. Our team was understaffed already, so our company’s management created and trained another team — aka the B Team — to maintain this suite of critical tools for the new customer.

At first, the arrangement seemed great. But the B Team proved to be so inept that management was soon calling on us for emergency interventions and serious mess-cleaning.

[ Want to find out when the latest tale of IT terror hits? Get a new tech tale in your inbox every week in InfoWorld’s Off the Record newsletter | Follow InfoWorld Off the Record on Twitter. ]

One Sunday around 7 p.m., I was the primary staffer on call, and my cell phone started ringing. It was my manager, asking me to please do a favor for the B Team, which had a problem it couldn’t solve.

A few minutes later the B Team called and explained the problem: The team had restarted a Unix server and couldn’t bring the tool back up. The B Team techs had been trying to bring the tool back up for nine hours, and the maintenance window would be over in a few hours.

I didn’t even have access to the servers, so I tried to diagnose the problem over the phone, dictating commands and having them read the output to me. After a few minutes it became clear that they didn’t know anything about the tool — it turned out they were trying to start the wrong server component.

The conversation continued, and they kept bringing new people in to the conference call, “just in case.” They spent almost an hour trying to find a DB2 admin, although I told them that the tool didn’t work with DB2. It took me three hours to get them to write the proper commands and get the thing restarted. It was all downhill from there.

One day at about 8:20 a.m., I was on my way to work. Again, I was primary on-call, and my cell phone started ringing. It was somebody from the B Team. She told me that the upgrade had already started and that I was due to begin my task in 40 minutes.

This was outrageous — I hadn’t been informed of this upgrade. Company regulations stated that I must approve, with several days’ notice, any upgrade or operation that I will carry out or take part in. I guessed the B Team wanted me to do their job because they suddenly realized that they didn’t know how to handle it.

I refused to take part in a major upgrade that I was not informed of, in an environment that I didn’t know, and for a corporation for which I wasn’t even providing support.

As soon as I reached the office, I received a call from the B Team’s manager, politely requesting me to do my part, though I still didn’t know what they wanted me to do. I told him no. He started screaming and ordered me to do it. I refused once more, and he promised to call my manager and give me hell. My manager, thankfully, came from a technical background and took my side. Later, I learned that the B Team got somebody from the customer’s IT team to do their job.

[ Get a $50 American Express gift cheque if we publish your tech experiences. Send your story of a lesson learned, of dealing with frustrating coworkers or end-users, or a story that illustrates a relevant takeaway to today’s IT profession to offtherecord@infoworld.com. ]

A few days later, I went on vacation. When I came back, I got a request to investigate an incident with the new customer’s tools. I got a hold of chat logs and command histories, and I was astonished at what I saw. While I was on vacation, the tool’s primary server component failed, and it lost connection to some of its clients.

The B Team came to the rescue, but instead of simply recycling the server component — which takes about 7 seconds — they messed around with settings. They changed the TCP communications ports on the server, then recycled the server. The server came back up, but with the new settings, it couldn’t communicate with any of the clients. At that point, the rescue team panicked and called the customer’s IT team to explain the problem and get them to solve it. Of course, the problem was solved by restoring the original settings.

I left that company shortly after, but I got word that because of incidents like these, the customer was seriously considering rescinding the contract.

I understand the need for diplomacy and the fact that management may have been trying to solve the situation in ways I was not aware of — all we saw were cosmetic attempts to make it look like they tried to fix the problem. But in these and other cases, the situation was so serious that there was no time for diplomacy. Scheduled backups were not taking place, and lack of maintenance was putting the customer’s data at a very real risk. The B Team’s incredible lack of basic monitoring came very close to destroying the customer’s main production environment.

I guess you can’t change things in large corporations, at least not if you’re in the trenches. The problems with the B Team were known by our managers, who should have taken immediate action, diplomacy be damned. But to summarize it nicely: In an environment dominated by PR, appearances are valued the most — until something breaks and the real price tag shows its face.

This story, “The second tech team makes life worse for the first,” was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.