A campus data center move meets tech incompetence

After some years working in IT departments for various corporations, I made a change and started teaching at a local university. I found myself in an interesting situation: observing the school’s tech decisions as mainly an outsider but still with an ear to the ground of what was going on.

The school’s IT team was a mixture of expertise levels. The senior managers were holdovers from a largely pretech world and often didn’t understand what their young staff and middle managers were talking about. These senior managers were experts in certain areas, but they weren’t up to date on current technology and left a lot of those decisions to their staff. Critical decisions could be made by any number of people on the command chain, and often the senior managers believed about any argument told them by their staff, which then became the official position. Needless to say, this culture made for some chaotic tech situations.

[ Have you encountered the technology pro’s six greatest enemies? Send your memorable tale to offtherecord@infoworld.com. If we publish it, we’ll send you a $50 American Express gift cheque. | Get a new tech tale delivered to your inbox every week in InfoWorld’s Off the Record newsletter. ]

One semester, the IT team implemented a new Web-based course management system to supplement for the face-to-face classes. Many faculty enthusiastically embraced the new system.

But then, the tech team made plans to move the school’s data center into a new building. The moving day was set for right after final exams so that classes wouldn’t be disrupted. 

However, the day before final exams, a thunderstorm swept through the area and caused a power outage. We momentarily panicked. No electricity on the last day of classes could be a disaster. We were all relieved when the lights came back on in just a couple minutes.

But the network didn’t come back to life with the lights. No problem, I kept thinking. The wireless routers probably just needed a couple minutes to reset. Maybe our computers just needed a restart.

The network still didn’t come back. After 10 or 15 minutes we got concerned. After an hour we got confused. Finally, the network came back to life, but the course management system wasn’t there. The URL got no response.

It took a while for the complete story to emerge, but what had happened was that someone had decided to move the UPS (uninterruptible power supply) to the new data center ahead of time to get things set for the next week’s move. I guess it never crossed their mind that we could have a power outage that week. Either that or they were too cheap to buy or rent a backup UPS.

When the power went out, the course management server crashed — and crashed hard. Final exams were almost over by the time it was running again, and some of the data never came back. Several faculty had no offline backup of their final exams or gradebooks and canceled their exams. Many grades were turned in late. As you can imagine, there were a lot of angry teachers, students, and parents.

In my corporate days, heads would have rolled over something like this, but no one was reprimanded. No one even offered a formal apology. The attitude from the IT team’s senior managers was “stuff like this happens, deal with it.”

Given that attitude, it shouldn’t have been a surprise the next year when our primary network drive crashed and was restored. None of the faculty or staff were notified that there had been a crash, but we logged in one day and all our Date Modified fields were at least nine months old and a lot of our files were missing. It came out that the automatic backup hadn’t actually backed anything up for the last nine months. No knew because no one from the IT team ever checked to see if the backups were working.

Eventually, the school’s administrators recognized the need to make a change to the campus’ IT management.

The moral of the story: If you’re going to be incompetent, make sure your boss knows less than you do and you won’t get into trouble. If your organization has incompetent people? Well, I have my own 500GB external drive in my office, I run my own backup software, and I check to make sure it works. 

This story “A campus data center move meets tech incompetence,” was originally published at InfoWorld.com. Read more crazy-but-true stories in the anonymous Off the Record blog at InfoWorld.com.