The high price of data loss

It’s hard to say which was worse: the humiliation of losing the data or the punishment from the company we hired to recover it

We maintain a storage server for our users home directories and as hard drives do, ours crashed. Not a problem, we thought. We had backup tapes, right? Yes, we had them but they … didn’t … seem … to have any data …

(Well, one of them had data but the backup was two months old – we’d somehow managed to lose three months’ worth of our users’ data. We had been creating backups and not getting any error messages, so we thought we were fine. We hadn’t inspected the tapes to verify whether they were good. Yes, we were idiots, but at least we knew we were lucky to still have jobs.)

My network administrator responded to the following advertisement:

Expedited service – $299 diagnostics fee and a 40 percent surcharge on top of the recovery cost. Diagnostics is [sic] complete within 4 business hours. This service involves an estimated turn-around time of 1-4 days, where work is performed Mon-Sun 9am-12 am. A dedicated engineer is assigned who will provide an update every 4 hours on the progress

Of course we needed “expedited service”; our users needed their files. We paid the $299 with a credit card and overnighted the two hard drives with 160 gigs of data to Cleveland, Ohio, for recovery evaluation.

The next day we called the company, who told us they had received the drives. That afternoon, when we did not receive a call as promised, we called them. They said they would have to order a part to get the drives running. Such was the response for the next 16 days. And always after we called them because their every-four-hour phone call never happened. Not even once.

Finally on Friday, day 16, around 1:00 p.m., the company contacted my network administrator and pointed him toward a Web site to verify the files they had recovered. Eureka, they were ours!

But there was a catch. They wouldn’t let us pay the $8,400 fee with our credit card. They wanted us to wire funds from our bank to theirs. I’d never heard of such. Our accounting department informed me that 2:00 p.m. was the cutoff to wire funds. I really wanted this data to be shipped so I’d have it on Saturday and the network administrator could have it restored on the network by Monday morning.

I called the drive recovery company, thinking I could talk good common sense and get them to ship my data. “I’m not trying to cheat your company, I’m trying to be fair to myself,” I said. “It’s been 16 days on the 1 – 4 day estimate and now because of the weird payment requirements, it’s going to take four more days. If you had informed us of the payment requirements, we could have had it arranged.”

Never in my life had I had such a good argument and not gotten what I wanted. The company would not ship the data without payment, and that payment could not come via credit card. They wouldn’t budge.

I had to call my boss, the CFO, whose mother had just passed away the day before and so he couldn’t offer much help. We tried Western Union online but they had a 2,999.99 limit. We tried a local office that had Western Union service; they wanted cash only. Then we tried Western Union directly and they could wire the funds to a Western Union office in Cleveland for local pickup. They would need the name of the employee who was going to pick up the funds and that person’s social security number for verification. The company balked at that idea so we had to wait until Monday.

The funds were wired on Monday. Shortly thereafter, I received a call asking how I wanted the data shipped. “D’uh,” I thought. “Overnight, you twits!” Instead, I merely said, “Overnight.”

“That will be $45. How do you want to pay for it?”

(We’ve changed to a raid 5 configuration with a two week backup rotation that will be verified. Files older than 3 years will be archived off to CD-ROM to reduce the size of the backup.)