What’s a Tiny Mistake?

This is a topic I’ve been meaning to write on for a while now, and the extremely untimely death of Steve Irwin has put it back in the front of my mind.

I’d like to start though by saying what an incredible loss Steve’s passing is. Steve Irwin was definitely one of the people we really needed kicking around on this planet. He was a devoted family man, and conservationist. We’ve been watching him for years and our entire family got to watch his kids as they grew, and through a lot of the personal episodes they did with him, we actually felt like we got to know him a bit as well. He and Terry clearly shared a love that most of us can only dream of, and you could tell how much she adored and admired him by the way she talked to him and about him. He did so much for animal conservation, and you can really tell what someone’s like when they’re willing to throw themselves on top of an alligator or a cobra. And as long as I’ve been watching him, I’ve always said he’s going to meet his fate at the hand of some animal he was annoying… one day he’s just going to zig when he should’ve zagged, and whammo. Well, unfortunately that’s not how it happened. Steve died in a manner that was equivalent to jumping over an active volcano on your dirt bike, and then dying of a bee sting on the way home. This is an extremely untimely death, and they never make any sense. My heart truly goes out to Terry and the rest of his family.

Now, about now, you’re asking yourself what all of this has to do with databases. Well, it’s actually a topic that’s pretty relevant. The question at hand is what constitutes a big change to your database? See, we had an incident a short while back that illustrates this. There was a query that had been running fine. Well, fine for that query anyway, but it was returning in about 10mins. Then for some reason, the developer went and made a simple little change. He changed part of the query from ‘or (col1 = 17)’ to ‘or col1 = 17’. That’s a pretty small change. However, it took his query from 10mins to over 6hrs. We actually have no idea how long the query was because it was just killed after 6hrs.

The point is that you never know what tiny little change is going to do in your environment. What may seem small could be enormous. The size of the change isn’t measured by the number of keystrokes it took to achieve. We’ve all played fast and loose in our databases. We’ve made very large changes that we either didn’t test at all, or not very well, and pushed them right into production. Somehow we always come out of it ok. But you do one little thing that shouldn’t make that big of a difference, and it kills your performance. Just like Steve did. I can’t count the number of times I saw him annoy some poisonous animal for our amusement. He’s lain down in a huge group of hungry kimodo dragons, been bitten by a croc, I think I even remember seeing him get pierced on the hand by a cobra. The list goes on and on. Yet, he goes swimming with some sting rays, and gets killed. One of the most docile fish in the ocean takes him out. All the things he’s done, and he gets taken out by the equivalent of a clown fish. I’ve seen several places let kids walk around in huge pools with these things without even removing their stingers. They’re just not vicious.

Anyway though, the more longer I stay in this field, the more afraid I am of those changes that are the easiest to make. Not only because they can have such hidden effects, but also because they’re so tempting to push into production without thoroughly testing. So the next time you’re making a tiny change that seems insignificant, just remember the disastrous effects it can have.

And Godspeed Steve-O, we love you.