The value of data-driven decison-making

Dear Bob …

Are you aware of any data showing whether there are demonstrable benefits from making decisions based on data, especially in a corporate or better yet an IT environment?

My boss and I were discussing how to get the rest of our team to strengthen the habit of analyzing problems first (“How big is the impact of this problem?” “Have we verified that this really is a problem?”), including gathering data about them, and then analyzing possible solutions, before jumping straight into the “fun part” of implementing the first technical solution that comes to mind that seems to solve the reported problem.

Of course the first thing we started doing over a year ago when our group was formed was modeling this behavior, since it’s how we’ve personally done things in the past. Now that we’ve been running for a while, we’ve realized we need to explicitly ask people to do the same, probably including it as one of the key credos for our group. We have already been working with people to coach them on how to do this in the context of specific issues as they come up, but we haven’t said “Hey, this approach is how we expect you will do things.” Now we’re just about to that point.

The explicit culture of the corporation certainly supports this (one of our 5 key values is “be science-based.”) I’m sure you won’t be shocked to hear that this attitude is not fully ingrained throughout our entire IT department, which has too often in the past rewarded firefighting and heroic efforts over the silent, steady work that just “keeps the joint running”.

Ok, enough preamble. I have two questions for you. First, any general advice on how to get this change in behavior through? Second, and the main reason I’m writing to you: are you aware of any data showing whether there are demonstrable benefits from making decisions based on data, especially in a corporate or better yet an IT environment?

– Data Dan

Dear Data Dan …

Studies? No. And it strikes me that there’s a circularity here: In order for your staff to appreciate such a study, they’d first have to appreciate the value of data-driven decision-making. At which point they wouldn’t need the study.

I’d like to emphasize that I don’t endorse data-driven decision-making anyway. I’m being a purist here: What’s needed is theory-driven decision-making, where nothing is considered a theory until it’s backed by data. The problem with data on its own is the forest/trees problem; the problem with taking data to the next level … pattern-recognition or correlation … is how easy it is to confuse correlation with causation. (This, by the way, is why Jim Collins’ book Good to Great isn’t great, although it is pretty good).

So what’s really needed for great decision-making is a set of clear principles, warranted by hard data that demonstrates they work in practice.

The most important “study” I know of that supports the value of data-driven decision-making is the entire field of engineering. Any engineer who just goes with gut feeling when building a bridge, for example, will build a bridge I don’t want to drive across. Nor would I want to cross a bridge built by an engineer guided by “principles” that haven’t been validated by experimental data.

What complicates this whole subject immensely is that where in science and engineering there’s a whole system of checks and balances that helps ensure published research adheres to high standards of data collection and analysis, in business the entire system consists of making sure the HTML tags render properly so as to present whatever damfool “data” and notions an author wants to persuade you is true.

Which is to say, in order to make data-driven decisions in business, you first have to be very careful about the data on which you base your decisions, if it isn’t data you haven’t collected and analyzed yourself.

The last piece of this puzzle is that in business you have to recognize the point of diminishing returns – when you’ve collected all the data it’s practical to collect, have performed as much analysis as it’s practical to perform, and you have to make a decision anyway, in spite of the remaining uncertainty. Reasonable people will disagree as to when you’ve reached the point of diminishing returns, of course. They will agree that there is such a point.

Or as I explained in a column awhile ago, business managers should model themselves on good bridge or poker players, able to make decisions in the face of uncertainty, based on the information that is available, the odds, and their knowledge and judgment of how the other players think and act.

– Bob

PS: I left out the whole subject of making the change happen. I’ll cover that in my next posting.