Don’t expect big-time results from big data

Big data is a big deal these days. It’s all about storing and mining massive data sets that defy traditional storage models and technologies — in many cases without the same level of design and planning that traditional data warehouses require.

Big data is the third hot industry trend to be analyzed in this space. As with the past two weeks, when we analayzed cloud services and BYOD, the question doesn’t center on potential value. Instead, the focus is on its potential for industry acceptance, which is quite a different matter.

Just to make sure we’re talking about the same subject, while big data overlaps with NoSQL database systems — another set of technologies that’s been getting attention recently — this analysis is limited to big data.

To recap: For any new technology to have a chance of success it must clear three hurdles.

• The customer and consumer can’t be different people. (Reminder: Customers make buying decisions about a product or service, in contrast to consumers, who are the people who use it.)
• The “wallet” (the source of money) can’t find the expenditure off-putting.
• The technology can’t be disruptive when mixed with the installed base.

Let’s look at how big data fares.

Breaking down big data

Examining big data from along the three axes outlined above, we get the following:

Customer vs. consumer: Big data’s consumers are business executives and managers, often the COO or CMO, who want to be smarter about their business than their competitors are. They’re the ones who will dig into the data with picks and shovels to mine for nuggets of gold (more accurately, analysts, statisticians, or “data scientists” will dig into it for them).

Who makes the buying decision? This can go one of two ways. Either the executive who wants big data asks IT to take the technical lead, at which point IT becomes the actual customer (IT decides what to buy), or the exec engages an outside big data consulting firm, at which point the consulting firm becomes the customer.

Either way, while the customer and consumer aren’t the same person, they are different in a relatively minor way. The consumer (the COO or CMO) still makes the decision to buy and so occupies the most important part of the overall customer role. IT or the consulting firm’s role is limited to deciding what to buy.

Wallet impact: This is a bit more complicated.

Investing in big data takes more than downloading Hadoop from the Apache website and installing it on a server cluster. It’s a major organizational undertaking that requires:

• Staffing: Money, because data scientists don’t come cheap.
• Education: More money, because retraining your analysts so they know how to deal with, not just a new technology but a new philosophy of dealing with data, is a nontrivial and not-inexpensive undertaking.
• Money: Even more of it, whether big data is implemented using on-premises equipment or the implementation goes into the cloud. On-premises implementations will need capex to cover storage and server capacity. Cloud implementations will spend opex on storage, server capacity, and either newer, faster network pipes or a boatload of flash drives transported via “FedEx Net” to the cloud provider of choice (some companies reportedly take this exact approach). None of this is free just because it’s on the other side of the Internet.

The new staff and retraining, and (if you go this way) cloud storage, server capacity, and network pipes or flash drives have significant collective budget impact. That means CFO approval will be required.

Capital investment (if you go on-premises) means the dreaded capex committee will have to be involved. Running a proposal through the capex committee is, in most companies, a lot like running across hot coals — it takes confidence and a lot of calluses. Even then, you’re likely to get burned.

With big data, the CFO and very likely the capex committee are wallets — the money comes out of operating and capital budgets. This will be a much bigger hurdle than big data’s relatively minor customer/consumer disconnect.

Disruption: In most companies, big data will be useful only if it’s disruptive.

Not technically — while it will take a significant investment in technology, big data systems will mostly be walled off, with data flowing in but not out. But culturally, big data will either be disruptive or worse than worthless.

Big data won’t be disruptive in those relatively scarce, healthy businesses that have a culture of honest inquiry in which the evidence leads and the company follows it. In these sadly rare organizations, executives and managers are hungry for information because it will help them make better decisions. For them, there’s no cultural disruption and big data can be a great fit.

Most businesses are far away from this — their executives and managers either make decisions by “trusting their guts,” or they make decisions by trusting their guts, then solve for the answer they’ve already decided is right. For these decision makers, in other words, evidence is used as ammunition, not for illumination.

The ones who trust their guts will have no use for big data — because what part of “they trust their guts” wasn’t clear?

The ammunition gatherers will be worse; instead of just arguing like they do right now, they’ll present convincing charts and graphs that “prove” they’re right about whatever it is. But big data’s contribution has been negative, because their actual accuracy hasn’t changed — only their persuasiveness.

Assessing big data’s long-term viability

From the perspective of immediate industry acceptance, the big barrier to big data is that the wallets — the CFO and capex committee — are likely to see it as big cost, without any clear value in return.

Should big data clear this hurdle, it’s good to go. Because while it will (or at least should) be highly disruptive to the average company, the need for this disruption will mostly go unnoticed: Few executives will change how they make decisions, just because they can.

My prediction: Big data will be here for the long term, but its success will be modest rather than spectacular. That’s the technology, as measured by sales and implementations.

Value is a different matter. For most companies that head down the big data path, the value will never be entirely clear, and the whole outcome will be vaguely but unsettlingly disappointing.

Those will be the lucky ones. For some it will be much worse, but that story will have to wait until next week.

This story, “Don’t expect big-time results from big data,” was originally published at InfoWorld.com.