Why the big data backlash is bad for IT

The big data trend has met with serious pushback in many IT departments. The testy version goes something like this: “Enough already. I have plenty of SQL data stores with lots of data, and I have all the business intelligence tools my business needs. So exactly what are you taking about?”

Sure, by now everyone has heard of Hadoop and the insight businesses have obtained from mining semi-structured data, but in many cases that value proposition is lost on conventional IT folks. I believe this intransigence is part of the reason an increasing portion of the IT spend is slipping from the hands of the CIO to the CMO and line-of-business managers.

[ Get the latest insight on the tech news that matters from InfoWorld’s Tech Watch blog. | Subscribe to InfoWorld’s Consumerization of IT newsletter today. ]

Last week I interviewed Rod Smith, vice president of emerging technologies for IBM, who has been deeply involved in IBM’s big data practice and product development for four years. I asked him why the resistance seems so entrenched in some cases.

As an example, Smith cited a big data Twitter engagement, where a “very large customer in the financial area” with 150,000 business partners wanted to monitor Twitter for information about potential new partners. If, for instance, a new restaurant tweeted that it got an award, it might be targeted with a congratulatory message as a way to begin a new customer relationship involving a loan or credit card.

The reaction of IT was less than enthusiastic, according to Smith, who observed resistance to seeing the Twitter app as a real application with real value:

It’s hard to convince the IT guys they’re going to do something where, outside, you can get to the Twitter data or the Facebook data. And it’s fleeting. It might be useful for a particular campaign you’re going to do or the launch of a product, but then it might be disposable at that point. So that makes it really tough for the IT guys to say, yeah, we’re going to spend with you here, because we know that we’ll build it and you’ll use it will have that value over and over again. They can’t touch it or feel it the same way anymore.

No surprise, then, that many of IBM’s big data engagements begin with the business side. “A line of business or the CMO doesn’t want to wait 18 months to have their application on some backlog, while the IT guy tries to figure out whether there’s real business value there.”

Last month I went to the O’Reilly Strata Conference on big data and saw a similar pattern. Many customers were using big data tools to analyze Web user behavior — to optimize Web app usability, shorten the path to online transactions, or serve the right Web ad at the right time. But quite often those customers were outsourcing both the analytics and the Web apps generating the clickstream data to be analyzed, rather than going through internal IT channels.

Whether or not IT leads the charge, the level of adoption is a fast-moving target. When I asked Smith what proportion of his customers have gone past the pilot phase and see big data analytics as something essential to what they do, I was surprised to hear him estimate 70 percent. Quite often in the past four or five months when he calls on a customer, “they introduce me to their big data architect.”

No doubt this reflects the circles Smith travels in. A recent study commissioned by the U.K. firm Interxion seemed to indicate the opposite: Only 7 percent of businesses saw big data as a priority today. But a whopping 62 percent of the same 750 respondents believe it will become a priority within the next three years.

Smith told me that not just e-commerce operations, but brick-and-mortar retailers are embracing big data as a way of profiling customers and, ultimately, creating recommendation engines that will point in-store customers to special deals of interest to them in real time. Smith is clearly most excited by the possibilities in health care, citing a recent Seton Heart Institute pilot program where analytics were run on EMR data from patients who had suffered congestive heart failure to help predict who would be likely to be readmitted.

I’ve lived through several bubbles that have burst, but I don’t think big data is one of them. Yes, there have been and will be clumsy attempts at big data analytics that yield little of value. But as expertise and software improve, the payoff will become more and more obvious — not just for marketing, but for verticals such as health care, manufacturing, finance, media, you name it.

Some IT departments will embrace the experimental, iterative nature of big data analytics, and some will shrug it off as a fad. I’m betting that the latter will end up being less important to the organization they serve.

This article, “Why the big data backlash is bad for IT,” originally appeared at InfoWorld.com. Read more of Eric Knorr’s Modernizing IT blog. For the latest business technology news, follow InfoWorld on Twitter.