Tackle big data with little bites

In response to a recent, wry Advice Line aimed at taking a little air out of the big data hype machine, reader Kayza Kleinman offered an excellent opportunity for follow-up advice on getting started with data analytics:

How does an organization with a serious amount of data that doesn’t reach the heights of big data (i.e., tens of thousands of clients, and millions of service records, rather than 10X or 100X that size) and without the technical resources for a big-data setup (i.e., data mart plus Hadoop or the like), get better analytics from the data they do have? Especially since not all of the data is in one database.

I’m not talking about a cookbook recipe, but a sane approach that can be done on a reasonable budget.

There’s no one right answer to this question. There might not be any right answers to this question, in fact — data warehousing/analytics projects seem to have a special propensity for getting out of hand.

[ Find out how to get a hot job in big data and get familiar with 7 top tools for taming big data. | For more of Bob Lewis’ continuing IT management wisdom, check out his Advice Line newsletter. ]

Here are some guidelines that should help you get started while keeping you out of trouble, beginning with the most important consideration: determining whether your company is ready to take on serious analytics in the first place.

How to tell if your company is ready for data analytics

With or without big data, the criteria provided in “You want big data?” are the best starting point for any effort focused on improving a company’s analytics abilities:

• Does your company have sophisticated statisticians and analysts on staff?
• Do your executives prefer data-driven decision making to “trusting their guts”?
• Overall, does the company have a culture of honest inquiry? (How to tell: If everyone understands that changing your mind when new evidence doesn’t fit your old opinion is a sign of strength rather than weakness, your company has a culture of honest inquiry. Otherwise, it doesn’t.)

Let’s take a closer look at these criteria, one at a time.

Data analytics criterion No. 1: Statisticians and analysts

Cool business intelligence tools let you put together dashboards and other types of interactive reports that anyone can use to “explore the data.” But they aren’t exploring the data for real, any more than someone who buys a book that provides a walking tour of some exotic locale is exploring that region.

The person who wrote the book did the exploring. In the same way, your analysts and statisticians are the people who will “do the exploring” of your data, by creating the dashboards and interactive reports — the walking tours — that executives can dig into.

This isn’t just a matter of business executives and managers lacking the patience to learn your BI tool of choice. It’s a matter of their lacking the patience to understand what constitutes a valid statistical sample and a valid statistical inference.

Giving them direct access to the data and analytical tools reminds me of a story Bill Cosby told a century or so ago, about his mother wanting to drive his Shelby Cobra. As I recall it, his mother got behind the wheel, revved the engine, let out the clutch, and approximately a tenth of a second later found herself several blocks away. At that point, she screeched to a stop, turned off the engine, ambled back to Cosby, handed him the keys, and without saying a word, walked away.

Changing metaphors, if your company’s BI toolkit is a Shelby Cobra, company decision makers will, for the most part, be passengers. Your analysts and statisticians are the drivers.

There are plenty of ways to mess up statistical analyses so that they deliver the wrong answer, especially when using data not originally collected for the purpose of supporting the analyses you’re conducting. Without professionals handling the tricky parts, having analytics will be worse than guessing because they’ll give you authoritative-looking but potentially very wrong answers.

Data analytics criterion No. 2: Evidence-based decision making

The best analytics in the world have no value if nobody uses them to make decisions. That being the case, before spending time, money, and effort on analytics capabilities, take a hard look at your company’s decision makers.

If they’re the sorts of folks who say, in a frustrated tone of voice, “How can we make this decision? We need data!” then get ready to rock ‘n’ roll. But if they’re the sort who instead say, “Here’s what we’re going to do. My gut tells me it’s the right answer,” all of your other priorities — including your ballroom dancing lessons — are more important than the analytics improvement project.

Data analytics criterion No. 3: Culture of honest inquiry

Without a culture of honest inquiry, the executives who prefer data-driven decision making are probably searching for ammunition, not for answers to questions. That is, they’re likely to start with the answer they want and solve for the analysis that supports it.

It’s a more sophisticated form of trusting one’s gut, but in the end, that’s all it is.

Getting started: Small team, adequate resources, full agile ahead

If your company has strong statisticians and analysts, executives who want evidence to support decision making, and a culture of honest inquiry, then it’s time to start building. The question now is how to go about it.

The answer: This isn’t the time to forget everything we ever learned about the advantages of agile over waterfall methodologies.

The best approach I know of for getting started is to pair up one of your best data analysts with one of your best statistical analysts, and get them together with the most gung-ho evidence-driven decision maker in the company. Be sure to set them up with their own virtual server, DBMS instance, and data access rights so that they don’t have to beg for computing resources — then turn them loose.

The role of the decision maker is to raise questions and to have a clear idea of what sort of analysis can provide useful answers. The role of the statistical analyst is to take this clear idea and turn it into statistical tests of available data. The role of the data analyst is to track down where the needed data lives, extract the data, massage it into a usable form, and load it into the private environment.

Nothing about what they do should have any permanence — quite the opposite. Right now you want smart people to muck about in the data, getting their hands dirty up to their elbows so that they can get a feel for the process. You also want to create an environment of haves and have-nots: You want the company’s other decision makers to become jealous because they aren’t getting the same attention.

That’s when you bring your team of three up for air and ask whether the kinds of questions being asked and answered have stabilized enough to turn their ad hoc work into a production environment. Assuming it has, that’s their next task.

Lather, rinse, repeat

Once you have one project stable and ongoing, ask your setup team what they could have done to get to production more quickly, if they had to do it all over again. Because they’re smart people, they’ll have ideas.

Then you can assign the data analyst and statistical analyst to work with the next decision maker in line, to go through the process again, while still providing support to the first one — you don’t want to lose what you’ve built, after all. Go through this a few times and you’ll have generated some solid business value while helping the whole company learn its way into a more sophisticated approach to analytics.

With this growing semi-mess of analytics-oriented data marts, you’ll also be in a much better position to design the “real” system the company needs — and to explain to the executive team what this more elegant system will cost and why it’s worth investing time, money, and effort to build.

Or you might not — this constantly churning collection of data marts might turn out to be exactly what the company needs. Just because it ain’t pretty doesn’t mean it ain’t functional, after all.

This story, “Tackle big data with little bites,” was originally published at InfoWorld.com. Read more of Bob Lewis’ Advice Line blog on InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.