Finding new value in data overload

Business intelligence is almost always among the top technology priorities of big companies. Yet by most accounts, satisfaction with business intelligence often falls short of expectations. Reasons abound: poor data quality, cost overruns, inaccessible data sources, and our old friend, miscommunication of requirements.

But businesses don’t like to fly without instruments. So year after year, companies pony up for business intelligence initiatives despite lackluster results.

A recent technology surge promises break this unfortunate cycle. At InfoWorld we find these new developments among the most interesting we’ve encountered in a while, which is why we’re launching an iGuide to the new business intelligence, a graphical portal that aggregates business intelligence articles and analysis.

Big Data dreams The hottest new technology trend goes by the nickname Big Data. Conventional business intelligence software operates on SQL data; Big Data processing crunches on semi-structured information, from social networking clickstreams to security event logs. Hadoop, an open source software framework for distributed processing, is the best-known technology for processing unstructured data, but there are others, including GemFire, MarkLogic, and neo4j.

When I interviewed IBM’s Rod Smith a few weeks ago, he characterized Big Data processing as “an exploratory tool that we haven’t had before” to “help people sift through data where maybe 90 percent of it is not very useful.” With the aid of data visualization and data mining, domain experts can then determine for themselves what’s worth exploring using conventional business intelligence tools. (Big Data processing techniques, by the way, were instrumental in Watson’s recent “Jeopardy” victory.)

For me, the data visualization aspect is the most fascinating. With the right visual tools, everyone from medical researchers to fraud investigators can wade into terabytes of the stuff to find useful patterns. And several visionaries see in the hyperabundance of Internet data as an opportunity to create a new, graphical communications medium (check out the work of David McCandless).

Practical intelligence More down to earth is the so-called operational business intelligence trend. Here, the idea is to process sets of (mostly) transactional data for specific lines of business, so line-of-business managers can stay on top of their division’s performance and make quick adjustments as needed.

Operational business intelligence — often the conventional variety, too — require a high level of data quality and freshness, with data integrated and reconciled from multiple sources.  New techniques that use semantic mapping for data mediation are helping to ensure that data from multiple systems integrates correctly to yield meaningful results.

One reason companies find themselves disappointed in business intelligence is that they end up with pretty reports that simply provide a clear view of the past. Predictive analytics, after years of promise, is finally delivering models that can foretell customer behavior — often using semistructured data from Web logs as a source.

And finally, those who want to avoid the investment in hardware and software to deploy business intelligence are turning to SaaS (software-as-service) providers. SaaS vendors such as Actuate, Good Data, and Jaspersoft are breaking new ground with high-end business intelligence features wrapped in intuitive Web UIs.

It’s hard to imagine a more exciting time for business intelligence. With effective new tools, the data glut that everyone laments becomes a new source of business value. As with our other iGudes, we’ll be adding new material to the portal over time.

This article, “Finding new value in data overload,” originally appeared at InfoWorld.com. Read more of Eric Knorr’s Modernizing IT blog, and for the latest business technology news, follow InfoWorld on Twitter.