NoSQL: Shadow IT that could sink you

In case you’ve been sleeping on Mars and missed the excitement, today’s hot industry trends (HITs) in IT center around devices, the cloud, and data.

Specifically, we’re talking BYOD (bring your own device); the big three among the ever-proliferating collection of cloud technologies: IaaS (infrastructure as a service), PaaS (platform as a service), and SaaS (software as a service); and big data, along with the concomitant collection of “NoSQL” technologies that support it.

[ Andrew C. Oliver answers the question on everyone’s mind: Which freaking database should I use? | Also on InfoWorld: The time for NoSQL standards is now and 7 hard truths about the NoSQL revolution. ]

Just about everyone with an opinion and a soapbox, including me, has broken down IT’s options when it comes to the HITs: IT has to lead, follow, or get out of the way. More accurately, IT has to lead, ignore, prevent, or — the most likely and labor-intensive alternative — support them.

This is the easy stuff. What’s been missing in most of these discussions is something central to IT’s purpose and pay: integration. When it comes to integration, NoSQL is where the most important IT challenges reside right now.

Integration — what IT does

Leave the HITs aside for a few moments and look at the array of technologies IT supports already. Up to and including the telephone system, what they all have in common is that they aren’t isolated islands of automation. They’re integrated, with each other and into the business.

Relational database management systems, for example, don’t stand alone. They provide services (transaction processing and query processing) used by other components within the enterprise technical architecture. The same is true for every other technical platform, for the various databases and information repositories that reside on them, and for the applications that comprise the company’s applications portfolio.

And they’re integrated into the business. Email, for example, isn’t just a technology IT manages. It’s a service IT manages for the company, and it’s integrated deeply into how the business operates. This is also true of every other application IT supports and, by extension, the technologies used by these applications.

Whether integration matters for a particular HIT is what determines how business departments will involve IT. If integration isn’t important, they’ll usually ignore IT and acquire the technology themselves, as they’ve been doing with Salesforce.com. If integration is important, they’ll mostly complain that IT didn’t anticipate their interest and have everything they now know they want ready and waiting for them.

Non-IT IT

As my InfoWorld colleague Eric Knorr wrote recently, most of the growth in IT spending is projected to occur outside the IT budget — 90 percent was the number he reported, leaving 10 percent for IT to enjoy. That’s growth, not total spend. As it isn’t likely that business leaders are going to wake up, smell the coffee, and say, “Hey, as long as it doesn’t go into the IT budget the checkbook is open for IT spending!” we aren’t talking about that much money.

This doesn’t make the number unimportant, but the picture is considerably more complicated than the “business is losing confidence in IT” conclusion so many commentators have drawn. The reason is that the (probably temporary) rise in non-integration is driving a lot of the trend.

Here’s my projection: Of the non-IT IT spend, the lion’s share will go to SaaS applications where the spenders don’t care whether it’s integrated. Salesforce.com will continue to dominate this category.

Of the rest, a lot will go into tablets — a technology that’s (1) fun, (2) personal, and (3) integrated, but not very much. Mostly, we’re talking about the email, directory and calendar systems, and that integration will be part of the 10 percent growth going to IT — if IT is lucky and the CIO is politically adept.

That leaves big data/NoSQL for the rest. And NoSQL poses a problem for a lot of IT professionals.

The NoSQL challenge

Relational database management technologies and the data normalization techniques associated with them aren’t just tools used for certain kinds of jobs. The RDBMS is baked into the IT culture by now — it’s How We Do Things Around Here.

It should be — when the job is transaction processing. When it isn’t, data designers have, over the past few decades, developed a number of techniques for finessing the shortcomings of this meat-and-potatoes technology. It’s How We Do Things Around Here.

But these are finesses rather than elegant solutions. Enter the hugely countercultural family of technologies known as NoSQL.

InfoWorld’s Andrew Oliver wrote an excellent snapshot of various NoSQL technologies a few weeks back. In it, he explained the sorts of data design challenges each of them addresses.

The missing piece: Data design challenges are one thing. Business situations are another, and I’ve yet to run across a clear, concise explanation of what business challenges each NoSQL technology is optimized for, let alone a clear account of what the complete packages look like that let you load data into them, then process it so as to address those challenges.

How this NoSQL challenge shakes up IT

More to the point from where you sit as an IT professional: If your CIO were to ask you to explain how your company might take advantage of NoSQL technology — of the business challenges the company faces, which NoSQL datastore could help address them and how, and what would be required to integrate that NoSQL technology into the rest of the enterprise technical architecture — could you answer the question?

If so, good for you. My guess, based on the informal contacts and conversations I’ve had on the subject, is that fewer InfoWorld readers can than can’t, and InfoWorld’s readers are more sophisticated than most on average.

If you’re among those who can’t, you have two alternatives. One is to keep on doing what you’ve been doing and hope someone else gets the question. That is, you can hide behind the herd and hope a different antelope gets picked off.

Or as W.C. Fields once said, you can take the bull by the tail and face the situation: You can approach your boss and explain what NoSQL is about. You can also state that, right now, not only does your IT organization lack competence in NoSQL, it doesn’t have enough knowledge to say what competence might look like. In other words, you don’t know what you don’t know.

Propose to fix this — to do the research, thinking, and planning needed to figure out which ones might matter to your company, why they might matter, and what it would take to bring them in and integrate them. You’d like four hours a week for a month to invest in this research, which would potentially be highly beneficial to the company, not to mention for your career.

If IT doesn’t make this investment somehow or other, here’s what will happen instead: A business executive — the CMO, perhaps — will get a sales pitch from a company that “does NoSQL” or hear a speech from one at a conference. He’ll sign a contract with them to bring it in and make it happen.

They’ll do it. Part of making it happen will be dumping tasks into the IT request queue (the integration parts), at which point any delays will lead to IT being blamed for the resulting overruns in the NoSQL project.

All in all, getting ahead of the game seems better, don’t you think?