Cloudera CEO: We’re taking Hadoop beyond MapReduce

From the number of times you’ve heard the word “Hadoop,” you’d think it referred to some magic elixir for making sense of big data. In reality, Hadoop is an open source framework for distributed data storage and processing — with enormous analytics potential for those who know how to use it.

To demystify Hadoop, and to get a personal perspective from one of the leading lights in the space, IDG Enterprise Chief Content Officer John Gallant and InfoWorld Editor in Chief Eric Knorr turned to Mike Olson, CEO of Cloudera. The hour-long interview, an edited version of which appears below, is part of the ongoing IDG Enterprise CEO Interview Series.

Olson began his career in the ’80s and ’90s building and then later selling and managing companies that developed relational database products. In 2000 he became CEO of Sleepycat Software, makers of the open source embedded database engine Berkeley DB. He later negotiated the sale of Sleepycat to Oracle in 2006. Olson stayed with Oracle as vice president of embedded technologies for two years; shortly after departing, he “stumbled across” Hadoop. “When I saw how it was being used in the consumer Internet…I got excited and thought there would be an opportunity in traditional enterprises,” he says.

As it turned out, three other entrepreneurs — Christophe Bisciglia (Google), Amr Awadallah (Yahoo), and Jeff Hammerbacher (Facebook) — all felt inspired to start a Hadoop venture at roughly the same time. “We banded together in the summer of 2008 to create just one, rather than four, such companies,” says Olson. A year later, Doug Cutting, co-creator of the Hadoop project itself, joined Cloudera as chief architect.

With an impeccable pedigree, and customers the likes of Chevron, eBay, Monsanto, Morgan Stanley, and Samsung, Cloudera is widely considered the leader among Hadoop pure plays. We began the interview by asking Olson about the relevance of Hadoop to enterprise customers.

Q: Hadoop is coming up in conversation everywhere. What is it our readers should absolutely understand about Hadoop?

A: Hadoop is an open source platform based on the work at Google, originally for indexing the Web and digesting user behavior. Hadoop is the open source version of that.

Hadoop consists of two pieces. There is large, reliable, cheap, scale-out storage. Gang a bunch of Dell or Hewlett-Packard computers together, all of them with local disk, and store as much data as you want. If you need to store a little more data, buy a couple more boxes — really incrementally scalable.

The software that manages that storage, HDFS, the Hadoop Distributed File System, it’s smart. You’ve got a thousand computers ganged together, you know for sure one is going to fail. No problem. The software has been designed to watch for a disk that goes south or a server that goes offline. Multiple copies of the data are stored around; automatically they’re re-replicated. You can lose individual pieces and not lose data, and as you need to expand your pool you just add a couple more boxes. You don’t need to move all the data to the new thing; it’s all automatic.

What makes Hadoop really interesting is the second component: an engine called MapReduce, the data processing layer. The idea is, you’ve got all this data spread out among all these machines. Used to be in the database world, if you wanted to ask a question, you got the data off the disk and moved it to the question-answering component. MapReduce is different. You’ve got data spread across 10 or 100 or 1,000 computers, so when you want to ask a question you send that question out to all the computers and they look at the little fragment of data they’ve got and they shoot their answers out. Then you collect the answers at the end.

At base, Hadoop is large-scale distributed storage and a way to spread computation out, to spread questions out across that storage. You can literally ask a question of a petabyte of data, 1,000 terabytes of data, and get an answer — now. That’s what Hadoop is.

The MapReduce framework can do brute-force data processing — clean my data, summarize my data, analyze or digest it. It can also run powerful algorithms. We want to use machine learning over all of that user behavior in the weblogs. I want to understand what it is, John, that you like to browse on the Yahoo website. Then I want to see what Mike likes to browse on the Yahoo website. To the extent we like the same thing, I want to recommend content for you based on stuff that Mike looked at.

The MapReduce framework is really good at very complicated workloads — machine learning, natural language processing — that can be pointed at a bunch of different business problems. That’s why Hadoop has done so well.

Q: How are people misunderstanding Hadoop?

A: A knock against Hadoop is that not all problems are MapReduce problems. It’s this big batch data processing algorithm. It’s really heavyweight and you’ve got to be a specialist to use it. It’s batch mode, so you ask a question, you wait a while before you get any answers back. Not all data access is like that.

It has been a misconception about Hadoop that the combination of HDFS and MapReduce are your only choices. Those are the two pieces that Google built first. But once you’ve got your data spread out among a thousand machines, you can imagine other things you’d like to do with it. Wouldn’t you want to ask interactive speed questions about it?

We’ve built Cloudera Impala, a new open source project that we launched a few weeks back. We’ve built an interactive speed, high-performance, distributed query engine. It actually is another example of software originally designed at Google.

The idea is you’ve got one big pool of storage. Put anything you like in there and you can use the MapReduce framework to do the analytics and the data processing. You can equally now use the Impala framework to ask interactive-speed queries, and you can share data between those two; you can run queries that create results that you then MapReduce. You can use MapReduce to analyze data that you then query. That combination is huge.

What really matters about it isn’t that Impala makes SQL queries fast on Hadoop, although that’s a big deal that opens up a lot of new use cases. What’s really important here is that Hadoop is a general-purpose data storage and processing engine. We expect the community will create new engines that get at that same data in new ways.

Q: Essentially you’ll be able to provide a portfolio of ways to look at that giant pool of data once it’s pulled together.

A: Indeed. We have an advantage. We have a very large installed base.

Q: How big is the installed base?

A: Many, many thousands of production clusters. The thing about open source software is you never really know the answer to that question. But “considerable” is the answer.

In general, they deploy it in data centers that have other infrastructure. Nobody stands up Hadoop by itself. It’s usually next to a relational database and maybe in service of a document system, so all these pieces are together. We can watch what they do. We talk to them. We ask them basically: what is it you bolted onto the side of your Hadoop cluster that you needed to do to solve a problem?

We saw over and over again the following issue with the installed base: They loved MapReduce because it let them do a bunch of stuff, but they’d run all these analytic jobs and then they blast the stuff into a MySQL database or a relational database from one of the big vendors, just so they can get interactive queries out of it. Ah, we thought — what if we could put that in there? We can look at what the installed base is doing right now for those other things nailed to the side of a Hadoop cluster.

We’ll be rolling out more of these engines in the coming quarters. Again, I don’t think we’ll be the only company, but I will say we recognized, maybe a little ahead of the rest of the market, that there was nothing magical about MapReduce as the one engine, that there would be lots of choices.

Q: When you hear people talking about big data, what is it that you think they’re not getting about the concept?

A: That’s a loaded question. I’ll confess, we’ve benefitted from [the big data hype], but I think we need to be a little bit careful about it too. Big data is a loaded term right now. It is the solution to all problems. If you believe what you read, it’s why Obama won the election, because he had better big data chops than anybody else.

Q: Do you think big data played a part?

A: Look, I think good analytics over large volumes of data really did change the kinds of decisions that certain groups made and that translated into an advantage. But let’s be honest, big data won’t make you any taller, it won’t make you any younger, it won’t make you any prettier. It’s a hugely valuable asset if approached in the right way, if used to solve a real problem. But Hadoop isn’t some pixie dust that’s going to drive your profits up just because it’s Hadoop.

Big data has a few properties. You’ve probably heard “volume, variety, velocity.” You can have a lot of data, you can have a lot of different kinds of data, you can have data showing up real fast. Any one of those can be trouble, right? If you’ve got two or more of those problems — a huge amount of highly variable data or wicked amounts of data showing up at a furious clip — then Hadoop is really the only choice on the market. It’s the only thing that was designed to scale in all three of those ways.

It’s a misconception to think that if I don’t have many, many terabytes, then Hadoop’s not useful to me. [If you have] modest amounts of data that are highly variable and you might want to combine them into a single analysis, well, MapReduce sings at that stuff. The key is to understand the question you want to answer, the problem you want to solve with the data, and then to be sure that you’ve got the data that does that.

One example I really like is a customer of ours named Explorys Medical. Imagine you get sick and go to the doctor. You see the doc, he looks down your throat, he takes your pulse, he records a bunch of stats about you, maybe orders a prescription. You go home, a week goes by and you’re not feeling any better. You call him up. Okay, we’re going to bring you back, send you to the lab, get some more tests done, maybe some imaging, maybe some blood, change your prescription a little bit. Ah, still not better? Let’s go see a specialist.

You know, if you’re ill and it takes a while to get better, the trajectory of your treatment can span a few weeks and can have lots of different data types in it. All by itself, if you capture all that information, that would be a good picture of your treatment — how it progressed, how you finally got better. Imagine, though, that it wasn’t just you. Imagine if you could do that for all the patients in a hospital. Now you can start to reason. Dr. Jones seems to have a pretty good outcome with patients with this condition, or this drug seems to be pretty effective with men, not women.

Explorys gangs together that information, not for a hospital but for thousands of hospital clients. Their goal is to recommend effective treatments to doctors that allow hospitals to deliver that care cost-effectively. No matter how you feel about the election that we just had, I think we’ll all agree that healthcare costs are out of control. Using big data [is the way] to reason about that stuff. Think about it, blood test results and images and doctor prescriptions and blood pressure, building a model based on all that, the machine learning required can only happen in Hadoop.

Q: Do you think people have unrealistic expectations of Hadoop? Isn’t it more like a platform on which you write applications to get the results you want? Out of a raw Hadoop installation, there’s not a lot you can do without people with specialized knowledge and the ability to develop applications.

A: What you say was absolutely true three years ago and it was pretty true one year ago. I’m going to tell you that things are much better today. I won’t say that they are all the way better yet, but you can go to MicroStrategy or Informatica or SAS — big analytic tool providers — and those folks have now integrated with Cloudera or with Hadoop generally. Business users and analysts can begin to get tooling that doesn’t require them to write MapReduce jobs or write Impala queries.

The availability of skills in the market is a gate on adoption of these platforms. Let’s be honest, the way we’re going to solve this is with software. We’re going to see better tools come into existence. In addition to those big established vendors, there are new vendors coming out now, companies like Platfora and Continuity and WibiData. These companies are building tools purpose-designed for Hadoop that know there are 500 computers under the hood that know how to do machine learning and can visualize complex data in new ways.

That’s when I think it’s going to get really interesting, [as] those tools, those apps become available. It’s still true that many of our customers do roll out custom analytics because the specific data they’ve got, the specific algorithm they want to use, isn’t yet available off the shelf. That just gets better over time.

We’ve added Impala to the mix and, yeah, we think that makes a lot more workloads available and accessible in Hadoop. But let’s be honest, Impala isn’t a high-performance OLAP or OLTP database. It’s designed for a certain set of query workloads, a certain set of data transformation operations. We think it’s great at those, but no one should misunderstand what it is we’re delivering. You wouldn’t bring in Hadoop plus Impala and get rid of your big Teradata or Oracle instance. There are mission-critical apps and jobs running on that big iron that belong on that big iron. Believe me, I spent a career building them. I understand why they’re good. The stuff we’re doing does allow us to help customers optimize some of their workloads, but it’s not an out-and-out replacement.

Q: What do you see across your customer base? Are they tackling old problems in better, newer ways or are they tackling new problems that they wouldn’t have even tried with the other sets of technologies?

A: I’ll be a weasely guy and say yes.

Q: I had a feeling you might go in that direction…

A: We see both use cases. I’ll say that there’s a certain class of innovative, transformative user at enterprises who say: You know what? I want to do behavioral analytics in a way that we could never do before, and I want to tie that to our marketing efforts to make better recommendations for customers. We’ve never done that before, but this platform seems a perfect build for it and I read about how Facebook is doing that and so we want to do some of that. Those folks have brought us in and they’ve built those systems and they’ve got their recommendation engines up and running and they work really well.

The number of people with that vision and the right command of both technical chops and business sense is a modest population. Until that’s shrink-wrapped and purchasable more easily, it’s going to be challenging to sell that vision, because some implementation execution is required.

There’s this other class of problems that folks have been struggling with for a long time. In financial services, noticing fraud in your transaction flow is a huge deal. A lot of companies make a lot of money watching transaction flows over relatively short periods of time and suggesting when fraud may be happening. If you carry a credit card, I’m sure you’ve had it happen that you just happened to be down at Walmart buying a TV, and that’s not a thing you do every day, and they declined your credit card because they made a bad decision. It happens, right? There are human escalations for that. But that’s a problem that has existed for a long time.

What our customers are able to do now is go beyond watching short-duration transaction flows. Rather than the last week’s or month’s or even quarter’s worth of data, why don’t we store the last decade’s worth of transactions? Let’s look for fraudulent patterns using machine learning that span not hours or days, but a year. Can we find patterns that suggest money laundering over very long periods of time? The bad guys often know how the good guys are detecting bad behavior and design their attacks to get around that. More data gives you more ways to search for and see those patterns. Actually there are patterns that are visible in a lot of data that are just invisible in a small amount of data.

Q: Can you talk about how you commercialize the technology and bring it to market?

A: Doug Cutting, who created the Hadoop project, works here. He’s chief architect at Cloudera. We bundle a collection of open source projects into one big distribution that anybody can install. One hundred percent open-source: freely available, don’t give us your name, download it, use it for free.

We believe deeply in the value of that open source channel — not for political reasons, not for religious reasons. CIOs have learned over time that open source insulates them from bad vendor behavior. Open source tends to keep infrastructure vendors more honest. When we go into an account with open source, we’re a lap and a half ahead of all of our proprietary competitors. We’ve [sold] past one big objection, which is lock-in and bad behavior.

We created the Impala project. We built it, but we released it 100 percent open source. We employ committers and contributors to all of the software that we distribute. In fact, 70 percent of the committers for the projects we bundle are Cloudera employees. About half our engineering expense goes to stuff that we give away to the open source community. We benefit a lot as well. The global Apache community is making Hadoop better. Much of it is happening off my payroll, so I get all that benefit as well.

Q: How are you monetizing?

A: We monetize in the following way. In addition to making the platform software available open source, we’ve also built a proprietary piece of software called Cloudera Manager. You gang together a thousand computers to run Hadoop. Our distribution is actually not one computer program, it’s dozens of computer programs, and those get spread out among the thousand computers in different combinations. They get configured differently. As you said, there’s user code in the mix. I want to run analytics, so I’ve got to write Java code, and when there’s a bug in that collection it’s just a mess.

How do you track it down? How do you understand provisioning new users, capacity planning, setting and enforcing security policies, and so on? Cloudera Manager abstracts everything we know about running a Hadoop cluster and renders it into software. It’s the spotlight you can shine into an operating Hadoop cluster to understand it. Why is the storage layer slow? Hey, John’s job is not finishing. What’s going on? It knows all that stuff. That’s unique software to us. No one else has it.

We bundle it with open source distribution and we deliver that combination on an annual subscription basis with 24/7 support. That’s how we make a living. We do, in addition, have a training business and a consulting business and they’re both good businesses. The subscription business is already the dominant component, the majority of our revenues. The way we think about training and the way we think about consulting is make the market smarter and get customers productive faster, but really it feeds our subscription business. When we train you, we make you more able to deploy our open source platform and, we hope, our proprietary Cloudera enterprise bundle. Likewise, if we come on site and deliver consulting services to you, we get you productive faster on Hadoop and that expands your dependence on our enterprise offering and that turns you into a long-term renewable subscription customer, which is the goal.

Q: At the beginning of that process, why would the customer choose your open source distribution over others?

A: Let me talk about why I think we get chosen. First of all, we’ve been in business for about 4 1/2 years. We’ve got an enormous installed base now. More people run our software in production than all other distributions combined.

Over that time period, over that big installed base, we’ve seen more failures than anybody else. I believe that we are better at supporting the platform for that experience. In fact, if your Hadoop cluster goes down, we help you get it back up and running. We also can go back and do root-cause analysis. We’ll look over your log files. What should we have seen three days ago that would have twigged us to the problem that something bad was going to happen and we didn’t know enough then? That stuff is now built into Cloudera Manager and we’ll alert on it. We’ll proactively tell you — hey, you might be getting low on space on node 19. We’ve seen enough of those failures that we’ve made it a part of the product.

Impala — and frankly the vision we’ve got for diversity on the platform, the new engines that we’ll be adding — set us apart. Nobody else is doing that; we are the only vendor in the space that does it. It’s driven in part, again, by our time in market. We started working on that two years ago because of demand that we saw back then.

The other point is that nobody buys our software because we make it manageable. They buy our software because it solves a business problem. There’s an application that they’re running that matters to them and they need it to be manageable. So we get the sale.

Our partner ecosystem is much, much larger than any other vendor: more than 400 companies that build on, integrate with, deliver services for, or can provide operating systems. If you want a platform that’s got the most tooling, that gives you the most choices, then we think we’re a good choice.

Q: Who do you consider your core competitors?

A: I know why you ask that question and I just hate it, because first, I don’t want to give those guys air time. Second, I don’t like quotes that get posted on locker room walls. Let me answer a different question, if I may. I’m going to be weasely, and you can even quote me as saying, “I’m going to be weasely.”

I think what’s interesting right now is what’s happening in the data management market broadly. Twenty-five years of my career was building and selling relational databases. It was boring. Tighten the screws, polish the chrome, charge 30 percent more every single year. We made a good living, but nothing ever happened.

Out of Google comes this new platform. We had no idea. We were taking care of our great big enterprise relational customers with a very demanding workload. We had an innovator’s dilemma problem. We were concentrating on the high end of the market and not what was happening anywhere else. So we got totally smoked by the introduction of this platform.

I’ll go further. Like most of the rest of the relational industry, I read the Google paper in 2004. I thought it was a joke. No query language, no transactions. I mean, we knew how to build data management. This was nothing like that. The scale of problem, the scale of data available today is so quantitatively larger that it’s become qualitatively different. You’re struggling not to store terabytes but petabytes. Facebook’s latest publicly announced single cluster manages 105 petabytes of data on spinning disk — just incalculable amounts of information.

When you’ve got that much data, you start to think about it differently. You can ask different questions, you can attack new problems. I think what’s really exciting is the opportunity for data capture and data analysis that is suddenly here because of the combination of data availability, all this machine-generated data.

Also, consider the compute infrastructure: blade-based servers that are rack mountable that you can buy in bulk to attack these problems. That combination lets us do stuff that just wasn’t possible a few years ago. We will answer important questions about disease, about feeding the world, producing energy. Yeah, by the way, about fraud and money laundering and transaction flows and about recommendation engines for consumers.

We’re going to be able to attack problems that have never been here before. I wouldn’t say that I’m competing with any established vendor for that. It’s a brand new opportunity that somebody is going to dominate, and we want to be that company. It’s the very early days, but we’d argue that we’re number one. But to become that company, survive long term, remain independent, deliver the breadth of service…a lot of flawless execution needs to happen.

Q: Do you think existing database companies can be players in this market?

A: In 1988, maybe ’87, if you had asked me, the highly insightful 25 year old that I was at the time, I would have told you IBM is the walking dead. I mean they’ve got the stink of a corpse on them. They gave away the PC market; their mainframe market is doomed. What I learned from that is that one should never discount the value of an enormous installed base and billions of dollars in cash.

Big vendors with assets like that will survive. Will they be the dominant players in subsequent years? Will they drive the standard? Will they drive the market? I don’t know. Often a disruptive new entrant gets to do that. Think about what Oracle did in the database space. It wasn’t even first, right? But it executed extraordinarily well.

Q: Who are some of your key partners and what are they doing for you?

A: I talked a little bit about some of the ISV ecosystem partners: Informatica, MicroStrategy, QlikTech (just announced), Tableau, Talant. Anyone making tools or applications that unlock this big data value for business users is huge for us because we don’t do any of that. We just make it work for the data centers.

Q: You don’t intend to go to the application level?

A: I had the advantage of watching how Oracle, Microsoft, and IBM became dominant in the database space. They stayed relentlessly focused on the platform layer and left the ecosystem to create the absent tools on top. That gets a lot of other vendors innovating and creating diverse offerings and making money at it. Every one of those sales pulls the platform in.

It’s a long, long time before this platform is so ubiquitous that my growth is constrained and I have to move up the stack. Vendors like that are good partners for us. We’ve got a great consulting business, but my goal is to be a software company. I don’t want to build a huge cadre of systems integration guys that I bill out by the hour. We’ve got great relationships with systems integrators — Capgemini is a good example — that help drive the product into deployment.

The other category of vendors that we work with are the hardware folks: Dell, Hewlett-Packard, SGI, Network Appliance, and, this one’s a little surprising, but in their engineered systems division, Oracle. All five of those companies build and sell hardware to enterprises and resell Cloudera software for us. If you go to Oracle and buy their big data appliance, it’s running Cloudera enterprise.

Hadoop is tricky to sell. It’s still true that the vast majority of our revenue is our direct field, getting out and talking to customers. The last thing I’d say, and you didn’t directly ask, but our sales model looks a whole lot more like IBM’s or Oracle’s than it looks like MySQL or JBoss. This isn’t downloaded over the Internet and bought with a credit card. The expertise required to deploy a big distributed processing system on hundreds of nodes demands a lot more support from us, and so we’ve got a very high touch, a very typical enterprise software sales team.

Q: Who buys it within that IT organization? Is it the same person who was buying the relational database?

A: Yeah, actually, it is. The real trick to that answer is there’s no one person who buys a relational database. There are people in the data center, IT staff, who are responsible for choosing infrastructure to run workloads. Those folks typically choose on the basis of tech specs and SLAs and vendor expertise. We win much of that business.

We sell into the data center. We’ll go all the way up to the CIO or the director or VP-level people in the data center. Likewise, every now and then we go find one of those mavericks. A line-of-business owner or senior marketing exec has a vision for solving a business problem that the enterprise never tackled, and that guy makes a different decision. Often, we have to find both those parties. We’ve got to get the business guy excited about the potential of the platform and then the technical staff in the data center enthusiastic about our support for that.

Q: That seems to be the hurdle on a lot of big data stuff — there really needs to be a business champion. But the business champion doesn’t understand the ways to make that happen. You either have to make the tools so easy they can go off and do some of it themselves, or you have to get these guys involved on the IT side in a more proactive way. So how do you go about bridging those two worlds?

A: It’s an excellent observation. The answer is yeah, we ain’t done building that bridge yet. Here’s how it has worked so far and why we’re optimistic. First, you get a good sales force that knows how to sell infrastructure software. Folks who sell data management for a living can sell this new kind of data management. Really, the ability to tell use-case-based stories relevant to the industry is enormous. If I can say — oh, yeah, banker, we know from the other work we do on Wall Street about this equities pricing or risk analysis problem. Or tell credit card processors about this fraud or money laundering application. We built that; we know how it works.

Q: What are the other gating factors? What keeps more people from going in this direction?

A: Skill set and accurate information about what the platform can do. A lot of people have heard about big data. A lot of people have heard about Hadoop, but they don’t really have a clue as to what problems they should attack. That kind of evangelism, helping them understand that, is a big deal.

It had been the case that the maturity of the platform really impeded its adoption. Reliability and availability are largely fixed, but not all the way. We at Cloudera, the community broadly, still have some pathway in front of us on that stuff, but it’s a lot better than it was a while ago.

Q: The incumbent business intelligence vendors say that what Hadoop is really good for is exploration: Just dump a bunch of data in there, decide what the patterns are that you want to pursue, and then organize it and port it into a conventional business intelligence tool. What’s your reaction to that?

A: It absolutely has been true. I would argue even today that yeah, you could make that case. In fact many of the BI tools are used as the exploratory dashboards now looking through Cloudera Impala to explore your data.

To tell that story in a slightly different way, Hadoop MapReduce has been awesome at using machine learning to build models of user behavior, but when you want to recommend apps to people in real time, you extract those models that you’ve built and you roll them out in different infrastructures. You have a special-purpose, every-single-day tool, and you use Hadoop for back-end, large, ruminative workloads.

That’s largely an artifact of MapReduce. As I told you earlier, MapReduce was the first engine and Impala is the second. What’s really going to be interesting is what are three through n, right? There is no reason to believe that the shortcomings of Hadoop today are endemic to Hadoop. The platform that we sell today is going to be as much more capable, and much more general, than it is today five years from now.

This article, “Cloudera CEO: We’re taking Hadoop beyond MapReduce,” was originally published at InfoWorld.com. Follow the latest developments in business technology news and get a digest of the key stories each day in the InfoWorld Daily newsletter. For the latest business technology news, follow InfoWorld on Twitter.