Paul Maritz: Pivotal is a bridge to the future

The first thing I had to ask Paul Maritz, CEO of the VMware spin-off Pivotal, was whether he had received any interesting calls from the Microsoft board on the heels of Steve Ballmer’s departure. Not surprisingly, Maritz said he would go “nowhere near” that question. But here’s why he is considered a potential successor: During his14-year stint at Microsoft, Maritz oversaw the development and marketing of some of the company’s most successful products, including Windows 95, Office, and SQL Server, and later presided over four years of impressive growth as CEO of VMware.

Maritz comes under the category of “visionary” CEO, a classification few would associate with Ballmer. But vision is a tricky thing. With Pivotal, Maritz is building a PaaS (platform as a service) for developing the next generation of applications — infused with big data and the onslaught of events that will flow from the Internet of things. The release date for Pivotal One, which will combine Cloud Foundry and Hadoop among other technologies, is set for the end of 2013. Is Pivotal too far ahead of its time? Or is the moment just right for building a platform to underlie massively interconnected, highly adaptable Internet applications?

Another question that has dogged Pivotal pertains to the mish-mash of companies from which it was formed, with 1,300 employees who once worked at various EMC and VMware acquisitions — Cetas, Cloud Foundry, GemFire, GreenPlum, and SpringSource — not to mention Pivotal Labs, a high-end Web and mobile development firm. “Startups” pieced together like this don’t exactly have a stellar track record. Plus, GE has its hand in, with a $105 million investment and high expectations for Internet-of-things applications that today exist mainly in the imagination.

To find unity in this diversity, Maritz told me that “You have to create a sense of mission and a sense of identity. It’s a great group and people are starting to really feel that they’re speaking to the future. We’re not playing defense here. We really are speaking to important things that will have real meaning. And that’s always the best tonic.”

For what it’s worth, when I visited Pivotal’s offices in San Francisco to do this interview, it wasn’t the usual earbud-quiet, heads-down environment. The Cloud Foundry group in particular was buzzing with conversation — maybe energized by IBM’s recent endorsement. A couple of days before, the company had announced that the Pivotal CF platform would run on the vCloud Hybrid Service, although Maritz  is quick to note that Pivotal will run across the whole gamut of IaaS clouds (it already runs on Amazon Web Services).

I began the interview with Maritz by revisiting the launch of Pivotal in April:   

Eric Knorr: Two years ago you talked to me about looking toward the next generation of Internet-centric applications. When you launched Pivotal, I figured, oh, that’s what he meant.

Paul Maritz: That’s basically correct.

Knorr: How long did you have Pivotal in mind as a spin-out? When did you first conceive of how it would come together and that Cloud Foundry would be central to it?

Maritz: I stepped down formally out of VMware at this time last year. I had been thinking about it for the better part of a year before that. There were several points of gestation. The principal one was thinking and saying, “Look, it’s becoming clear that infrastructure-level clouds, a la VMware, a la OpenStack, a la whatever else, are going to become the computers of the future, both on-premise and off-premise.”

The question is what does that enable that you couldn’t do before, because typically, in our industry, you get fundamental change when something in the hardware changes. What really enabled the client server generation?  It was CPU cycles, by historical standards, becoming free. All of a sudden, which you could never have done cost-effectively in a mainframe, you could throw all these CPU cycles at a graphical user interface or, equally importantly, a relational database. All of this quantity suddenly becoming plentiful and free enabled you to think about the different set of applications that could be built.

The thesis is that with infrastructure clouds becoming the new “hardware” (I’m using the word “hardware” incredibly loosely here) there are going to be some things that become plentiful and free by historical standards. What’s the impact going to be of that on how we build applications?

The big Internet pioneers — the Googles and the Facebooks — they do use the new hardware. The opportunity we saw is to help enterprises build a new generation of value and experiences and applications that use the new hardware, and came to the conclusion that that was a real enough and important enough opportunity that warranted repeating the VMware play.

Knorr: How do you see Pivotal as a VMware-type play?

Maritz: In the sense that when EMC acquired VMware, they looked at it and said, “This is a big enough and transformative enough opportunity that we’re not going to integrate it back into EMC. We’re going to keep it separate for a variety of reasons to give it focus, to allow it to work with the EMC competitors, and above all to give the people who are in VMware a sense of ownership and connection to their own destinies.” It was … incredibly successful, and it gave the respective board of directors enough belief to say it’s worth doing that again. The downside of the deal is when you expose it to the cold light of day at a very early stage — you get that onus put on you. 

Knorr: I met with John Roese, CTO of EMC the other day, and he said that Pivotal was two years ahead of its time. Do you see it that way?

Maritz: It depends on which point in the market are you trying to measure. We get more and more convinced that we’re pointed in the right direction and the early adopters are moving with us now. Until you get the broad bulk of the market, like every market, it takes a while. If you look at virtualization, the first VMware Conference was 10 years ago. It took eight years for them to cross the 50 percent threshold. So these things happen over a period of time, but the key thing is to be aligned with the tides of history. And this is one where we’re convinced that the tide is definitely coming in.

Knorr: Is there a way to characterize how applications built on your platform are going to be different?

Maritz: Well, let’s take that from a couple of angles. One is that by putting your application on Cloud Foundry, you’re getting two things: You’re getting cloud independence and automation, which is true of both existing and future applications. The future applications are really characterized by using new data fabrics. If PC servers and minicomputers made CPU cycles free versus the mainframe, what is the cloud making free versus PC servers and Unix minicomputers and the rest of it? The things that they’re making free are basically the ability for a developer to work with a large number of machines. For a developer today, it’s no longer an avant-garde thing to say I need 10 machines, 50 machines, 100 machines, to throw at the problem.

So you get a lot of machines that a single app can dedicate to its problem. And certainly storage is becoming very cheap, so you can say you want to store this big blob of bits and store it for all time. That’s no longer an extreme thing to do.

Those two things coming together means that the data fabric — the database — gets remade. Hadoop is an early example of the remaking of the database. Think of what Hadoop is. When it was pioneered first at Google, they in some sense deconstructed the traditional relational database and they took the persistence, the actual storage of the information and put it into a scale-out object store, which in Hadoop’s case is HDFS. And they took the processing of the information and they moved that into collection of machines working in parallel on top of HDFS. So they were taking advantage of these two quantities that became “free” — a large number of machines working with the information stored in a cheap scale-out object store.

We think that paradigm gets reapplied to all of the important ways we’re working with data, because it allows you to work with much bigger data sets, much more quickly, much more cost-effectively. We think that’s going to happen to relational query, to transaction processing, to high-speed event ingestion, etc. The whole database world is going to get remade to take advantage of large numbers of machines working in parallel with big bodies of data stored in an HDFS or similar object stores.

By taking advantage of those capabilities, you can build applications that reason over much bigger datasets, much more diverse datasets, and getting more value out of them, doing it more quickly, doing it more cost-effectively, and then being able to actually use that to drive some interaction with the user. So it’s not just about analytics, where you kind of get some insight into the data. It’s about how you use that in the context of some application that’s going to drive a transaction or cause some interaction with the user.

So that’s why we talk about applications and data. We’re not just in the big data business. We’re in the applications and data business, using these new resources that the cloud is going to manufacture for us to allow you to do things you simply couldn’t cost-effectively do on a traditional, cloud server relational database architecture.

Knorr: In that whole description I don’t think you mentioned the Internet of things once, which was a key point in your launch and announcement of the GE partnership. You could argue that big data won’t be really big until the Internet of things ramps up.

Maritz: The Internet of things dramatically exacerbates it. As you heard me mention, one of the important data modalities is going to be this high-speed event ingestion.

The Internet of things is going to mean there will be many more events flowing in much greater quantities. In a lot of cases, it won’t just be about ingesting them and analyzing them, but about reacting to them in real time. This is not about having a data scientist look it through. We’re going to do something now.

Knorr: But Hadoop is batch processing, not real-time.

Maritz: That’s why Hadoop is the beginning, not the end. Hadoop today is HDFS plus MapReduce. In the future it’s going to be HDFS with MapReduce plus relational query, plus transactions, plus complex event processing. All of these additional ways of working with data are going to be added on top of the HDFS substrate, and they’ll all be pulling information out of and pushing information into this big “data lake” at the bottom, which is a phrase they’ve started to hear more and more customers talk about. The data lake is an important notion, because if there’s one commonsense thing about getting value out of data, it’s that the more Balkanized your data is, the harder it is to get value out of it.

Knorr: Yeah. Would you be open to also looking into other types of NoSQL data stores like Cassandra or whatever?

Maritz: Yes, absolutely. But what you don’t want to have to do is say, “Every time there’s a particular view on the data, I have to create a new repository for that, and then Balkanize my data.” So we think that this notion of using HDFS as the common substrate on which these data modalities get rebuilt is the direction of the future. So when we put Pivotal together, we very explicitly looked inside the EMC and VMware family and said, “Who knows how to work with a large number of machines over an underlying persistence store?”

It turns out we had two people who had been living and watching that movie for some time, which is the Greenplum team, because what they did is essentially pull the query processor out of Postgres … and put back into Postgres a query processor that is parallelized, that knows how to do query execution over a large number of machines working in parallel. They just happened to do it on top of a Postgres substrate. That’s what the Greenplum database is.

So we realized that they’d come to this realization themselves, and said, “Hey, if we take that query processor out, we can, instead of applying it to Postgres, we can apply it to HDFS.” That’s a hard thing to do, by the way. They spent ten years working on parallel query, which is a notoriously hard problem to work on, but they’ve gone a long way down that road.

So all of a sudden that technology was available to be re-manifest on top of HDFS, which is what Pivotal HD is. The other team that had seen elements of this movie before was GemFire, which works in very high-end event processing and transaction processing. Again, they had problems where the event rate in transactions was too big to handle in a traditional big-iron clustered approach.

Knorr: Huge memory type stuff.

Maritz: Yes. The key thing is a lot of memory space; it’s in memory but scale-out. We’re talking about hundreds of machines that you can throw at the problem. We’re taking GemFire’s expertise at how to handle large numbers of transactions in memory, scale out in memory, lots of machines, taking advantage of what that free, cheap quantity that the cloud gives you and applying that to an HDFS substrate. So we’re starting to build out that suite of data modalities on top of HDFS and saying that you can take HDFS, do MapReduce, do relational query, do transaction processing, do high-speed event ingest as a suite of data capabilities on top of a number of common underlying substrate.

Knorr: It sounds to me like developing the complex event processing, as you were saying in the context of the Internet of things, is absolutely critical.

Maritz: Yes. And that was clearly one of the reasons why General Electric got interested in us, because they’re looking to build a new generation of applications where a lot of them will be having to deal with the Internet of things.

Knorr: Internet of things applications would seem to be a long way off. Today aren’t you leading with professional services? One of the key points some people have missed is that Pivotal’s name comes from Pivotal Labs, a high-end Web development firm EMC acquired 18 months ago. I would imagine, certainly in terms of the revenue for the company in the near term, it’s largely going to come from Pivotal Labs?

Maritz: We don’t see it as an either/or thing. Having Pivotal Labs as a next-generation services capability is an incredible asset for us. They are speaking to a great need in customers to do things differently.

A lot of customers are saying, “Look, we’re tired of engagements that last for years and result in something we’re not sure we wanted in the first place.” The Pivotal Labs model has been very intense, almost joint-venture engagements with the customers. That’s something we find extraordinarily interesting, and not only because people like the fact that they can get something done quickly.

In a lot of cases customers are having to rediscover software development. They’re saying, “Look, all of a sudden the nature of my competition is changing. I need to get back into using software as a differentiator and I want to discover how software development is done today, not as it was done 20 years ago when I was a young programmer.” And so they are very interested in coming to work with Pivotal because they think it gives an insight into how modern agile software development is done.

We deliberately chose the Pivotal name to give to the whole company because we think Pivotal Labs embodies a lot of the forward-looking, more agile way of doing things that we’re trying to actually put back into our own behavior.

Knorr: And obviously, with Cloud Foundry, you’re targeting developers.

Maritz: Yes, enterprise IT and developers.

Knorr: And providing, maybe, a bridge for enterprise IT, which seems somewhat under siege right now?

Maritz: Correct. Enterprise IT is being asked to do something very difficult right now: To become fundamentally more efficient in their existing IT operations and innovate and give capabilities to the enterprise on the other hand.

In one case they’ve been asked to create savings from their existing operations in order to fund innovation going forward. And it’s always a chicken and the egg thing. It’s hard to do both at the same time. So you try and get some savings first and then reinvest that in the innovation — or you try to do some innovation to build credibility. So it’s a tough dance that enterprise IT is in today. I think there is a deep realization that they can’t just keep doing what they’re doing and expect that somehow some miracle will happen along the way.

Our view is that both perspectives are valid. Some people are going to focus for now on getting efficiencies and taking complexity out of the environment — whether by moving toward a private or public cloud or hybrid cloud infrastructure, trying to move more of our applications onto a common PaaS platform, etc. The more advanced folks are realizing that if you go down that road, at some point you have to deal with PaaS, because at the end of the day a lot of your complexity and cost is caused by applications. So just moving onto your virtualized infrastructure is good but it’s not enough.

The people who are coming at it from the innovation side are saying, “Look, the most important thing I need to do for my organization is deliver some of these new experiences, these new ways of being able to reason over data, etc.” And we’re finding ourselves speaking to both constituencies at this point in time. If you want to go down the efficiency route, you should be working to move towards a hybrid cloud infrastructure — and here’s Cloud Foundry as the next step along that road. If people are talking to an innovation perspective of the world, we’re saying, “You’ve got to start looking very carefully at the data fabrics and how you build applications, and here’s a set of things that you could do to put yourself on a different footing.” Those two things are going to coexist for a long while, and you have to be able to position to speak to both.

Knorr: When will you have an application that shows the full capabilities of the Pivotal platform? I would imagine you want to have that concurrent with the launch of Pivotal One.

Maritz: Yes. We’ll obviously have some of our own internal uses. But I’ve been building platforms for a long time and I know that this is an iterative process and that the most important thing is to be aligned with the tides of history, and then execute well. If you don’t align with the tides of history, everything is hard. And if you are aligned and you execute badly, life is still hard. You need to do both.

This article, “Paul Maritz: Pivotal is a bridge to the future,” originally appeared at InfoWorld.com. Read more of Eric Knorr’s Modernizing IT blog. And for the latest business technology news, follow InfoWorld on Twitter.