Microsoft CIO: We’re dog-fooding the cloud

I first spoke with Tony Scott in 2003 when he was CTO of General Motors. At the time, the obsession of the day was Web services, which Scott wryly called “an excuse to get people to talk together” about business processes, a role many grand IT initiatives fill.

Today Scott is CIO of Microsoft, a position he’s held since Feb. 2008. When I interviewed him just before the holiday, the main excuse for us to talk was the tech industry’s current obsession, cloud computing — and how Microsoft is leveraging its own vast cloud computing infrastructure to serve its employees. We also touched on the consumerization of IT and how he is supporting a glut of new mobile devices.

[ InfoWorld editor in chief Eric Knorr lays out Microsoft’s advances in cloud computing: Microsoft’s big grab for the cloud | Office in the Cloud, take two | Get the no-nonsense explanations and advice you need to take real advantage of cloud computing in InfoWorld editors’ 21-page Cloud Computing Deep Dive PDF special report. | Stay up on the cloud with InfoWorld’s Cloud Computing Report newsletter. ]

Between his GM and Microsoft jobs, Scott served as CIO for the Walt Disney Company; further in the past, he was vice president of information services at Bristol-Myers Squibb. This golden resume puts Scott in a rarified group of IT leaders who are comfortable running huge global IT operations.

But being CIO of Microsoft is not the same as being CIO of any old $200 billion corporation. We began by talking about that difference.

Eric Knorr: What’s it like to be CIO of a tech company like Microsoft? I bet it’s fun have a bunch of technologists always second-guessing what you’re doing.

Tony Scott: Well, but it’s no different than at home. [laughs] Everybody is getting used to this world of having technology at their fingertips, and there’s a belief that what scales at my house should scale in the corporate world. I find it fun and endlessly challenging in terms of how we’re going to solve some of these big problems where people want access to everything all the time, in a very convenient way on whatever device they happen to be on or in or around.

Knorr: Just to be clear about your purview, you have nothing to do with the big new cloud infrastructure data centers? You’re concerned with supporting 4,000 Microsoft users.

Scott: Maybe I should give a little context here. One of the roles that Microsoft IT plays, and has played for a long time, is to dog-food all of the products that are destined for the enterprise. In the past, that would have meant that when we did a new release of an operating system or a new Office release or whatever, we would start in the very early phases of the development cycle, deploying in very small quantities — and then over the course of the development cycle, deploying internally at greater and greater scale.

In the example of Windows 7, by the time it reached its beta phase, we had virtually every employee in Microsoft all around the world on the beta release. Virtually every product that we sell into the enterprise follows that pattern. The world of services, whether it’s the business productivity stuff or Azure or whatever, we treat no differently. I have many large projects underway now that are on our cloud infrastructure and will be delivered over the next couple of years.

Knorr: So how do you reconcile dog-fooding beta software with supporting your users? You have to test products for the business, but you also have to keep your users happy and make sure everything works.

Scott: Sure. As you would imagine, we’ve developed a rather robust process for this, so we and the [product] groups negotiate what we call “ship criteria,” and this can range from performance indicators, quality indicators, all kinds of different things. We’ve done the same thing with our cloud services. A lot of people don’t know, but I run all of Microsoft’s mail on the exact same servers that we sell customers.

I am another customer just like any of our other big customers would be. The only difference between me and the other customers is I’m always running the beta version or the next set of things that they’re going to release into the service. And we kind of educate and socialize the fact with our employees that this is an important part of your role as an employee here at Microsoft: to help us find bugs and correct things that would not deliver a suitable customer experience. In the tradition that we’ve had with other products, we do the same thing with the cloud.

Knorr: I see.

Scott: That gets everybody’s [skin in] the game in a little different way than if it’s somebody else’s job.

Knorr: So you’re a consumer of the cloud services that are launching. Are you also using Office 365 now?

Scott: We are on a version of this that will become Office 365. We’re dog-fooding that as we speak. But let me be clear — I don’t have the whole company on it yet. It works the way I just explained it: We take a small group and then we go big as the product goes further and further along in its development cycle.

Knorr: So is the end goal then to have everybody up and using, say, the enterprise version with Office Professional Plus by some certain date? Do you have a schedule for that?

Scott: We do and I don’t remember it off the top of my head. Now, what we’re going to do in the case of Office 365 is we think there’s a whole bunch of hybrid scenarios where companies are going to want hosted Exchange for some [of the] company perhaps or use the Office 365 service for some other parts of the company. We also support mixed modes like that where we’ll include the next version of the server product. I said, “Thanks a lot guys, my world just got doubly hard,” when we started offering services. But those are real customer scenarios, as it turns out, and something that we should do.

Knorr: Are you doing something similar with Azure?

Scott: Yep, absolutely. One of the biggest enterprise, Tier-1, can’t-break apps in Microsoft is the set of applications that supports our licensing portfolio. I made the commitment over a year ago to move that whole platform to Azure, so we’re right in the middle of doing that work right now.

Knorr: As a CIO who also has a bottom line to meet, you have to worry about costs and efficiency. Are you actually seeing some measurable efficiency in terms of your own costs with cloud deployments? Are you seeing some economies of scale there?

Scott: Yeah. It’s come about in two ways. One is that, for all of the critical production environments that we have, there’s a bunch of upstream stuff we call the development environment and the test environment and the pre-production environment. And historically we’ve tended to size those environments to mirror what we expect to need in production because that’s the best way to test these sorts of big hairy applications.

But the reality is that much of that upstream stuff just sits there idling and underutilized much of the time, particularly when you’re not in a heavy development cycle. Sometimes it gets reused or repurposed, and certainly virtualization helped us get some more efficiency. But with the cloud, we’re seeing that we can take that to the next level. That has been a big part of the economic story, even if we don’t, at the end of the day, always get huge benefits in the production environment.

Knorr: I would assume it’s all Hyper-V virtualization internally?

Scott: Yeah. We have our feet in all three buckets, so we have some very traditional nonvirtualized environments, we have the set of apps that have been heavily virtualized, and then the next step in the journey is to get them to the cloud. But there are limits to what you can do in the virtualized world. Here’s the way I think about it: I call it the “end of the double-double rule.”

The double-double rule is sort of the unwritten rule that we all have used for years when sizing the hardware for a traditional three-tier sort of environment. That is — you figure out what you’re going to need, and then you double it, and then you double it again. You do that because you want to provide for growth. There are peak periods, and you might guess wrong and all that sort of stuff. And the strict evidence that the double-double rule is followed is you can go into any of these traditional environments and you see utilization around 5 to 15 percent max.

We deliberately did it and we keep doing it for those environments because making a mistake is costly if you underprovision. Now when we got to virtualization, we said, “Oh, we have some flexibility here and we have some ability to stretch, so maybe I don’t do double-double, but I’m still going to do double and then a little.” You look at the average utilization in even the best virtualized environments and you still don’t see high utilization numbers, but it’s better than it was. Maybe you see 30, 40, but that’s kind of where people start to feel less comfortable. I’ll call that the “double-and-a-little-something rule.”

What we’re seeing in the cloud, because it’s a lot easier to get extra capacity or expand when you need it almost on the fly, now it’s — I don’t need double-double, I don’t need double-and-a-little-something, I can just figure it out and I can tune and I can play with things and consume just what I need.

Knorr: And why is that? Concretely, why is it so much easier to scale? I mean you still need to have the infrastructure, right?

Scott: You do, but the cloud gives you a different place, a bigger canvas on which you can put the scene. Our developers can decide how much they need and they don’t worry about how many physical machines that is or how many VMs it is or how many whatever. That’s already provisioned automatically for them in the background. If they guessed wrong, they can get a little more. If they guessed too high, they can get some and take it away and only pay for what they need.

Knorr: All this is assuming a large pool of shared infrastructure available on which to deploy that.

Scott: Yeah, but that’s the beauty of the cloud, right?

Knorr: You know, a lot of people when they talk about the private cloud really mean virtualization.

Scott: But it’s beyond virtualization. And I’m not saying “private cloud” here. If you’re talking about private cloud limits, yes, there will be some more limits. I’m talking about Windows Azure and SQL Azure now specifically, where those limits are far bigger than what you would see in a private cloud scenario.

I’ll just give you an example of how it manifests itself here. October is a big month for charitable giving at Microsoft, and we developed years ago this application that’s kind of an eBay for charitable giving purposes. You could donate something and then Microsoft employees could come in and bid on it, and at the end of the auction period, the highest bidder wins and all the proceeds get donated to a charitable cause — a very popular thing, as I got here three years ago, and all built by volunteers and all that sort of stuff.

But because the auction ended at a specific time at the end of October, people did the normal thing and they waited until the last minute. Then all of a sudden, there’s this bidding frenzy that goes on, so the individuals could win the thing that they were interested in in the auction — same thing eBay sees when auction items expire. But we had no way, really, of gauging how much activity there was going to be or how intense it was going to be, and pretty much for several years we stretched the limits of capacity in terms of this particular tool. The experience at the end of the auction became rather unsatisfying, let’s just say.

So two years ago, when Azure was very young and SQL Azure was very young, we said: Let’s put this on the cloud. Then we did it again this last year and both years in a row for the whole month of October. On an average hour, on an average day, let’s say we needed x capacity, and that’s what we provisioned for. But we knew from historical patterns that at the end of the month we were going to need a lot more.

Two days before the month end, we said — all right, let’s goose this baby up and let’s give it 5x. And then in the last couple of hours, we said — all right, now we want it 10x. And we’d do that with a click of a mouse essentially. And sure enough, the auction tool ran like nobody’s business. Nobody knew how easy it was for us to just increase capacity on the fly like that.

Knorr: So internally, have you implemented chargeback?

Scott: Yeah. I get billed for this just like any other customer would.

Knorr: And also self-service? I mean developers can go and provision their own…

Scott: Yeah. We have internal capability for that and we’ve got hundreds of projects going on on that internal capability.

Knorr: We’re coming to the end of our interview, and I don’t want to let you go before I ask you about supporting mobile users, since that’s a big challenge right now for CIOs. Keeping track of these devices, and particularly the endpoint security for these devices, is a big concern. How are you dealing with that?

Scott: Well, I think it is a big concern for CIOs. It’s a part of a bigger fabric of the consumerization of IT, where you’re getting more and more devices that want to connect and consume information and be used by people in a whole bunch of different ways. Here at Microsoft, we use our own tools, our own infrastructure to manage that. Probably the best way you could think of it is — we have devices that have the full capabilities for encryption and to manage certificates, and we can make sure that the storage is encrypted, and we can enforce strong passwords and those kinds of things.

If you’re using one of those devices, you can have the full complement of capabilities that one would expect to have. You can read rights-managed email, you can see content that we restrict permissions on and restrict access to and all those kinds of things. On the other end of the spectrum, if you have a strictly consumer device that has none of those capabilities, you’re going to be a lot more restricted in terms of the kinds of things that you can do, but you’re still going to have browser access and Internet access and those kinds of things.

Based on the device and based on its capabilities, we have a series of gates that you go through that then determine what you’re going to be able to do on that phone. It’s actually something we’ve had to do for a long time.

One of the things that people don’t appreciate is that unlike a lot of corporations where the CIO says, “You can buy this PC or that PC or you can have one of these two phones,” Microsoft is in the business of supporting a pretty broad ecosystem of devices. For years we’ve had PCs from every OEM on the planet, and some of them I would say even came from other planets when I look at them. But they’re the full range of both corporate devices and consumer devices, so we’ve had to develop the infrastructure to manage and support those and develop security policies commensurate with the capability of the hardware.

Knorr: So you’re supporting iPhone and Android and Windows Phone 7 and everything under the sun, but you’re not letting them all connect to email though, are you?

Scott: Again, it depends on the capabilities of the device.

Knorr: If it connects the Exchange server, does it have to have encryption?

Scott: It depends. You can connect through Outlook Anywhere and read email and depending on the device and what its capabilities are, sometimes you can read rights-managed mail and sometimes you can’t. It’s all dependent on the device and its capabilities. But certainly for storage, we want to see encrypted storage before we want people storing sensitive company information on that device.

Knorr: Right. Are you supporting Windows Phone 7 now?

Scott: Oh yeah. Yeah. We gave Windows Phone 7 to every employee. Actually, no — I’m sorry: We pay for it, but they procure their own phone. There’s a high take-up rate on that.

Knorr: They have their choice of phone?

Scott: Oh yeah.

Knorr: OK. Lastly, since we are wrapping up here, what’s your one biggest challenge of all the stuff we’ve talked about?

Scott: Well, I think the big challenge that every CIO is facing today, at least all the ones I talk to, is that businesses are digitizing at a very rapid rate, and this digitization means things are getting much faster, much closer to real time.

We’re all in the business of moving information in bits, not in the business of moving atoms, to a greater and greater extent. This is putting new pressure on companies to be faster, more flexible, and more responsive in the marketplace, and to accelerate the pace of development — to accelerate the pace of virtually everything we do.

There’s just a ton of work in every company that I know of to get to these digitization platforms, so it’s a pretty exciting time and I think there’s a lot Microsoft and IT can do to further that along. That’s probably the big ongoing challenge that I see.

Knorr: Great. Well thank you very much, Tony, for your time. I really do appreciate it.

This article, “Microsoft CIO: We’re dog-fooding the cloud,” originally appeared at InfoWorld.com. Read more of Eric Knorr’s Modernizing IT blog and get a digest of the key stories each day in the InfoWorld Daily newsletter and on your mobile device at infoworldmobile.com.