Open source and global development

In an upcoming InfoWorld feature, we explore a set of interrelated themes that Andy Singleton has identified under the rubric of “IT deflation”: a global pool of talent, a surplus of software components (often freely available), and the research and communication skills necessary to translate these resources into IT successes.

We interviewed Andy for the story, and I talked to a number of others as well. One especially useful perspective was Brian Behlendorf’s. As co-founder of both the Apache Software Foundation and then CollabNet, Brian observes the confluence of open source and outsourcing from a unique vantage point. He believes, and I agree, that we should embrace the reality that software projects are not separable from the people who work on them.

Behlendorf: “One of the key insights of open source is that there are good reasons to attach people to code. Apache isn’t just a Web server, it’s a Web server with a community around it. To treat software like Legos, without thinking about the context and the community, is a losing proposition. There was a lot of noise a couple of years ago about building corporate component libraries. But the problem is that by simply having that code there, you didn’t have the context.”

Of course it’s no accident that the Internet is the soil in which all open-source projects are rooted.

Behlendorf: “We have a couple of customers who are using SourceCast [CollabNet’s ASP-style collaborative infrastructure for software development] to coordinate worldwide development. A lot of enterprise tools have been designed for the LAN, not the WAN. The primary deployment protocol of [Rational] ClearCase is NFS, and it uses Unix permissions — it just wasn’t designed to be shared over a very wide area network. When you have a tool that’s built for wide area use, the benefits are more than just seeing the day-to-day CVS commits. Participants get to know one another as individuals.”

The hallmark of open source is transparency. Every aspect of a project — the CVS repository, the issue tracker, the e-mail correspondence — is visible to everybody. What works for global open-source projects can also work for globally distributed enterprise IT.

Behlendorf: “We started a project with Barclays Global Investors over a year ago. They have a development center in Australia and also have teams in India, California, theUnited Kingdom. The teams were very spread out; they had no global knowledge of where code actually sat. In lots of cases, it was on developers’ laptops, not centrally managed. By simply getting on the same page, by having a place to store it, they’re finding redundant efforts. And they’re finding teams working on the same kind of project who don’t know about each other and who can loosely couple.”

When projects run in transparent mode, and when Google can see them, it becomes possible to search broadly for people and for software components. If there’s something that could save you a couple of weeks, if only you knew about it, the odds that you will be able to know about it are constantly improving. As a result, some skills not classically thought to be part of the software developer’s repertoire come to the forefront.

Behlendorf: “Like the ability to communicate clearly.”

Yes, and the ability to cast a wide net and quickly evaluate what you find.

Behlendorf: “What always frustrated me, in computer science, was how we learned all the low-level things — which we have libraries for nowadays — but we didn’t learn large-scale integration. What’s the skillset to be able to jump into the codebase of something like Mozilla, read the architecture docs, and figure out the makefiles? Computer science classes don’t teach you how to dive into foreign codebases.”

Of course, that’s increasingly becoming a social networking skill. I recently got interested in the new XML version of Sleepycat’s Berkeley DB. If you go through the front door, on the Sleepycat Web site, you won’t find out much about that project, which is still in beta. But John Merrells, who’s working on the beta project, has a Weblog. You can get in touch with him, and he can help guide you.

It’s becoming a game of free agents, and by operating in this public and transparent way, free agents advertise themselves — as a brand — along with the products and components they have expertise with.

Behlendorf: “The applicability of this to outsourcing is really fascinating. The general mindset, whether it’s a domestic or foreign outfit, has been that you hire a company, you give them specs, they turn around and give you widgets. That sounds more like manufacturing than the creative process it really is. I hope these outsourcing organizations will start to see themselves as talent brokers and pump up the individuals in their organizations as the key asset, rather than just saying ‘Hey, we’re ISO 9002 compliant.'”

Indeed. The world’s full of smart people who have, collectively, a lot of the intellectual bandwidth needed to absorb and master open-source infrastructure. It’s the scarcity of expertise with the software that has made open source uneconomical in a lot of cases. As people in India and Russia and elsewhere dig into open-source technologies, they can broker that expertise and help bridge the gap between the theory and the practice of reuse.

Behlendorf: “Because of the whole zero-cost thing, open source is counter-cyclical. In an environment like the late 90s, where money was more plentiful than clue, you were more apt to spend money on a solution that sounded like an all-in-one thing, than to think that you could pull together a loose confederation of things that worked 90 percent of the time but required investment to fix the last 10 percent. Fast forward five years. No one wants to spend money on IT anymore. The idea of investing talent time — and talent’s a lot cheaper now, especially if you can have somebody on the other side of the world help you — has started to make more sense.”