Hadoop project ODP regroups under Linux Foundation’s umbrella

After major criticism within the Hadoop community regarding its nature and aims, Open Data Platform — an initiative to create a reference-standard Hadoop distribution — announced Monday it will now be hosted at the Linux Foundation as a Collaborative Project.

The goal with this new organizational structure is to ameliorate the perception that the ODP Initiative is vendor-owned and vendor-controlled, rather than just participated in by Hadoop vendors among many others.

Less regress, more progress

John Mertic, senior program manager for the ODPi, explained in a phone call that the main goal of the project is to bring together representatives from different parts of the Hadoop world — not only the distributions themselves, but the makers of tool sets, the ISVs, and the customer solution providers — and remove both the duplication of effort and the divisions that have arisen between the parties deploying Hadoop.

Most of the issues, Mertic explained, revolve around the way each distribution implements common parts of Hadoop in slightly differently. “Things as simple as filenaming strategies; environment variable exposure; not changing base, core, public APIs — each vendor had a slightly different approach,” Mertic said. They’ve proven to be a hindrance to customers, he said.

What’s to ensure that the ODPi doesn’t slip back into the hands of the vendors that work with it?  “For any decision in regards to what the focuses of a spec should be, what projects we look to include or remove, all of that is entirely driven by the vendors,” Mertic said. “If a member has a project they want to add, they pass it upstream to our release team.” The latter is composed of both technical and business members.

The release team sets priorities for the requested items and determines what a release for such features will look like. The results are then passed back to the vendors for a vote, where each member has a single ballot, regardless of size.

Pulling this process under the umbrella of the Linux Foundation provides “a way for vendors to engage and work together and collaborate in a vendor-neutral environment,” without resorting to as “committer proxy war” strategies at the project level.

Under a new umbrella

Placing the ODP under the aegis of the Linux Foundation, rather than the Apache Foundation, further distinguishes the initiative’s goals. “The Apache Foundation is build more around individual contributors and projects,” Mertic explained, implying that it’s mainly about development — “a place to play for people who are deep into these toolsets.”

People who want to address their specific user cases, such as analytics or SaaS vendors, want to find a way to do that without dealing with specific projects, Mertic said. The ODP’s work is thus intended to complement the Apache Foundation’s work rather than replace it.

“[The Apache Foundation] has paths for individual contributors, but it doesn’t really have paths for companies,” Mertic said. Most of that is accomplished by having employees from those companies contribute directly to projects. The ODPi, then, is meant to be “a different level of engagement for a different sort of audience,” Mertic said.