Video codecs: The ugly business behind pretty pictures

When Google announced last week it had finally made peace over its VP8 codec with the patent pool MPEG-LA, some declared the company had sold out and was joining the software patent circus. But it’s not.

The truth is far more colorful and could presage a big shift in the struggle to control our online viewing and listening habits. As a consequence, powerful interests have quickly stepped in to try to spike VP8’s guns before they can be fired.

[ Simon Phipps tells it like it is: Why software patents are evil. | Stay up to date and subscribe to InfoWorld’s Open Source newsletter. | Read Bill Snyder’s Tech’s Bottom Line blog for what the key business trends mean to you. ]

Codec context

The realm of video codecs is complex and infested with acronyms and political intrigue stretching back decades. Even those closest to the situation disagree about both the reality and history of the situation. Here’s a condensed overview:

When you view or stream video, you may think it comes as QuickTime, Flash, or even Ogg, but those are just delivery mechanisms. Raw video is an enormous amount of data, and delivering it to you requires data compression. The actual video is encoded in a format created by video compression software, then displayed on your screen after that format is decoded by decompression software.

The codec is the software that performs this processing. The theoretical work behind codecs is exceptionally complex, and there’s always a trade-off between maximum compression, processing time, and optimal quality. As a result, a wide range of codec designs exists, and the know-how in implementing them is a valuable commodity.

As early as 1993, it became obvious that format standardization for the data handled by codecs would be needed. International standards bodies ISO and IEC formed an expert working group called the Motion Picture Expert Group — MPEG — that has since devised a series of standards for various fields.

The field is heavy with patented techniques. Codec standardization is rooted in the thinking of the telecommunications industry, where it’s common for patented techniques to be permitted in standards and then used to derive royalty payments from implementations. To make collection of payments easy, a company called MPEG-LA, LLC (that is, confusingly, entirely unafilliated with MPEG) formed to manage a patent pool acting on the behalf of many — but not all — the patent holders contributing to MPEG standards. All use of standards, such as H.264, is likely to necessitate a license from MPEG-LA — even free uses.

That arrangement worked well in the old, control-point-based world where software was created by corporations. But the new meshed society and the techniques it necessitates, such as open source, don’t blend well with a world where each new use demands permission first. Elements that need permission in advance — control points — are anathema to the Internet. They get treated as damage, and clever minds seek ways to work around them.

The rise of open codecs

Once it was clear that the open Web demanded open codecs to process open media formats, clever minds started working around the problems. The science of codecs is well documented, but using any of the well-known techniques was highly likely to result in breaching some software patent associated with MPEG-LA. It wasn’t so simple as just modifying an MPEG-derived standard to route around patents. Those standards created such a patent thicket that any new work using the same math was almost certain to encroach on a patent portfolio somewhere.

Creating open codecs demanded new thinking. Fortunately, there were commercial concerns working on alternative codec ideas. In particular, a company called On2 created a family of codecs that used ideas that were outside of the MPEG patent thicket and filed its own patents to prevent encroachment. In 2001, it released codec technology called VP3 as open source, protected by its own set of patents. That technology formed the base of what would become Theora. On2 continued its work to produce a series of niche codec technologies until it was acquired by Google in 2010.

VP8 was On2’s ultimate codec technology, offering excellent quality with good compression. Soon after the acquisition, Google released the implementation of VP8 under an open source license, made an irrevocable open covenant around all the patents it embodied, and declared that the new WebM project would offer an entirely open and free format for media.

Naturally, MPEG-LA recognized the threat and quickly decided to reciprocate. It almost immediately announced it was forming a patent pool to sell licenses to patents it was sure WebM and VP8 must infringe upon and invited the usual licensors to tell them all about those patents.

A glimmer of hope

And then … tumbleweeds. It seems MPEG-LA’s saber-rattling was more scabbard-rattling. The agreement Google announced with MPEG-LA was very carefully phrased to protect the dignity of those involved, but seems to indicate MPEG-LA came up empty:

Today, Google Inc. and MPEG LA, LLC have announced that they have entered into an agreement granting Google a license to techniques, if any, that may be essential to VP8. Furthermore, MPEG LA has agreed to discontinue efforts to form a patent pool around VP8.

You can tell there’s nothing much of value being “licensed” here, as Google is apparently entitled to the following:

… to sublicense the techniques to any user of VP8, whether the VP8 implementation is by Google or another entity; this means that users can develop independent implementations of VP8 and still enjoy coverage under the sublicenses.

The announcement goes on to make two more important statements. First, Google intends to submit VP8 to MPEG for standardization. This would be a profound change in direction, potentially steering future efforts away from the patent thicket and toward open ground. Second, Google intends to propose V P8 be selected as the “mandatory to implement” codec in the RTCWEB group at IETF that’s defining protocols to enable real-time communications in Web browsers: WebRTC.

If all this were to succeed, it would unlock immense opportunity for open source software and the open Web. Freed from constant rent-seeking by patent owners, open source developers would at last be free to innovate on audio and video applications of all kinds without constantly looking over their shoulders or asking permission to innovate.

Naturally, strong, vested interests still want to slow down this revolution or even stop it dead. As soon as Google posted its announcements about VP8, agreement with MPEG-LA, and intent to standardize, two messages were posted on the IETF RTCWEB mailing list. The first, by Microsoft Skype employee Matthew Kaufmann, tried to pour cold water on progress in standardization and invoke process and debate to keep VP8 out of the next standards discussion. The second, by former Nokia patent specialist Stephan Wenger, also invoked process, but more ominously implied MPEG-LA may not be the only game in town. That fear was soon realized by a message from Nokia employee Markus Isomaki stating Nokia — not a member of MPEG-LA — intends to assert that VP8 violates one of its patents.

This is the daily story of life in the world of codecs, and it’s highly educational about the dangers of software patents. Once accepted and accommodated into normal processes, they become control lines for everything. Despite VP8 being from a different technology heritage, having consciously avoided the patent thicket around earlier MPEG work, and having then been scrupulously opened up by Google (which had to respond to corrections in the process and did so admirably), still the toxic world of software patent control tries to claw it in and force it to submit to the taxation of innovation by the winners of an earlier race.

We can do little more than watch nervously as all the players of the open Web support Google’s initiative. In the process, it becomes clearer than ever: The reform of the patent system to address a meshed society of equals is long overdue.

This article, “Video codecs: The ugly business behind pretty pictures,” was originally published at InfoWorld.com. Read more of the Open Sources blog and follow the latest developments in open source at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.