AV1 and The Prickly Business of Video Compression

Around the time the fictional tech entrepreneur Richard Hendricks was battling a patent troll over compression tech in HBO's "Silicon Valley," the real-life Silicon Valley was dealing with patent frustrations of its own -- also over compression tech. This technology is critical for moving media around the web efficiently. There are many variations on video compression, but the media and tech industry tend to favor just a few for much of the world's digital video. In this article, we explore how patents, licenses, and royalties made video compression a sometimes-frustrating subject for engineers and business leaders; why switching to a royalty-free model isn't as easy as it sounds; and why that may finally be changing thanks to the Alliance for Open Media.

The Role of Compression on The Web

Let's start with a little background context. According to Sandvine's 2023 Global Internet Phenomena Report, about 65% of internet traffic is video. Ever since we started sharing videos on the web, compression has been around to help us make videos smaller so we can move and store them more efficiently. In media technology, we generally refer to each compression technology as a different video "codec." Without the help of a codec, every piece of hardware that helps to move video across the web would need to be many times as powerful to handle the extra data. The alternative is to settle for longer downloads or to make a significant compromise in image quality.

The Business of Video Compression

Chances are good that if you look around your computer, you'll find a video file named something like this: filename.mp4 -- go ahead, I'll wait while you look. That video file uses technology owned and licensed by MPEG-LA. They're an organization that pools together intellectual property rights (in the form of thousands of patents) from the many individuals and companies that contributed to creating the de-facto gold standard video codec: AVC (also known as H.264).

Developers that want to implement the codec in their products must purchase a license from MPEG-LA for that right -- about $0.10 per unit of hardware or software. The figures can add up, but companies have mostly been okay with paying it because it fosters an environment of broad adoption and legal clarity about their right to use it and include it in something they're selling. In MPEG-LA's words, they provide companies with "reduced litigation risk and predictability in the business planning process." Here's an example.

In 2010, Microsoft was deciding on what compression technology it would support for IE9. Writing on Microsoft's IEBlog, Dean Hachamovitch (in charge of the Internet Explorer team) posted that their main concern was committing to a technology worth supporting for many years, "the long-standing licensing program for [AVC] provides a stable system from which we can support our customers."

On the one hand, Microsoft had this stable, low-risk licensing option. Meanwhile, on the other hand, there was still a push for adding other codecs as well.

"The biggest obstacle to supporting [other codec options] is the uncertainty. [...] Many people seem to assume that availability of source code under an open source license implies that there are no additional costs, or that the code has properly secured necessary intellectual property rights from all rightful owners". - Dean Hachamovitch

For Microsoft (and likely others), the legal uncertainty around the alternatives to AVC meant potentially significant business risks. Sure, you could try to implement other codec options in Windows, but that comes with a greater risk some individual or group would later emerge and claim rights over the technology. Even if Microsoft could remove an infringing codec from Windows, the situation would likely incur costs beyond the legal dispute itself. Licensing the compression technology from MPEG-LA was a strategy with a long track record of predictable costs and low legal risks.

The best counterexample I can provide for this is Google. They wanted to go the route of a royalty-free alternative to AVC, so they essentially purchased one called VP8 through the $125 million acquisition of On2 Technologies.

Google then made the codec open source for all to use in their WebM project to encourage more innovation in the developer community. They later had to settle a legal dispute over the technology with MPEG-LA. In their 2013 press release, MPEG-LA said they grant Google "a license to techniques that may be essential to VP8 and earlier generation VPx video compression techniques." They also granted Google "the right to sublicense those techniques."

So, Google eventually achieved a sense of legal clarity. But, the situation highlights the risk for developers and other businesses should they want to take advantage of Google's, now open source, VP8.

A New Patent Pool and Confusion Over HEVC

With the rights dispute over, Google could continue to advance VP8 as a royalty-free competitor to AVC. Attention turned to 4k streaming video. Google released VP9, and MPEG-LA now had HEVC (also known as H.265).

As best as I can tell, trouble began when a new patent pool HEVC Advance launched with their own claims to HEVC. If you're feeling confused, you're not alone. The launch stirred confusion and uncertainty in the marketplace, which had already been licensing compression tech from MPEG-LA.

Companies faced significant new costs in addition to dealing with two organizations for the rights to implement the codec in their products. One report on StreamingMedia.com estimated the rates as being "seven to fifteen times more expensive than MPEG LA's licensing fees."

The new dispute undermined the legal confidence that came from working with MPEG-LA. This situation with HEVC, previously anticipated to drive the push to 4k, may have bolstered arguments for a royalty-free and legally smart alternative.

The Alliance for Open Media

Granting Google the freedom to continue developing VP8/VP9/VPx may have been a loss for MPEG LA but a big win for the open-source community. Six months after the "reduced litigation risk" for HEVC started looking shaky, the Alliance for Open Media (or simply, "AOMedia") launched. Its seven founding members were Amazon, Cisco, Google, Intel, Microsoft, Mozilla, and Netflix. The team started with Daala, Thor, and "VP10," as their foundational contributing technology.

They aimed to deliver a next-gen video format that, among other other things, would be open and royalty-free, work with any device at any bandwidth, and be optimized for hardware. That last part is critical for success as hardware encoding is required for the new video codec to work in real time. Fortunately, that's well within the wheelhouse of their founding members. The team got to work. By March of 2018, almost three years later, they released the AOMedia Video Codec 1.0 (AV1) specification.

Fast forward to 2023, both Intel and NVIDIA both have a GPU line featuring AV1 hardware-accelerated encoding. MediaTek has added AV1 decoding hardware to their smartphone chipsets. You can stream AV1-encoded videos from YouTube and Discord  added support for the codec as well saying users of their platform can now, "livestream in 4k at just 8mbps."

With the adoption growing and hardware support, I'm starting to feel pretty excited about AV1. Soon we'll have all kinds of nifty new media tools for the codec produced by independent developers (and possibly yours truly). UltraHD freedom, baby!!

Sisvel Has Entered The Chat

Before we wrap things up, let's talk a little about Sisvel Group. They are, as you've probably guessed, another patent pool.

The Sisvel Group is seeking to collect royalties by licensing AV1 (promoted as being royalty-free) under their claim that "the AV1 codec relies on many patented coding tools developed by companies not in the [Alliance for Open Media]." They've since published a patent list currently over 1500 patents long, each of which is potentially a conflict with the royalty-free aim of the Alliance for Open Media.

The Alliance for Open Media acknowledged the Joint Patent Licensing Effort saying, "AOMedia was founded to leave behind the very environment that [the joint patent effort] endorses." They're confident that they can overcome these challenges "by settling patent licensing terms up front with the royalty-free AOMedia Patent License 1.0 [which] gives all [members and non-members] royalty-free frights to [the codec] in exchange for the same royalty-free patent commitment [and establishes] a patent defense program to help protect AV1 [from patent claims]."

Alas, things are not so clear after all. Sisvel, legitimate or not, has taken the paid-vs-royalty-free codec debate full circle. Have we come all this way for nothing?

Where Do We Go From Here?

While Sisvel appears potentially threatening (their website features a prickly porcupine-queen-looking mascot), it's unclear if their patent claims have the teeth to chew threw AOMedia's defenses. At the same time, as far as I can tell, their claims haven't been outright dismissed either. That hasn't stopped big tech companies from supporting these popular codecs.

These patent scuffles happen because that's where the innovation is. In pushing the envelope with innovative content and technology, you must accept the inherent uncertainty of being on the bleeding edge. That's often where the opportunities are because as we push forward, we encounter new and exciting problems to solve.

Here is how I would expect to handle this as a product and technology management decision. I'd ship with the codecs that solve my customers' problems. If they need support for HEVC, I'll buy the license. If they need AV1, they'll get that too. It's a calculated risk. In this case, I must weigh the value of these codecs for prospective customers against the risk that someone unexpected will come knocking and asking for royalties.

As a consumer, I'm excited by what these codecs do for media. AV1 looks great; and has good hardware support on the horizon. It's also getting tested in low-latency streaming solutions that can synchronize content delivery for the entire audience. That topic will have to wait for a future post.