For our May “Community Choice” Project of the Month, the community elected Libjpeg-turbo, a SIMD-accelerated libjpeg-compatible JPEG codec library. D.R. Commander, the developer and maintainer of Libjpeg-turbo, shared some thoughts about the project’s history, purpose, and direction.
SourceForge (SF): Tell me about the Libjpeg-turbo project please.
D.R. Commander (DRC): Libjpeg-turbo is a high-speed JPEG compression/decompression library that implements the libjpeg API used by a wide variety of applications (both open source and proprietary.) libjpeg-turbo accelerates the most common compression and decompression paths for JPEG images by as much as 6x relative to libjpeg, through the use of SIMD instructions on x86, ARM, PowerPC, and MIPS CPUs. libjpeg-turbo is now included in most open source operating systems (soon including Android) and is used by popular web browsers (notably Firefox and Chrome), so most people reading this are probably using libjpeg-turbo on a daily basis, whether they realize it or not.
SF: What made you start this?
DRC: My other pet projects (VirtualGL and TurboVNC) are high-performance open source remote display tools for 3D applications, so they have always needed some sort of high-speed image codec in order to stream the output of 3D applications in real time over various types of networks. “Back in the day”, I was using vendor-specific high-speed JPEG codecs, such as the Intel Performance Primitives and Sun mediaLib, but these weren’t always open source. Miyasaka Masaru developed MMX and SSE2 extensions for libjpeg in 2006. The TigerVNC developers adopted his code and did some further work on it, then they hired me in early 2009 to add 64-bit SSE2 support and to add colorspace extensions for compressing from/decompressing to 32-bit buffers. Eventually this codec matured enough that I was able to “eat my own dog food” and adopt it in VirtualGL and TurboVNC. At some point, developers in the streaming video community caught wind of our work and encouraged me to spin off the “libjpeg/SIMD” codec into a standalone project, so that other applications could easily take advantage of it. Thus, in early 2010, libjpeg-turbo was born. Some of the earliest adopters of it, outside of the remote display community, were applications that monitor security cameras.
SF: Has the original vision been achieved?
DRC: I believe it has. In fact, in most cases libjpeg-turbo now out-performs those proprietary/vendor-specific JPEG libraries that I used to use “back in the day.” Furthermore, I believe that libjpeg-turbo has helped to further Tom Lane’s original vision for libjpeg. In the release notes for libjpeg v6b, the last version that he worked on, he mentions performance enhancements as being “of great interest” for future development. Tom’s vision was to encourage wide adoption of the JPEG standard by providing a ubiquitous and legally-unencumbered JPEG codec library. libjpeg-turbo further encourages wide adoption of the JPEG standard by making JPEG fast enough to use with real-time compression/decompression tasks, which previously weren’t possible with libjpeg. Because libjpeg-turbo implements the industry-standard libjpeg API, applications that were already written to use this API can, without modification, take advantage of the additional speed in libjpeg-turbo. New applications also have the option of using the TurboJPEG API, which allows them to more straightforwardly compress/decompress images without dealing with the complexities of the libjpeg API. The TurboJPEG API also exposes functionality that would be very difficult to achieve with the libjpeg API, such as compressing from/decompressing to YUV planar image formats and doing multiple lossless transforms on the same JPEG image. We are making JPEG a lot faster and easier to use, and this enables use cases that previously wouldn’t have been possible with open source tools.
SF: Who can benefit the most from your project?
DRC: Applications that need to compress/decompress JPEG images in real time (for instance, video applications or remote display applications) benefit the most from libjpeg-turbo. Those applications were previously forced to trade off better performance for better compression, but libjpeg-turbo allows them to have both.
SF: What core need does Libjpeg-turbo fulfill?
DRC: The need for an industry-standard, ubiquitous, easy-to-use, high-speed JPEG library. Both libjpeg-turbo and libjpeg v7 and later evolved from Tom Lane’s work (he was the sole developer of libjpeg v6b and prior); but whereas libjpeg in the post-Tom Lane era has moved in the direction of introducing clever but incompatible and non-standard extensions to the JPEG format, libjpeg-turbo has instead moved in the direction of greatly speeding up the functionality that was already there, and making it easier to use. Apparently a lot of people preferred our approach.
SF: What’s the best way to get the most out of using Libjpeg-turbo?
DRC: Applications that were written to use the libjpeg API should “just work” with libjpeg-turbo, but if they have to compress from/decompress to 32-bit buffers, then they can take advantage of the libjpeg-turbo colorspace extensions in order to increase performance even more. A lot of new applications are using the TurboJPEG API instead, because it’s so much easier to use (although it doesn’t support some of the more advanced features of the libjpeg API, such as buffered I/O.) It also goes without saying that libjpeg-turbo will be of the most benefit on CPUs whose SIMD instructions we support (x86, ARM, PowerPC, MIPS.) Also, it will be necessary to compress/decompress baseline JPEG images in order to achieve peak performance (SIMD acceleration for progressive JPEGs is still a work in progress.)
SF: What has your project team done to help build and nurture your community?
DRC: libjpeg-turbo is a one-person shop. Over the years, I have received some big code contributions, particularly for things like SIMD support on ARM and MIPS processors, but I’ve always been the sole maintainer and, with the exception of outside patch submissions, the sole developer. The libjpeg-turbo community initially formed on its own, because there was a pent-up demand for a library that furthered Tom Lane’s work without breaking backward compatibility with libjpeg v6b. Also, a lot of developers liked the fact that our project maintains a fully open code repository and issue trackers, things that have traditionally not existed with libjpeg. Since its early days, I have built the libjpeg-turbo community simply by being as responsive as I could to the needs of libjpeg-turbo users, and evolving the project in the direction that most of them seemed to want it to go.
SF: Have you all found that more frequent releases helps build up your community of users?
DRC: There’s always a balance to be struck between releasing too often and not releasing often enough. Our community of “users” largely consists of application developers, and they (as I) are more concerned with quality over quantity. Thus, libjpeg-turbo has traditionally fallen somewhere between ESR’s classic metaphors of the cathedral and the bazaar. Major releases are dictated by new features, and new features are dictated by community demand. I always strive to implement and release bug fixes as quickly as possible, but I take great care to ensure that all new features are fully vetted– that they truly solve a well-defined problem in as elegant a way as possible, without creating any backward incompatibilities or regressions. Above all, I want libjpeg-turbo (and all of my other projects) to have the maximum possible utility, stability, and performance. I treat performance as a measure of quality, so any drop in performance is considered to be a bug. Our releases are dictated by whether there are sufficient changes relative to the prior release to justify a new release, and I would much rather delay a release than put out something that is half-baked or irrelevant. I use a traditional release model, whereby I put out a beta release, a final release (usually 3 months following the beta), then usually one or two subsequent bug-fix releases (minor revisions) for each major revision. So to answer your question, it isn’t the frequency of releases that builds the community of users– it’s being responsive to what the users really want. They mostly want libjpeg-turbo to change only if the change is compelling, and they want assurance that these changes will be implemented elegantly, with an eye toward ease of readability and maintenance, and without compromising stability or performance.
SF: What was the first big thing that happened for your project?
DRC: Probably the inclusion of libjpeg-turbo in Fedora. I never set out to build an industry-standard JPEG library. Originally I just wanted to build a fast, open source JPEG library that I could use in my own remote display projects. Red Hat was the first one to include libjpeg-turbo in their O/S distribution, and it kind of snowballed from there.
SF: What helped make that happen?
DRC: In addition to its increased performance, many O/S distributors (including Red Hat) and application developers also gravitated toward libjpeg-turbo because it maintained backward ABI compatibility with libjpeg v6b. Thus, if an O/S was already using libjpeg v6b, they could switch all of their bundled applications to libjpeg-turbo without rebuilding them. libjpeg v7 and later broke backward ABI compatibility, and thus applications had to be recompiled if they wanted to switch from libjpeg v6b to one of the newer IJG releases. Backward ABI compatibility has always been a problem in libjpeg, because all of the structures in the API are exposed. However, libjpeg v6b was out there for such a long time before libjpeg v7 was released (more than 11 years) that the libjpeg v6b ABI became something of a de facto standard, and when libjpeg v7 broke backward ABI compatibility, that caused problems for a lot of developers and integrators. Many of them decided that the features introduced in libjpeg v7 and later weren’t compelling enough to justify breaking backward compatibility, so that drove them toward libjpeg-turbo, irrespective of the performance improvements.
SF: How has SourceForge and its tools helped your project reach that success?
DRC: I particularly like the file release system on SourceForge, because it helps me track the number of downloads for a particular release (which helps to gauge its rate of adoption.) Also, because SourceForge supports SSH, I can easily push new releases and pre-releases to SF as part of my automated build process. Back when the Allura transition took place, I suggested (and the SF developers implemented) several enhancements to the code viewer that helped me maintain my preferred code review workflow.
SF: What is the next big thing for Libjpeg-turbo?
DRC: I’m currently working on extending the library to support AVX2 instructions, which we’re hoping will increase performance by 20-30% on newer x86 platforms that support that instruction set. This feature is being enabled by code contributions and funding from Intel. Another project on the horizon is SIMD acceleration for progressive JPEG compression, which should speed up the compression of progressive JPEGs by about 2x.
SF: How long do you think that will take?
DRC: I expect that both features will land this Fall.
SF: Do you have the resources you need to make that happen?
DRC: The two aforementioned features are funded, but in general, I rely on donations and funded development opportunities to keep this project moving forward. I am an independent developer, so I don’t draw a salary for my work on open source projects. I do things that way because it allows me to keep these projects vendor-agnostic and free of any one organization’s agenda, but the downside is that I have to frequently ask for money in order to keep the lights on. Even if someone else develops a new feature and submits it as a patch, the best-written patches will still require many hours of work to clean up and integrate. It’s hard sometimes to make people understand that maintaining a high-quality open source project costs money, even though the software itself is free. In general, only about half of the work I do on libjpeg-turbo is paid for (either directly, through funded development and donations, or indirectly, through funding from my other open source projects that use libjpeg-turbo.) The other half is pro bono, and it’s a constant struggle to balance the need to move the project forward with the need to put food on the table.
SF: If you had to do it over again, what would you do differently for Libjpeg-turbo?
DRC: I would have probably chosen a different name. There has been a great deal of confusion over the fact that our project is called “libjpeg-turbo” but we provide an API called “TurboJPEG”. However, there’s too much traction at this point to consider a name change.