libjpeg-turbo-devel — libjpeg-turbo Developer Discussion (moved to Google Groups-- see http://www.libjpeg-turbo.org/About/MailingLists)

[Libjpeg-turbo-devel] libjpeg-turbo mailing lists have moved to Google Groups

[DRC <dco...@us...>]

SourceForge removed the ability for project admins to view/manage the
members of mailing lists, which was the final straw.  Thanks to all who
gave their feedback, but given that my other OSS projects are using
Google Groups and I was unable to find anything better, I decided to use
Google Groups for this project as well.  You can join the new groups
either using a Google account (which gives you full access to the group
from the web interface) or a non-Google e-mail address (which gives you
access to post from e-mail only.)  Links for joining all of the groups
can be found here:

http://www.libjpeg-turbo.org/About/MailingLists

(Note that you have to be logged in with a Google account in order to
join with that account.)

To unsubscribe from the old lists, visit one or more of the following URLs:
https://sourceforge.net/projects/libjpeg-turbo/lists/libjpeg-turbo-announce/unsubscribe
https://sourceforge.net/projects/libjpeg-turbo/lists/libjpeg-turbo-users/unsubscribe
https://sourceforge.net/projects/libjpeg-turbo/lists/libjpeg-turbo-devel/unsubscribe

The direct addresses for e-mailing the new groups are:
lib...@go...
lib...@go...

Other notes:
-- If you join with a non-Google e-mail address, Google makes you go
through a really annoying Turing test.  I apologize in advance for that.
-- The old mailing lists will be retained on SourceForge for archival
purposes.  Direct links to these archives are available at
http://www.libjpeg-turbo.org/About/MailingLists.
-- The new mailing lists have already been added to mail-archive.com.
-- The new lists are public (anyone can join or view the messages), but
the subscriber lists are visible only to me.
-- As with the existing lists, only members can post (and only I can
post to the announcement list.)
-- The new lists are configured such that your Google profile will not
be revealed if you post.  Only your chosen display name will be shown.

Please e-mail me directly (http://www.libjpeg-turbo.org/About/Contact)
or post a GitHub tracker issue
(https://github.com/libjpeg-turbo/libjpeg-turbo/issues/new) should you
have any questions, concerns, or problems subscribing to the new lists.

The old lists will be closed to new messages shortly, so please transfer
your subscriptions immediately.

DRC

[Libjpeg-turbo-devel] Moving libjpeg-turbo mailing lists-- FEEDBACK REQUESTED

[DRC <dco...@us...>]

SourceForge removed the ability for project admins to view/manage the
members of mailing lists, which was the final straw (I had to jump
through hoops just to send this e-mail from my SourceForge e-mail
address, which for some reason was no longer subscribed or registered as
a list admin.)  I've already moved my other project mailing lists
(VirtualGL and TurboVNC) to Google Groups, but given that this project
has a much broader reach, I wanted to solicit feedback from the
community.  In particular, I realize that Google is banned in some
countries that are heavy users of libjpeg-turbo, so I welcome all
suggestions for how to migrate this aspect of our project to a platform
other than SourceForge.  Please comment here or on the following GitHub
issue:

https://github.com/libjpeg-turbo/libjpeg-turbo/issues/165

If I don't hear any feedback in a timely manner, I will proceed with
migrating to Google Groups.

DRC

[Libjpeg-turbo-devel] libjpeg-turbo 1.5.2 has been released

[DRC <dco...@us...>]

Official binaries are here:

https://sourceforge.net/projects/libjpeg-turbo/files/1.5.2

Change log is here:

https://github.com/libjpeg-turbo/libjpeg-turbo/releases/tag/1.5.2

Re: [Libjpeg-turbo-devel] passing the right amount of data to libjpeg-turbo

[DRC <dco...@us...>]

Aye, there's the rub.  I'm not sure if that's possible.  :(  You could
create a progress monitor, but the progress monitor won't be called
between MCUs.  It is only called once for each call to
jpeg_read_scanlines(), so I think the best granularity you could achieve
would be for the progress monitor to be called between each MCU row.
Also, since the MCU column counter is internal, there wouldn't be any
way of tracking the progress in terms of individual MCUs decoded unless
you made a backward-incompatible modification to the library.

I really think the best thing you can do is to use a larger buffer in
order to reduce the ratio of slow to fast MCU decodes to the point that
the slow MCU decodes are an insignificant part of the overall execution
time.  That's essentially what the TurboJPEG wrapper does.

On 6/16/17 2:44 PM, Leon Scroggins via Libjpeg-turbo-devel wrote:
> Thanks! Is there a good way to ensure that fill_input_buffer is called
> (i.e. the buffer is empty) at the start of an MCU? Otherwise I could
> imagine that we might use a large enough value, but still end up hitting
> the slow case because we exhaust our buffer in the middle of the first MCU.
> 
> On Fri, Jun 16, 2017 at 3:17 PM, DRC via Libjpeg-turbo-devel
> <lib...@li...
> <mailto:lib...@li...>> wrote:
> 
>     Referring to the code in jdhuff.c, the slow case will be used if:
>     (1) There are fewer than cinfo->blocks_in_MCU * 128 bytes in the
>     buffer, or
>     (2) There is an unread marker from a previous entropy decode (such as a
>     restart marker), or
>     (3) The fast case encounters a marker (such as a restart marker)
> 
>     There is nothing you can do about (2) and (3), but you should be able to
>     minimize the use of the slow path by ensuring that at least
>     cinfo->blocks_in_MCU * 128 bytes are in the buffer.  You may not be able
>     to ensure that at the end of the image, though.
> 
>     On 6/16/17 11:29 AM, Leon Scroggins via Libjpeg-turbo-devel wrote:
>     > What is the best way to determine how much data to pass to
>     > fill_input_buffer? Often we are streaming data, so we need to copy it
>     > into a buffer before providing it. We've noticed that depending on the
>     > amount of data in the buffer, libjpeg-turbo will call either call
>     > decode_mcu_slow or decode_mcu_fast, which can make a difference in
>     > decoding speed. We can always use a large buffer in order to make us
>     > more likely to hit the fast case, but we can still hit the slow case
>     > sometimes. I was wondering if there might be some query we can use to
>     > determine the ideal amount of data to supply in a given call to
>     > fill_input_buffer so that we never hit the slow case?
>     >
>     > --
>     >
>     >  •  Leon Scroggins
>     >
>     >  • sc...@go... <mailto:sc...@go...>
>     <mailto:sc...@go... <mailto:sc...@go...>>
> 
>     ------------------------------------------------------------------------------
>     Check out the vibrant tech community on one of the world's most
>     engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>     _______________________________________________
>     Libjpeg-turbo-devel mailing list
>     Lib...@li...
>     <mailto:Lib...@li...>
>     https://lists.sourceforge.net/lists/listinfo/libjpeg-turbo-devel
>     <https://lists.sourceforge.net/lists/listinfo/libjpeg-turbo-devel>
> 
> 
> 
> 
> -- 
> 
>  •  Leon Scroggins
> 
>  • sc...@go... <mailto:sc...@go...>

Re: [Libjpeg-turbo-devel] passing the right amount of data to libjpeg-turbo

[Leon S. <sc...@go...>]

Thanks! Is there a good way to ensure that fill_input_buffer is called
(i.e. the buffer is empty) at the start of an MCU? Otherwise I could
imagine that we might use a large enough value, but still end up hitting
the slow case because we exhaust our buffer in the middle of the first MCU.

On Fri, Jun 16, 2017 at 3:17 PM, DRC via Libjpeg-turbo-devel <
lib...@li...> wrote:

> Referring to the code in jdhuff.c, the slow case will be used if:
> (1) There are fewer than cinfo->blocks_in_MCU * 128 bytes in the buffer, or
> (2) There is an unread marker from a previous entropy decode (such as a
> restart marker), or
> (3) The fast case encounters a marker (such as a restart marker)
>
> There is nothing you can do about (2) and (3), but you should be able to
> minimize the use of the slow path by ensuring that at least
> cinfo->blocks_in_MCU * 128 bytes are in the buffer.  You may not be able
> to ensure that at the end of the image, though.
>
> On 6/16/17 11:29 AM, Leon Scroggins via Libjpeg-turbo-devel wrote:
> > What is the best way to determine how much data to pass to
> > fill_input_buffer? Often we are streaming data, so we need to copy it
> > into a buffer before providing it. We've noticed that depending on the
> > amount of data in the buffer, libjpeg-turbo will call either call
> > decode_mcu_slow or decode_mcu_fast, which can make a difference in
> > decoding speed. We can always use a large buffer in order to make us
> > more likely to hit the fast case, but we can still hit the slow case
> > sometimes. I was wondering if there might be some query we can use to
> > determine the ideal amount of data to supply in a given call to
> > fill_input_buffer so that we never hit the slow case?
> >
> > --
> >
> >  •  Leon Scroggins
> >
> >  • sc...@go... <mailto:sc...@go...>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Libjpeg-turbo-devel mailing list
> Lib...@li...
> https://lists.sourceforge.net/lists/listinfo/libjpeg-turbo-devel
>



-- 

 •  Leon Scroggins

 •  sc...@go...

Re: [Libjpeg-turbo-devel] passing the right amount of data to libjpeg-turbo

[DRC <dco...@us...>]

Referring to the code in jdhuff.c, the slow case will be used if:
(1) There are fewer than cinfo->blocks_in_MCU * 128 bytes in the buffer, or
(2) There is an unread marker from a previous entropy decode (such as a
restart marker), or
(3) The fast case encounters a marker (such as a restart marker)

There is nothing you can do about (2) and (3), but you should be able to
minimize the use of the slow path by ensuring that at least
cinfo->blocks_in_MCU * 128 bytes are in the buffer.  You may not be able
to ensure that at the end of the image, though.

On 6/16/17 11:29 AM, Leon Scroggins via Libjpeg-turbo-devel wrote:
> What is the best way to determine how much data to pass to
> fill_input_buffer? Often we are streaming data, so we need to copy it
> into a buffer before providing it. We've noticed that depending on the
> amount of data in the buffer, libjpeg-turbo will call either call
> decode_mcu_slow or decode_mcu_fast, which can make a difference in
> decoding speed. We can always use a large buffer in order to make us
> more likely to hit the fast case, but we can still hit the slow case
> sometimes. I was wondering if there might be some query we can use to
> determine the ideal amount of data to supply in a given call to
> fill_input_buffer so that we never hit the slow case?
> 
> -- 
> 
>  •  Leon Scroggins
> 
>  • sc...@go... <mailto:sc...@go...>

[Libjpeg-turbo-devel] passing the right amount of data to libjpeg-turbo

[Leon S. <sc...@go...>]

What is the best way to determine how much data to pass to
fill_input_buffer? Often we are streaming data, so we need to copy it into
a buffer before providing it. We've noticed that depending on the amount of
data in the buffer, libjpeg-turbo will call either call decode_mcu_slow or
decode_mcu_fast, which can make a difference in decoding speed. We can
always use a large buffer in order to make us more likely to hit the fast
case, but we can still hit the slow case sometimes. I was wondering if
there might be some query we can use to determine the ideal amount of data
to supply in a given call to fill_input_buffer so that we never hit the
slow case?

-- 

 •  Leon Scroggins

 •  sc...@go...

Re: [Libjpeg-turbo-devel] libjpeg-turbo as JPEG reference software?

[DRC <dco...@us...>]

On 4/26/17 2:41 PM, Thomas Richter wrote:
> Please don't worry too much about this step right now, we're very early
> in the process, and a formal review will be performed as part of the
> process which will give all participants the chance to update their
> implementations as necessary. We are currently at "draft call for
> proposal" stage, which means that we are trying to contact interested
> parties on the call, and receive feedback on how to improve the call
> before we finally issue it. This will happen at our next meeting in Torino.
> 
> Following that, we will receive contributions - hopefully including
> libjpeg-turbo. This will happen approximately in October at the Macau
> meeting. Following that, we will have time until January to iteratively
> test software and improve it. This will likely follow the typical "ISO
> guidelines" of going through several stages, and will take probably 2
> years. So no immediate worries, and there will be plenty of time to
> update, improve and review the software(s).

Cool.  Please keep me in the loop as to when and what I would need to
submit.


> Lossless JPEG is a horse of a different color, but it is in use in the
> medical market, which is why we seek implementations of it. In
> particular, there are a couple of known misconceptions, which is why a
> good reference has some relevance.
> 
> Anyhow, all we need is probably to cover all features of the standard in
> at least one implementation listed in the reference software. If you
> cannot provide lossless, probably some other implementation might. So
> please don't feel forced at this time to implement all of it.

It seems that there is a patch to add lossless JPEG capabilities to
jpeg-6b
(https://github.com/spectra/conquest-dicom-server/blob/master/jpeg-6c/ljpeg-6b.patch),
although I would need to do some work in order to integrate it without
breaking ABI compatibility.


> Of course, I did some tests with Thomas Lane's 6b version, and it
> satisfies these bounds *unless* you use the "fast" option. "fast"
> employs a simpler (scaled) DCT that is not quite precise enough for what
> we need.

Good to know.

Re: [Libjpeg-turbo-devel] libjpeg-turbo as JPEG reference software?

[DRC <dco...@us...>]

Hi, Thomas.  You can always find my direct e-mail address here:
http://www.libjpeg-turbo.org/About/Contact

but I'm responding on the mailing list so our conversation will be
easily searchable.  I am 100% in favor of libjpeg-turbo becoming an
official reference implementation.  However, whereas I have a deep
understanding of the software, my understanding of the official specs is
much less deep.  One of the goals of libjpeg-turbo has always been to
retain bitwise compatibility with jpeg-6b, and I've operated under the
assumption that, if jpeg-6b was conformant to the relevant ISO specs,
our library would be as well.  However, I don't think such has ever been
formally reviewed.

If I understand correctly, we provide most of the features listed in the
draft CfP, but I would appreciate some assistance in evaluating whether
libjpeg-turbo provides conformant versions of those features.  Also, if
I understand the terminology correctly, we don't currently provide
lossless (predictive) Huffman coding, which is listed as a high-priority
item, so we'd need to discuss how to implement that.  There are other
high-profile features whose terminology I don't fully understand, such
as "Conformance to the (tighter) error bounds specified in 18477-4".  I
would really like to open a dialog with you or someone else with
experience in these standards regarding whether libjpeg-turbo currently
conforms to them and, if not, what it would take to become conformant.

Darrell


On 4/26/17 3:12 AM, Thomas Richter wrote:
> Darell,
> 
> (unfortunately, I did not find a valid email to reach you).
> 
> 
> libjpeg-turbo is certainly the most prominent and most-used
> implementation of the JPEG standard, and as such the most important
> lossy still image compression deployed on the world. Now, strangely
> enough, "we", namely the ISO standardization group behind JPEG, also
> named JPEG (more formally SC29WG1), have long missed the creation of a
> formal "ISO reference" for a JPEG implementation, of which libjpeg-turbo
> would be of course an excellent candidate.
> 
> To give you an idea: A "reference software" is a code that points users
> and vendors to a piece of software that tells them "how JPEG is supposed
> to be done correctly", it is a guidance for JPEG technology.  Currently,
> there simply is no such code.
> 
> Now, recently, we started an activity that will, hopefully, result in
> the creation of such a standard, and created for that a "draft call for
> proposals" asking for input or contribution in this direction.
> 
> Now, certainly, libjpeg-turbo is the de-facto reference as far as JPEG
> implementations are concerned, and you are certainly a very knowledgable
> person in this area, so I would be very glad if you could comment on or
> even respond to our "draft call for proposals" for such a reference
> software - which you find here on the official JPEG webpage:
> 
> https://jpeg.org/items/20170413_cfp_jpeg_reference_software.html
> 
> I would be very happy if you as the libjpeg-turbo maintainer would
> respond to this "draft" call, or even better, would be able to
> contribute your software as a candidate for such a reference software.
> 
> If you have further questions, please respond to this mail directly. My
> email address is valid.
> 
> Best regards,
> 
> Thomas Richter

[Libjpeg-turbo-devel] libjpeg-turbo as JPEG reference software?

[Thomas R. <ri...@ti...>]

Darell,

(unfortunately, I did not find a valid email to reach you).


libjpeg-turbo is certainly the most prominent and most-used
implementation of the JPEG standard, and as such the most important
lossy still image compression deployed on the world. Now, strangely
enough, "we", namely the ISO standardization group behind JPEG, also
named JPEG (more formally SC29WG1), have long missed the creation of a
formal "ISO reference" for a JPEG implementation, of which libjpeg-turbo
would be of course an excellent candidate.

To give you an idea: A "reference software" is a code that points users
and vendors to a piece of software that tells them "how JPEG is supposed
to be done correctly", it is a guidance for JPEG technology.  Currently,
there simply is no such code.

Now, recently, we started an activity that will, hopefully, result in
the creation of such a standard, and created for that a "draft call for
proposals" asking for input or contribution in this direction.

Now, certainly, libjpeg-turbo is the de-facto reference as far as JPEG
implementations are concerned, and you are certainly a very knowledgable
person in this area, so I would be very glad if you could comment on or
even respond to our "draft call for proposals" for such a reference
software - which you find here on the official JPEG webpage:

https://jpeg.org/items/20170413_cfp_jpeg_reference_software.html

I would be very happy if you as the libjpeg-turbo maintainer would
respond to this "draft" call, or even better, would be able to
contribute your software as a candidate for such a reference software.

If you have further questions, please respond to this mail directly. My
email address is valid.

Best regards,

Thomas Richter

Re: [Libjpeg-turbo-devel] Inclusion of iccjpeg.c and header file in libjpeg-turbo

[DRC <dco...@us...>]

This feature is now available in the dev branch and Linux pre-release
builds (OS X CI builds are still pending, and Windows builds are
currently failing because the NASM web site is down.)

Please test it and let me know what breaks.

On 12/3/16 5:53 PM, Thomas Weber wrote:
> Hi, 
> I would like to discuss a possible inclusion of iccjpeg.{c,h} into
> libjpeg-turbo. In 2013, Paweł Hajdan, Jr. already opened a discussion[1]
> about the possibility of such an inclusion, but there was no final
> decision.
> I maintain lcms in Debian and have a feature request open to expose
> iccjpeg via lcms' API[2], such that Chromium (and probably Firefox and
> lcms) could link against it.
> Looking at the discussion at [1], I have the feeling that the old
> discussion focused a bit too much on the fact that iccjpeg is currently
> part of lcms. While this is true, it is not really an integral part of
> lcms and ended up there more by accident. Indeed, it was originally
> meant for inclusion in libjpeg[3]. In lcms, it is today only used to
> build an utility named jpgicc, an ICC profile applier for JPEG.
> 
> [1] https://sourceforge.net/p/libjpeg-turbo/mailman/message/30387709/
> (sorry for not replying to that thread, but I did not find a way to
> extract the message-id)
> [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=747839
> [3] https://github.com/mm2/Little-CMS/issues/37#issuecomment-66450180
> 
> 	Thomas

Re: [Libjpeg-turbo-devel] Earlier GPU decoding for JPEGs

[DRC <dco...@us...>]

JPEG is JPEG, so in terms of bandwidth and latency, it doesn't really
matter whether it's compressed by the GPU or the CPU.  On a
bandwidth-limited network, the network will still likely be the
bottleneck, so (up to a point) the speed of compression isn't as
important in such environments as how well the image compresses.
Whereas VirtualGL uses pure JPEG, most people these days use VirtualGL
inside of an X proxy, and X proxies tend to have more advanced forms of
compression.  The one I produce and recommend for use with VirtualGL
(TurboVNC) uses a hybrid encoding scheme whereby various indexed color
techniques are used to send areas of the display that have a low number
of unique colors, and JPEG is used only for areas of the display that
have a high number of unique colors.

The VirtualGL Transport (VirtualGL's built-in image compression/delivery
system) isn't particularly useful on high-latency/low-bandwith networks,
because it relies on remote X11 to send any non-OpenGL areas of the
application's window(s).  The biggest bang for the buck in terms of
improving TurboVNC on such networks is going to likely come from using
progressive or arithmetic-coded JPEGs.  There are already patches
submitted that would improve progressive encoding performance in
libjpeg-turbo by about 2x.  It will take significant time to integrate
those patches properly, which is why it hasn't been done yet (I'm
seeking funding to pay for that labor.)  While it would still be
significantly slower than baseline encoding, a 2x improvement would put
the progressive encoder into the range of 15-30 Megapixels/sec, which is
enough to produce at least usable frame rates on a 2-megapixel display
(by comparison, 15-30 Megapixels/sec is about what we were getting with
baseline JPEG encoding when VirtualGL was first released in 2004.)  The
average compression with progressive JPEG would be about 10% better than
with baseline.  That would be combined with an existing mode in TurboVNC
that offers a similarly extreme tradeoff for the non-JPEG tiles
(https://github.com/TurboVNC/turbovnc/issues/37), resulting in an
overall 10-20% improvement in compression while using 2x the CPU cycles
(per pixel.)  10-20% is fairly yawn-inducing, though.  In any extremely
low-bandwidth situation, you're already going to have to sacrifice a lot
of image quality to get decent frame rates, and the incremental
sacrifice in quality necessary to achieve 10-20% more compression is not
likely to be very noticeable.

The other thing I'm looking at is leveraging the onboard H.264 encoding
ability on nVidia GPUs (https://github.com/TurboVNC/turbovnc/issues/19),
since H.264 does improve compression relative to JPEG on some workloads.
 GPU encoding in VirtualGL is kind of a tricky business, though, because
of two factors:

(1) The GPU is also being used actively for 3D rendering, so care has to
be taken to avoid contention between rendering and encoding.  VirtualGL
currently pipelines both of those operations, so the CPU can be encoding
the current frame while the GPU renders the next one.

(2) In an X proxy environment, GPU encoding introduces all sorts of
architectural hurdles, since the image encoding is no longer occurring
in the same process as the 3D rendering.  If GPU encoding took place
within VirtualGL, then it might be necessary to serialize it with the 3D
rendering, so the overall performance relative to the existing solution
could very well drop.  In terms of provisioning, GPU cycles are still a
more precious resource than CPU cycles.

If you want to continue this discussion, let's move it to either
VirtualGL-devel or TurboVNC-devel, since this list is specifically for
libjpeg-turbo developers, many of whom probably don't care about remote
display solutions.


On 1/15/17 6:59 PM, Shanon Loughton wrote:
> 
> Hi DRC
> 
> Is this project conceptually a possible improvement on libjpeg-turbo
> codec for high latency low bandwidth connections? To be used in
> conjunction with VirtualGL.
> 
> https://github.com/negge/jpeg_gpu <https://github.com/negge/jpeg_gpu>
> 
> cheers
> Shanon
> 
> 
> *Shanon Loughton*

Re: [Libjpeg-turbo-devel] Earlier GPU decoding for JPEGs

[Shanon L. <slo...@cl...>]

Hi DRC

Is this project conceptually a possible improvement on libjpeg-turbo codec
for high latency low bandwidth connections? To be used in conjunction with
VirtualGL.

https://github.com/negge/jpeg_gpu

cheers
Shanon


*Shanon Loughton*

Re: [Libjpeg-turbo-devel] Optimizing JPEG compression speed

[DRC <dco...@us...>]

Good to know that it really is mostly as simple as I thought, as long as
RST markers are used.  Without them, it does seem that my assertion was
correct that Huffman coding has to occur sequentially:

http://www.sersc.org/journals/IJMUE/vol7_no2_2012/10.pdf

which limits the speedup somewhat.  From a strategic point of view, it
should be possible to implement such a multi-threaded encoder/decoder
within the TurboJPEG API, since that API is much more coarse-grained.
However, the likelihood that I could get someone to fund that project is
pretty slim, since not a lot of people use the TurboJPEG API.  For the
moment, I'm focusing on securing funding to finish the AVX2 project,
since that will provide about 25% speedup on average relative to the
current releases of libjpeg-turbo.  That should put our compressor at
least in the ballpark of PicTools.

On 12/20/16 11:50 PM, Hao Hu wrote:
> It's operating single image with multi-thread.
> 
> I shared my design a while ago on the mailing list here. But just to
> share it once more. (I guess DRC was right about using RST marker, but
> the details are different)
> I can't provide the code since it's in proprietary code base. But the
> entire implementation should be less than 300 lines of code.
> 
> HOWEVER, it only works when you control both sides. Also, I only
> implemented decoding side since that's the place I need good perf. But I
> guess the similar theory could be applied to compression side.

Re: [Libjpeg-turbo-devel] Optimizing JPEG compression speed

[Hao Hu <ih...@gm...>]

It's operating single image with multi-thread.

I shared my design a while ago on the mailing list here. But just to share
it once more. (I guess DRC was right about using RST marker, but the
details are different)
I can't provide the code since it's in proprietary code base. But the
entire implementation should be less than 300 lines of code.

HOWEVER, it only works when you control both sides. Also, I only
implemented decoding side since that's the place I need good perf. But I
guess the similar theory could be applied to compression side.

[image: 内嵌图片 1]

2016-12-20 8:42 GMT-08:00 Siddharth Bidasaria <
sid...@ci...>:

> Thank you for your responses, they are much appreciated!
>
>
>
> I meant that Libjpeg-turbo was faster than Pegasus by 40% for decoding
> jpegs, but slower by 30% when used for encoding rgb images to jpegs.
>
>
>
> By multi-threading, do you mean encoding multiple images in parallel, or
> splitting up 1 image into multiple parts and encoding them in parallel and
> rejoining them? If it’s the second case, could you point me to a an example
> implementation or resource that describes such a process. I tried looking
> for it myself, but found nothing so far.
>
>
>
> Thanks,
>
>
>
>
>
> *From:* Hao Hu [mailto:ih...@gm...]
> *Sent:* Tuesday, December 20, 2016 12:04 AM
> *To:* libjpeg-turbo Developers <lib...@li...>
> *Subject:* Re: [Libjpeg-turbo-devel] Optimizing JPEG compression speed
>
>
>
> If you search Pegasus on this link https://sourceforge.net/
> p/freeimage/discussion/36111/thread/fb51778d/
>
> You'll see your result is aligned with others observation.
>
>
>
> If you control the compression/decompression, then there are ways to make
> libjpeg-turbo many times faster by design a multi-threading based wrapper
> code.
>
> Then if you have 4 cores, you usually get at least 3x faster. But you'll
> need to come up with the design based on your specific case.
>
>
>
> 2016-12-19 15:21 GMT-08:00 DRC <dco...@us...>:
>
> Nothing obvious stands out in your code.  I don't quite understand your
> claims below that "libjpeg-turbo came out behind by 30%" but
> "libjpeg-turbo seemed to be 40% faster while decoding images at the same
> quality level".  Those seem to be contradictory claims.  Please clarify.
>
> You can get probably 10-15% better performance from the dev (1.6
> evolving) builds, if you're using an AVX2-equipped CPU (I expect that to
> be more like 25% when libjpeg-turbo 1.6 ships, but I'm seeking funding
> to finish out the AVX2 acceleration project.)
>
> I'm not really surprised that Pegasus is faster.  They do use SIMD
> instructions, and they also use multi-threading.  The latter is
> something we cannot do within our library because of the libjpeg
> architecture, although applications are of course free to implement
> multi-threaded compression or decompression on their own.  Our initial
> performance goal was to match the performance of the Intel Performance
> Primitives, which we pretty much have done, but Pegasus has always been
> a notch faster than IPP in my experience.
>
> Back in the day, I had a TurboJPEG API implementation for Pegasus, which
> can be found here:
> https://github.com/VirtualGL/virtualgl/tree/029f870488d251b48c3be0b17804e3
> 002f7602cd/jpeg
>
> You should be able to build turbojpegp.c and jpgtest.cpp against Pegasus
> and obtain benchmark data that closely resembles that of tjbench in
> libjpeg-turbo.  That would be a good sanity check.
>
> On 12/19/16 2:56 PM, Siddharth Bidasaria wrote:
> > Hi all,
> >
> >
> >
> > I am currently using a proprietary jpeg encoding library called Pegasus,
> > and was looking to switch to libjpeg-turbo.
> >
> >
> >
> > I have been conducting some benchmarks between the two libraries and
> > libjpeg-turbo came out behind by 30%. I was quite surprised by this
> > because Pegasus was last updated 4 years ago, and I believe it doesn’t
> > make use of SIMD instructions either. Moreover, libjpeg-turbo seemed to
> > be 40% faster while decoding images at the same quality level.
> >
> >
> >
> > Hence, I wanted to rule out that I am incorrectly/inefficiently using
> > the libjpeg-turbo api. If you could have a look at my compression code
> > below, and let me know if there are any glaring things I
> > should/shouldn’t be doing please let me know!
>
> ------------------------------------------------------------
> ------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today.http://sdm.link/intel
> _______________________________________________
> Libjpeg-turbo-devel mailing list
> Lib...@li...
> https://lists.sourceforge.net/lists/listinfo/libjpeg-turbo-devel
>
>
>
> ------------------------------------------------------------
> ------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today.http://sdm.link/intel
> _______________________________________________
> Libjpeg-turbo-devel mailing list
> Lib...@li...
> https://lists.sourceforge.net/lists/listinfo/libjpeg-turbo-devel
>
>

Re: [Libjpeg-turbo-devel] Optimizing JPEG compression speed

[DRC <dco...@us...>]

I'm really just guessing.  It's been over 10 years since I played with
PicTools (well before this project existed), so I'm not sure why
libjpeg-turbo would be faster at decompression and slower at
compression.  My best guess is that their multi-threaded optimizations
are focused more on the compression side (see below for a possible
reason behind this.)

My own downstream project (TurboVNC) implements multithreading by
encoding multiple regions of a large image (specifically a remote
desktop image) in parallel.  I'm able to get away with that because I
control both the server and the client, and the protocol (RFB) is
designed such that you can encode different parts of a framebuffer
update as separate images.  That kind of parallelism is trivial, but
using multithreading to encode or decode a single image is somewhat less
trivial.  I am not an expert on that, but my basic understanding of the
strategy is that you would split the input image into horizontal strips
that contain whole numbers of MCU rows.  Each strip would be encoded
separately, then pieced together into a final image through the use of
restart markers.  Restart markers allow Huffman coding to stop and
restart, which is necessary when implementing a parallel compressor
(otherwise the Huffman encoder would have to run serially on the whole
image, once the parallel threads are complete.)  Decoding is trickier.
If the input image doesn't contain restart markers, then generally the
Huffman decoder will have to run serially as well.  Even with restart
markers, the problem becomes how to know where they are in advance (the
libjpeg[-turbo] decoder generally doesn't know about restart markers
until it encounters them while decompressing.)  It may be that Pegasus
decided to avoid these pitfalls and use multithreading only in the
compressor.


On 12/20/16 10:42 AM, Siddharth Bidasaria wrote:
> I meant that Libjpeg-turbo was faster than Pegasus by 40% for decoding
> jpegs, but slower by 30% when used for encoding rgb images to jpegs.
> 
>  
> 
> By multi-threading, do you mean encoding multiple images in parallel, or
> splitting up 1 image into multiple parts and encoding them in parallel
> and rejoining them? If it’s the second case, could you point me to a an
> example implementation or resource that describes such a process. I
> tried looking for it myself, but found nothing so far.
> 
>  
> 
> Thanks,
> 
>  
> 
>  
> 
> *From:*Hao Hu [mailto:ih...@gm...]
> *Sent:* Tuesday, December 20, 2016 12:04 AM
> *To:* libjpeg-turbo Developers <lib...@li...>
> *Subject:* Re: [Libjpeg-turbo-devel] Optimizing JPEG compression speed
> 
>  
> 
> If you search Pegasus on this
> link https://sourceforge.net/p/freeimage/discussion/36111/thread/fb51778d/
> 
> You'll see your result is aligned with others observation. 
> 
>  
> 
> If you control the compression/decompression, then there are ways to
> make libjpeg-turbo many times faster by design a multi-threading based
> wrapper code.
> 
> Then if you have 4 cores, you usually get at least 3x faster. But you'll
> need to come up with the design based on your specific case.
> 
>  
> 
> 2016-12-19 15:21 GMT-08:00 DRC <dco...@us...
> <mailto:dco...@us...>>:
> 
>     Nothing obvious stands out in your code.  I don't quite understand your
>     claims below that "libjpeg-turbo came out behind by 30%" but
>     "libjpeg-turbo seemed to be 40% faster while decoding images at the same
>     quality level".  Those seem to be contradictory claims.  Please clarify.
> 
>     You can get probably 10-15% better performance from the dev (1.6
>     evolving) builds, if you're using an AVX2-equipped CPU (I expect that to
>     be more like 25% when libjpeg-turbo 1.6 ships, but I'm seeking funding
>     to finish out the AVX2 acceleration project.)
> 
>     I'm not really surprised that Pegasus is faster.  They do use SIMD
>     instructions, and they also use multi-threading.  The latter is
>     something we cannot do within our library because of the libjpeg
>     architecture, although applications are of course free to implement
>     multi-threaded compression or decompression on their own.  Our initial
>     performance goal was to match the performance of the Intel Performance
>     Primitives, which we pretty much have done, but Pegasus has always been
>     a notch faster than IPP in my experience.
> 
>     Back in the day, I had a TurboJPEG API implementation for Pegasus, which
>     can be found here:
>     https://github.com/VirtualGL/virtualgl/tree/029f870488d251b48c3be0b17804e3002f7602cd/jpeg
> 
>     You should be able to build turbojpegp.c and jpgtest.cpp against Pegasus
>     and obtain benchmark data that closely resembles that of tjbench in
>     libjpeg-turbo.  That would be a good sanity check.
> 
>     On 12/19/16 2:56 PM, Siddharth Bidasaria wrote:
>     > Hi all,
>     >
>     >
>     >
>     > I am currently using a proprietary jpeg encoding library called Pegasus,
>     > and was looking to switch to libjpeg-turbo.
>     >
>     >
>     >
>     > I have been conducting some benchmarks between the two libraries and
>     > libjpeg-turbo came out behind by 30%. I was quite surprised by this
>     > because Pegasus was last updated 4 years ago, and I believe it doesn’t
>     > make use of SIMD instructions either. Moreover, libjpeg-turbo seemed to
>     > be 40% faster while decoding images at the same quality level.
>     >
>     >
>     >
>     > Hence, I wanted to rule out that I am incorrectly/inefficiently using
>     > the libjpeg-turbo api. If you could have a look at my compression code
>     > below, and let me know if there are any glaring things I
>     > should/shouldn’t be doing please let me know!

Re: [Libjpeg-turbo-devel] Optimizing JPEG compression speed

[Siddharth B. <sid...@ci...>]

I was also curious if you had any thoughts on why Libjpeg-Turbo might be faster than Pegasus for decoding, but not for encoding?


From: Siddharth Bidasaria [mailto:sid...@ci...]
Sent: Tuesday, December 20, 2016 11:43 AM
To: libjpeg-turbo Developers <lib...@li...>
Subject: Re: [Libjpeg-turbo-devel] Optimizing JPEG compression speed

Thank you for your responses, they are much appreciated!

I meant that Libjpeg-turbo was faster than Pegasus by 40% for decoding jpegs, but slower by 30% when used for encoding rgb images to jpegs.

By multi-threading, do you mean encoding multiple images in parallel, or splitting up 1 image into multiple parts and encoding them in parallel and rejoining them? If it’s the second case, could you point me to a an example implementation or resource that describes such a process. I tried looking for it myself, but found nothing so far.

Thanks,


From: Hao Hu [mailto:ih...@gm...]
Sent: Tuesday, December 20, 2016 12:04 AM
To: libjpeg-turbo Developers <lib...@li...<mailto:lib...@li...>>
Subject: Re: [Libjpeg-turbo-devel] Optimizing JPEG compression speed

If you search Pegasus on this link https://sourceforge.net/p/freeimage/discussion/36111/thread/fb51778d/
You'll see your result is aligned with others observation.

If you control the compression/decompression, then there are ways to make libjpeg-turbo many times faster by design a multi-threading based wrapper code.
Then if you have 4 cores, you usually get at least 3x faster. But you'll need to come up with the design based on your specific case.

2016-12-19 15:21 GMT-08:00 DRC <dco...@us...<mailto:dco...@us...>>:
Nothing obvious stands out in your code.  I don't quite understand your
claims below that "libjpeg-turbo came out behind by 30%" but
"libjpeg-turbo seemed to be 40% faster while decoding images at the same
quality level".  Those seem to be contradictory claims.  Please clarify.

You can get probably 10-15% better performance from the dev (1.6
evolving) builds, if you're using an AVX2-equipped CPU (I expect that to
be more like 25% when libjpeg-turbo 1.6 ships, but I'm seeking funding
to finish out the AVX2 acceleration project.)

I'm not really surprised that Pegasus is faster.  They do use SIMD
instructions, and they also use multi-threading.  The latter is
something we cannot do within our library because of the libjpeg
architecture, although applications are of course free to implement
multi-threaded compression or decompression on their own.  Our initial
performance goal was to match the performance of the Intel Performance
Primitives, which we pretty much have done, but Pegasus has always been
a notch faster than IPP in my experience.

Back in the day, I had a TurboJPEG API implementation for Pegasus, which
can be found here:
https://github.com/VirtualGL/virtualgl/tree/029f870488d251b48c3be0b17804e3002f7602cd/jpeg

You should be able to build turbojpegp.c and jpgtest.cpp against Pegasus
and obtain benchmark data that closely resembles that of tjbench in
libjpeg-turbo.  That would be a good sanity check.

On 12/19/16 2:56 PM, Siddharth Bidasaria wrote:
> Hi all,
>
>
>
> I am currently using a proprietary jpeg encoding library called Pegasus,
> and was looking to switch to libjpeg-turbo.
>
>
>
> I have been conducting some benchmarks between the two libraries and
> libjpeg-turbo came out behind by 30%. I was quite surprised by this
> because Pegasus was last updated 4 years ago, and I believe it doesn’t
> make use of SIMD instructions either. Moreover, libjpeg-turbo seemed to
> be 40% faster while decoding images at the same quality level.
>
>
>
> Hence, I wanted to rule out that I am incorrectly/inefficiently using
> the libjpeg-turbo api. If you could have a look at my compression code
> below, and let me know if there are any glaring things I
> should/shouldn’t be doing please let me know!

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Libjpeg-turbo-devel mailing list
Lib...@li...<mailto:Lib...@li...>
https://lists.sourceforge.net/lists/listinfo/libjpeg-turbo-devel

Re: [Libjpeg-turbo-devel] Optimizing JPEG compression speed

[Siddharth B. <sid...@ci...>]

Thank you for your responses, they are much appreciated!

I meant that Libjpeg-turbo was faster than Pegasus by 40% for decoding jpegs, but slower by 30% when used for encoding rgb images to jpegs.

By multi-threading, do you mean encoding multiple images in parallel, or splitting up 1 image into multiple parts and encoding them in parallel and rejoining them? If it’s the second case, could you point me to a an example implementation or resource that describes such a process. I tried looking for it myself, but found nothing so far.

Thanks,


From: Hao Hu [mailto:ih...@gm...]
Sent: Tuesday, December 20, 2016 12:04 AM
To: libjpeg-turbo Developers <lib...@li...>
Subject: Re: [Libjpeg-turbo-devel] Optimizing JPEG compression speed

If you search Pegasus on this link https://sourceforge.net/p/freeimage/discussion/36111/thread/fb51778d/
You'll see your result is aligned with others observation.

If you control the compression/decompression, then there are ways to make libjpeg-turbo many times faster by design a multi-threading based wrapper code.
Then if you have 4 cores, you usually get at least 3x faster. But you'll need to come up with the design based on your specific case.

2016-12-19 15:21 GMT-08:00 DRC <dco...@us...<mailto:dco...@us...>>:
Nothing obvious stands out in your code.  I don't quite understand your
claims below that "libjpeg-turbo came out behind by 30%" but
"libjpeg-turbo seemed to be 40% faster while decoding images at the same
quality level".  Those seem to be contradictory claims.  Please clarify.

You can get probably 10-15% better performance from the dev (1.6
evolving) builds, if you're using an AVX2-equipped CPU (I expect that to
be more like 25% when libjpeg-turbo 1.6 ships, but I'm seeking funding
to finish out the AVX2 acceleration project.)

I'm not really surprised that Pegasus is faster.  They do use SIMD
instructions, and they also use multi-threading.  The latter is
something we cannot do within our library because of the libjpeg
architecture, although applications are of course free to implement
multi-threaded compression or decompression on their own.  Our initial
performance goal was to match the performance of the Intel Performance
Primitives, which we pretty much have done, but Pegasus has always been
a notch faster than IPP in my experience.

Back in the day, I had a TurboJPEG API implementation for Pegasus, which
can be found here:
https://github.com/VirtualGL/virtualgl/tree/029f870488d251b48c3be0b17804e3002f7602cd/jpeg

You should be able to build turbojpegp.c and jpgtest.cpp against Pegasus
and obtain benchmark data that closely resembles that of tjbench in
libjpeg-turbo.  That would be a good sanity check.

On 12/19/16 2:56 PM, Siddharth Bidasaria wrote:
> Hi all,
>
>
>
> I am currently using a proprietary jpeg encoding library called Pegasus,
> and was looking to switch to libjpeg-turbo.
>
>
>
> I have been conducting some benchmarks between the two libraries and
> libjpeg-turbo came out behind by 30%. I was quite surprised by this
> because Pegasus was last updated 4 years ago, and I believe it doesn’t
> make use of SIMD instructions either. Moreover, libjpeg-turbo seemed to
> be 40% faster while decoding images at the same quality level.
>
>
>
> Hence, I wanted to rule out that I am incorrectly/inefficiently using
> the libjpeg-turbo api. If you could have a look at my compression code
> below, and let me know if there are any glaring things I
> should/shouldn’t be doing please let me know!

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/intel
_______________________________________________
Libjpeg-turbo-devel mailing list
Lib...@li...<mailto:Lib...@li...>
https://lists.sourceforge.net/lists/listinfo/libjpeg-turbo-devel

Re: [Libjpeg-turbo-devel] Optimizing JPEG compression speed

[Hao Hu <ih...@gm...>]

If you search Pegasus on this link
https://sourceforge.net/p/freeimage/discussion/36111/thread/fb51778d/
You'll see your result is aligned with others observation.

If you control the compression/decompression, then there are ways to make
libjpeg-turbo many times faster by design a multi-threading based wrapper
code.
Then if you have 4 cores, you usually get at least 3x faster. But you'll
need to come up with the design based on your specific case.

2016-12-19 15:21 GMT-08:00 DRC <dco...@us...>:

> Nothing obvious stands out in your code.  I don't quite understand your
> claims below that "libjpeg-turbo came out behind by 30%" but
> "libjpeg-turbo seemed to be 40% faster while decoding images at the same
> quality level".  Those seem to be contradictory claims.  Please clarify.
>
> You can get probably 10-15% better performance from the dev (1.6
> evolving) builds, if you're using an AVX2-equipped CPU (I expect that to
> be more like 25% when libjpeg-turbo 1.6 ships, but I'm seeking funding
> to finish out the AVX2 acceleration project.)
>
> I'm not really surprised that Pegasus is faster.  They do use SIMD
> instructions, and they also use multi-threading.  The latter is
> something we cannot do within our library because of the libjpeg
> architecture, although applications are of course free to implement
> multi-threaded compression or decompression on their own.  Our initial
> performance goal was to match the performance of the Intel Performance
> Primitives, which we pretty much have done, but Pegasus has always been
> a notch faster than IPP in my experience.
>
> Back in the day, I had a TurboJPEG API implementation for Pegasus, which
> can be found here:
> https://github.com/VirtualGL/virtualgl/tree/029f870488d251b48c3be0b17804e3
> 002f7602cd/jpeg
>
> You should be able to build turbojpegp.c and jpgtest.cpp against Pegasus
> and obtain benchmark data that closely resembles that of tjbench in
> libjpeg-turbo.  That would be a good sanity check.
>
> On 12/19/16 2:56 PM, Siddharth Bidasaria wrote:
> > Hi all,
> >
> >
> >
> > I am currently using a proprietary jpeg encoding library called Pegasus,
> > and was looking to switch to libjpeg-turbo.
> >
> >
> >
> > I have been conducting some benchmarks between the two libraries and
> > libjpeg-turbo came out behind by 30%. I was quite surprised by this
> > because Pegasus was last updated 4 years ago, and I believe it doesn’t
> > make use of SIMD instructions either. Moreover, libjpeg-turbo seemed to
> > be 40% faster while decoding images at the same quality level.
> >
> >
> >
> > Hence, I wanted to rule out that I am incorrectly/inefficiently using
> > the libjpeg-turbo api. If you could have a look at my compression code
> > below, and let me know if there are any glaring things I
> > should/shouldn’t be doing please let me know!
>
> ------------------------------------------------------------
> ------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today.http://sdm.link/intel
> _______________________________________________
> Libjpeg-turbo-devel mailing list
> Lib...@li...
> https://lists.sourceforge.net/lists/listinfo/libjpeg-turbo-devel
>

Re: [Libjpeg-turbo-devel] Optimizing JPEG compression speed

[DRC <dco...@us...>]

Nothing obvious stands out in your code.  I don't quite understand your
claims below that "libjpeg-turbo came out behind by 30%" but
"libjpeg-turbo seemed to be 40% faster while decoding images at the same
quality level".  Those seem to be contradictory claims.  Please clarify.

You can get probably 10-15% better performance from the dev (1.6
evolving) builds, if you're using an AVX2-equipped CPU (I expect that to
be more like 25% when libjpeg-turbo 1.6 ships, but I'm seeking funding
to finish out the AVX2 acceleration project.)

I'm not really surprised that Pegasus is faster.  They do use SIMD
instructions, and they also use multi-threading.  The latter is
something we cannot do within our library because of the libjpeg
architecture, although applications are of course free to implement
multi-threaded compression or decompression on their own.  Our initial
performance goal was to match the performance of the Intel Performance
Primitives, which we pretty much have done, but Pegasus has always been
a notch faster than IPP in my experience.

Back in the day, I had a TurboJPEG API implementation for Pegasus, which
can be found here:
https://github.com/VirtualGL/virtualgl/tree/029f870488d251b48c3be0b17804e3002f7602cd/jpeg

You should be able to build turbojpegp.c and jpgtest.cpp against Pegasus
and obtain benchmark data that closely resembles that of tjbench in
libjpeg-turbo.  That would be a good sanity check.

On 12/19/16 2:56 PM, Siddharth Bidasaria wrote:
> Hi all,
> 
>  
> 
> I am currently using a proprietary jpeg encoding library called Pegasus,
> and was looking to switch to libjpeg-turbo.
> 
>  
> 
> I have been conducting some benchmarks between the two libraries and
> libjpeg-turbo came out behind by 30%. I was quite surprised by this
> because Pegasus was last updated 4 years ago, and I believe it doesn’t
> make use of SIMD instructions either. Moreover, libjpeg-turbo seemed to
> be 40% faster while decoding images at the same quality level.
> 
>  
> 
> Hence, I wanted to rule out that I am incorrectly/inefficiently using
> the libjpeg-turbo api. If you could have a look at my compression code
> below, and let me know if there are any glaring things I
> should/shouldn’t be doing please let me know!

[Libjpeg-turbo-devel] Optimizing JPEG compression speed

[Siddharth B. <sid...@ci...>]

Hi all,

I am currently using a proprietary jpeg encoding library called Pegasus, and was looking to switch to libjpeg-turbo.

I have been conducting some benchmarks between the two libraries and libjpeg-turbo came out behind by 30%. I was quite surprised by this because Pegasus was last updated 4 years ago, and I believe it doesn't make use of SIMD instructions either. Moreover, libjpeg-turbo seemed to be 40% faster while decoding images at the same quality level.

Hence, I wanted to rule out that I am incorrectly/inefficiently using the libjpeg-turbo api. If you could have a look at my compression code below, and let me know if there are any glaring things I should/shouldn't be doing please let me know!

Thank you,

Sid


BOOL compressLibjpeg(PBITMAPINFOHEADER pBitmapInfo, PBYTE pBitmap, UINT32 pitch, PBYTE pOutput, UINT32 *outSize, int qualityLevel, BOOL subSampling)//pOutput is an already allocated buffer with size outSize
{
       struct jpeg_compress_struct cinfo;
       struct tw_err_mgr err;
       JSAMPROW row_pointer[1];
       int row_stride;

       cinfo.err = jpeg_std_error(&err.pub);
       err.pub.error_exit = tw_error_exit;

       if (setjmp(err.setjmp_buffer))
       {
              /* If we get here, the JPEG code has signaled an error. We need to clean up the JPEG object and return.*/
              jpeg_destroy_decompress(&cinfo);
              return FALSE;
       }

       jpeg_create_compress(&cinfo);

       jpeg_mem_dest(&cinfo, &pOutput, outSize);

       cinfo.image_width =  pBitmapInfo->biWidth;
       cinfo.image_height = pBitmapInfo->biHeight;
       cinfo.input_components = (pBitmapInfo->biBitCount)/8;
       cinfo.in_color_space = JCS_EXT_BGRX;
       jpeg_set_defaults(&cinfo);
       cinfo.write_JFIF_header = TRUE;
       cinfo.dct_method = JDCT_IFAST;


       if (subSampling == FALSE)
       {
              //disable subsampling
              cinfo.comp_info[0].v_samp_factor = 1;
              cinfo.comp_info[0].h_samp_factor = 1;
              cinfo.comp_info[1].h_samp_factor = 1;
              cinfo.comp_info[1].h_samp_factor = 1;
              cinfo.comp_info[2].h_samp_factor = 1;
              cinfo.comp_info[2].h_samp_factor = 1;
       }

       jpeg_set_quality(&cinfo, pegasusToLibjpegMap[qualityLevel-1], FALSE); //quality metric
       jpeg_start_compress(&cinfo, TRUE);

       row_stride = pitch;
       while (cinfo.next_scanline < cinfo.image_height)
       {
              row_pointer[0] = &(pBitmap[cinfo.next_scanline * row_stride]);
              (void)jpeg_write_scanlines(&cinfo, row_pointer, 1);
       }
       jpeg_finish_compress(&cinfo);
       jpeg_destroy_compress(&cinfo);
       return TRUE;
}

Re: [Libjpeg-turbo-devel] Inclusion of iccjpeg.c and header file in libjpeg-turbo

[DRC <dco...@us...>]

I took a quick look at the code, and if I understand correctly, this
would add three new functions to the libjpeg API:

  setup_read_icc_profile()
  read_icc_profile()
  write_icc_profile()

Is that correct?  If so, then I would be open to including that feature
in libjpeg-turbo 1.6, although I need the following changes before I can
consider integrating and adopting the feature:

- Prefix the function names with "jpeg_" to be consistent with the rest
of the libjpeg API
- If possible, make setup_read_icc_profile() automatic.  It should be
possible to store a state variable in the jpeg_decompress_master
structure to indicate whether the setup function has been run, then
check that variable in the body of jpeg_read_icc_profile().
- Merge the function declarations from iccjpeg.h into jpeglib.h.
- Merge the API documentation from iccjpeg.h into libjpeg.txt (you can
add a new section to the document to accommodate this.)
- Add an appropriate copyright and license header to iccjpeg.c.  Refer
to the other copyright/license headers in the libjpeg API code for examples.
- Add options to cjpeg and djpeg that allow those programs to test the
ICC profile feature.  This will allow the appropriate regression tests
to be added to the build system.

I can do all of the above for you, but unfortunately not for free.  :(

DRC


On 12/3/16 5:53 PM, Thomas Weber wrote:
> Hi, 
> I would like to discuss a possible inclusion of iccjpeg.{c,h} into
> libjpeg-turbo. In 2013, Paweł Hajdan, Jr. already opened a discussion[1]
> about the possibility of such an inclusion, but there was no final
> decision.
> I maintain lcms in Debian and have a feature request open to expose
> iccjpeg via lcms' API[2], such that Chromium (and probably Firefox and
> lcms) could link against it.
> Looking at the discussion at [1], I have the feeling that the old
> discussion focused a bit too much on the fact that iccjpeg is currently
> part of lcms. While this is true, it is not really an integral part of
> lcms and ended up there more by accident. Indeed, it was originally
> meant for inclusion in libjpeg[3]. In lcms, it is today only used to
> build an utility named jpgicc, an ICC profile applier for JPEG.
> 
> [1] https://sourceforge.net/p/libjpeg-turbo/mailman/message/30387709/
> (sorry for not replying to that thread, but I did not find a way to
> extract the message-id)
> [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=747839
> [3] https://github.com/mm2/Little-CMS/issues/37#issuecomment-66450180
> 
> 	Thomas

[Libjpeg-turbo-devel] Inclusion of iccjpeg.c and header file in libjpeg-turbo

[Thomas W. <tw...@de...>]

Hi, 
I would like to discuss a possible inclusion of iccjpeg.{c,h} into
libjpeg-turbo. In 2013, Paweł Hajdan, Jr. already opened a discussion[1]
about the possibility of such an inclusion, but there was no final
decision.
I maintain lcms in Debian and have a feature request open to expose
iccjpeg via lcms' API[2], such that Chromium (and probably Firefox and
lcms) could link against it.
Looking at the discussion at [1], I have the feeling that the old
discussion focused a bit too much on the fact that iccjpeg is currently
part of lcms. While this is true, it is not really an integral part of
lcms and ended up there more by accident. Indeed, it was originally
meant for inclusion in libjpeg[3]. In lcms, it is today only used to
build an utility named jpgicc, an ICC profile applier for JPEG.

[1] https://sourceforge.net/p/libjpeg-turbo/mailman/message/30387709/
(sorry for not replying to that thread, but I did not find a way to
extract the message-id)
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=747839
[3] https://github.com/mm2/Little-CMS/issues/37#issuecomment-66450180

	Thomas

[Libjpeg-turbo-devel] CMake build system

[DRC <dco...@us...>]

For those who haven't been following the discussion at

https://github.com/libjpeg-turbo/libjpeg-turbo/issues/56

the dev branch now contains a 100% CMake-based build system.  The
decision to eliminate autotools in libjpeg-turbo 1.6 was not one I took
lightly, but CMake's built-in support for assembly and Java code, as
well as other features that are generally more friendly to the needs of
our build and packaging system, solved a lot of problems for us and
eliminated a lot of hacks.  In addition, this eliminates the need to
maintain two separate build, packaging, and testing systems going
forward, which will reduce costs for new feature implementations.  I
spent something like 60 unpaid hours implementing this, but it will
probably save me much more time than that in the long run.

This will not land in libjpeg-turbo 1.6 beta for at least several
months, which should hopefully give downstream integrators enough time
to test it and make the necessary modifications to their distribution
builds.  Please take a moment to test the new build system on your
favorite platform and let me know what breaks.  The most common
cross-compilation builds (including iOS, Android, and MinGW) are
documented in BUILDING.md.

If your project or organization will benefit from this work, then please
consider making a donation using the "Donate" button at
http://www.libjpeg-turbo.org/.  I have spent hundreds of unpaid hours in
the past couple of months implementing this new build system and the
continuous integration system.  Since I don't draw a salary for my work
on open source code, donations and funded development opportunities are
the only way I can get paid for my work, and those sources of funding
are the only thing keeping libjpeg-turbo afloat as a project.

DRC

[Libjpeg-turbo-devel] Continuous integration is now online

[DRC <dco...@us...>]

We are now spinning continuous official builds from both the master
(stable) and dev (evolving) branches, using Travis and AppVeyor:

http://www.libjpeg-turbo.org/DeveloperInfo/PreReleases

The official Linux builds are spun using a docker image
(dcommander/buildljt:latest) that closely resembles my local build
environment (recipe: https://github.com/libjpeg-turbo/docker.)  The
official OS X and Windows builds use a Travis and AppVeyor config that
is also designed to mimic my local build environment.  These pre-release
builds should resemble, as closely as possible, the official release
builds, except that the pre-release builds will not be signed using my
code certificate or GPG key.  The reason behind this is both practical
and philosophical.  Practically, there is no easy way (of which I'm
aware) to sign the builds through a CI system without risking exposure
of the private keys.  Philosophically, a signed build means that I am
personally standing behind the integrity of the binaries, and I can only
do that if I built them on my local machines.  Thus, official releases
will continue to be spun locally.  This ensures that there is no "man in
the middle", i.e. the official releases will never exist on the network
in unsigned form.  However, with the exception of signing, the build
procedure used to generate both pre-release and release binaries is
identical and reproducible (see
http://www.libjpeg-turbo.org/DeveloperInfo/BuildInstructions).

Travis is also being used to continuously run ASAN and to validate
building the source with 12-bit and jpeg-8 support.  Future enhancements
will likely include regression testing the ARM, MIPS, and PPC code as well.

Special thanks to Matthieu, who created the proof of concept for
integrating libjpeg-turbo with Travis and AppVeyor.  My work is based
heavily on his.

Any contributions or suggestions are welcome.  I'm new to the CI game,
so I'm learning as I go along.  In particular, I am open to adding any
other useful regression tests that you can think of.  It would also be
nice at some point to figure out a better means of deploying the
pre-release Linux/OS X binaries than uploading them to our SourceForge
web page.  AppVeyor has a convenient mechanism for publishing
"artifacts" with each build, and I am taking advantage of that in order
to provide the Windows pre-release binaries through their site.
Unfortunately, however, doing the same with Travis would require a paid
AWS subscription.