Re: [Ziproxy-devel] ziproxy 3.1.x CPU load problem... patch to not do PNG recompress.

[Daniel M. C. <da...@gm...>]

Try the attached patch and see how Ziproxy behaves in your machine.


.dan

Em Monday 21 June 2010, Andrew Leahy escreveu:
> This particular ziproxy machine has an EPIA Eden ESP 600 "Samuel 2"
> (600MHz, 64Kb L2 Cache). Which is over 5 years old. It's on the lean-side,
> but it still grunts along, fan-less system & consuming only 7-10Watts.
>
> cpu MHz    : 601.471
> cache size : 64 KB
> flags        : fpu de tsc msr cx8 mtrr pge mmx 3dnow up
>
> I'll upgrade it eventually!
>
> I've implemented those DCT changes you suggested, not sure it's had much
> affect.
> Probably need to set up some test pages to compare/benchmark.
>
> Andrew
>
> On 21 June 2010 04:11, Daniel Mealha Cabrita <da...@gm...> wrote:
> > That's peculiar. I didn't realize that libpng could be so heavy.
> > AFAIR libpng has some kind of fuzzy analysis to guess which is the best
> > compression strategy -- perhaps that's the reason.
> >
> > I will collect some compression statistics later and see what's the best
> > to do
> > about.
> >
> >
> > Parallel to this, it seems that your CPU is one of the old models with a
> > rather slow FPU (how much L2 cache does it have?).
> > While keeping the changes you've made, could you try replacing the two
> > occurences of:
> > cinfo.dct_method = JDCT_FLOAT;
> > to
> > cinfo.dct_method = JDCT_ISLOW;
> > And see if it improves performance?
> >
> > Em Sunday 20 June 2010, Andrew Leahy escreveu:
> > > Dan, I've done a little digging to find why image 'decompression' (JP2
> > > to JPEG) has been slow since 3.1.x on my tiny EPIA box.
> > >
> > > Things were slow enough that the 90second timeout was sometimes being
> > > hit with a maximum of 4 simultaneous connections. CPU Load would sit
> > > above 4 (~25% for each ziproxy). The machine was CPU flatlining on
> > > image recompressing, no network I/O was happening while the image
> > > recompress was busy number crunching.
> > >
> > > The low-end EPIA has no SSE flags. Recompiling with -O3 made almost no
> > > difference. Limiting simultaneous connections to 1 or 2 actually made
> > > things a little better... as it made more CPU resources available for
> > > image recompress.
> > >
> > > Anyhow, the main culprit appears to be the dual recompression of JP2
> > > into JPEG and PNG in image.c
> > >
> > > This is the ziproxy debug log, with a little extra added to show some
> > > of the flags, before and after the 'decision logic' -
> > >
> > > [10036] Starting image decompression...
> > > [10036] Image parms (non-palette) -- w: 600, h: 900, bpp: 3
> > > [10036] Deciding image compression strategy...
> > > [10036] Init flags: has_transparency=0 try_lossless=1 try_lossy=1
> > > [10036] Final flags: has_transparency=0 try_lossless=1 try_lossy=1
> > > [10036] Strategy defined. Continuing...
> > > [10036] Attempting JPEG compression...
> > > [10036] Attempting PNG compression...
> > > [10036] Compression return codes -- JP2K:33 JPEG:0 PNG:0
> > > [10036] Image modification/compression took 21537422 us.
> > > [10036]   and returned 0.
> > >
> > > As you can see it took 21seconds to complete the recompress for a
> > > 600x900 JP2 image (this is with  MaxConnections=1)
> > >
> > > What I've done is added an extra bit of logic to image.c -
> > >
> > > If source is JP2, has no transparency, the target is JPEG, and lossy
> > > encode (PNG) is still enabled... then turn lossy off!
> > >
> > >
> > > This has returned my client ziproxy to it's normal speed (big images
> > > taking 4-5 seconds), increased MaxConnections to my "normal" 6.
> > >
> > >
> > > I don't think this is a solution for everybody, and you may have a
> > > better idea about when it is really appropriate to do a PNG
> > > recompress?
> > >
> > >
> > > image.c in compress_image()
> > >
> > > #ifdef JP2K
> > >         if ((source_type == IMG_JP2K) && (has_transparency == 0) &&
> > > (try_lossy == 0))
> > >                 try_lossless = 0;
> > >         if ((source_type == IMG_JP2K) && (has_transparency != 0) &&
> > > (target_lossy == IMG_JP2K))
> > >                 try_lossless = 0;
> > >         if ((source_type == IMG_JP2K) && (has_transparency == 0) &&
> > > (target_lossy = IMG_JPEG) && (try_lossy == 1)) /* Alf 20100620 */
> > >                 try_lossless = 0;
> > > #endif
> > >
> > >
> > >
> > > My debug output looks like this...
> > >
> > > [10070] Starting image decompression...
> > > [10070] Image parms (non-palette) -- w: 600, h: 900, bpp: 3
> > > [10070] Deciding image compression strategy...
> > > [10070] Init flags: has_transparency=0 try_lossless=1 try_lossy=1
> > > [10070] Final flags: has_transparency=0 try_lossless=0 try_lossy=1
> > > [10070] Strategy defined. Continuing...
> > > [10070] Attempting JPEG compression...
> > > [10070] Compression return codes -- JP2K:33 JPEG:0 PNG:33
> > > [10070] Image modification/compression took 3488304 us.
> > > [10070]   and returned 0.
> > >
> > > So from 21seconds to 3seconds.
> > >
> > > Cheers, Andrew.
> >
> > -------------------------------------------------------------------------
> >----- ThinkGeek and WIRED's GeekDad team up for the Ultimate
> > GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> > lucky parental unit.  See the prize list and enter to win:
> > http://p.sf.net/sfu/thinkgeek-promo
> > _______________________________________________
> > Ziproxy-devel mailing list
> > Zip...@li...
> > https://lists.sourceforge.net/lists/listinfo/ziproxy-devel