ziproxy-devel — discussion regarding development of ziproxy

[Ziproxy-devel] Ziproxy 3.3.2 released

[Daniel M. C. <da...@gm...>]

------------------------------------------------------------------------------
Ziproxy 3.3.2 released
2021-03-19

Only bugfixes related to compilation issues:

Corrected -fnocommon compilation issues in OpenBSD.
Added compatibility with giflib 5+ API changes.

------------------------------------------------------------------------------

[Ziproxy-devel] Ziproxy 3.3.1 released

[Daniel M. C. <da...@gm...>]

------------------------------------------------------------------------------
Ziproxy 3.3.1 released
2014-12-03

Fixed stdndup-related compilation error.
------------------------------------------------------------------------------

[Ziproxy-devel] patch to compile ziproxy with giflib5

[Alexander E. <al...@ar...>]

hi,
 some easy   patch to compile ziproxy with giflib 5

it's stolen from:
https://lists.gnu.org/archive/html/emacs-diffs/2013-10/msg00114.html

-alex

---- snip ----


diff --git a/src/image.c b/src/image.c
index 2dcf4fc..9601633 100644
--- a/src/image.c
+++ b/src/image.c
@@ -905,7 +905,12 @@ int gif2bitmap(char *inbuf, int insize, raw_bitmap
**out, long long int max_raw_
        desc.size=insize;
        desc.x.pos=0;
 
+#if GIFLIB_MAJOR < 5
        if ((GifFile = DGifOpen((void*)&desc, &gif_mem_input)) == NULL)
+#else
+        if ((GifFile = DGifOpen((void*)&desc, &gif_mem_input, NULL)) ==
NULL)
+#endif
+
                return( IMG_RET_ERR_UNKNOWN +
IMG_RET_FLG_WHILE_DECOMP);//more possible reasons
 
        bmp = new_raw_bitmap();


-alex

Re: [Ziproxy-devel] webp image conversion in ziproxy?

[Daniel M. C. <da...@gm...>]

There are no plans for WebP support.
I'm aware of those study, which has just PSNR stats known to be meaningless in 
practice. There are cases where WebP excels, but in average the 
quality/compression result is rather disappointing.

There are superior alternatives to WebP and JP2k, such as H.264 which is a 
patent minefield.


.dan

Em Sunday 06 January 2013, Andrew Leahy escreveu:
> Hi, any thought being put to implementing webp image conversion alongside
> JPEG2000 and PNG? There's native in-line support for WebP images in Chrome.
> 
> very rough compression comparison at
> https://developers.google.com/speed/webp/docs/c_study
> 
> Cheers, Andrew
> eResearch | UWS

[Ziproxy-devel] webp image conversion in ziproxy?

[Andrew L. <al...@gm...>]

Hi, any thought being put to implementing webp image conversion alongside
JPEG2000 and PNG? There's native in-line support for WebP images in Chrome.

very rough compression comparison at
https://developers.google.com/speed/webp/docs/c_study

Cheers, Andrew
eResearch | UWS

[Ziproxy-devel] Ziproxy 3.3.0 released

[Daniel M. C. <da...@gm...>]

------------------------------------------------------------------------------
Ziproxy 3.3.0 released
2013-01-06

Added option to remove alpha channel after a specified average opacity 
threshold.
New option: AlphaRemovalMinAvgOpacity
------------------------------------------------------------------------------

[Ziproxy-devel] Ziproxy 3.2.1 released

[Daniel M. C. <da...@gm...>]

------------------------------------------------------------------------------

Ziproxy 3.2.1 released
2012-02-08

This version fixes compilation problems with the newer libpng 1.5.x.

------------------------------------------------------------------------------

Re: [Ziproxy-devel] Fwd: [patch] ziproxy does not build with the newest libpng

[Florian B. <me...@th...>]

On Mon, Feb 6, 2012 at 4:15 PM, Cyril Russo <sta...@la...> wrote:
> Florian, it arrived.
Thanks.

I previously mentioned the issue on the -users ML, and Daniel (the
current maintainer) answered:

> Could you submit the patch to ziproxy-devel (if you're subscribed to) or
> directly to me please?
>
> I'll review and commit to the tree.

So this seems to be taken care of ;)

Florian

[Ziproxy-devel] Fwd: [patch] ziproxy does not build with the newest libpng

[Florian B. <me...@th...>]

Trying again as it doesn't seem to have arrived.


---------- Forwarded message ----------
From: Florian Bruhin <me...@th...>
Date: Mon, Feb 6, 2012 at 7:45 AM
Subject: [patch] ziproxy does not build with the newest libpng
To: zip...@li...


Hi,

since ziproxy does not build anymore in Archlinux since the libpng
version build, I made a patch so it builds correctly again.
Thanks a lot to Andrew Leahy for helping.

Florian

---- snip ----

diff -Nur ziproxy-3.2.0.old/src/image.c ziproxy-3.2.0/src/image.c
--- ziproxy-3.2.0.old/src/image.c       2012-02-06 07:02:05.330008313 +0100
+++ ziproxy-3.2.0/src/image.c   2012-02-06 07:02:56.493341202 +0100
@@ -68,6 +68,7 @@
 #undef GLOBAL

 #include <png.h>
+#include <zlib.h>

 #ifdef JP2K
 #include <jasper/jasper.h>
@@ -556,7 +557,7 @@
       bmp = new_raw_bitmap();
       *out = bmp;

-       png_set_read_fn (png_ptr, (voidp) &desc, mem_to_png);
+       png_set_read_fn (png_ptr, (void*) &desc, mem_to_png);

       png_read_info (png_ptr,info_ptr);
       png_get_IHDR (png_ptr, info_ptr, &width_png_uint_32, &height_png_uint_32,

[Ziproxy-devel] [patch] ziproxy does not build with the newest libpng

[Florian B. <me...@th...>]

Hi,

since ziproxy does not build anymore in Archlinux since the libpng
version build, I made a patch so it builds correctly again.
Thanks a lot to Andrew Leahy for helping.

Florian

---- snip ----

diff -Nur ziproxy-3.2.0.old/src/image.c ziproxy-3.2.0/src/image.c
--- ziproxy-3.2.0.old/src/image.c	2012-02-06 07:02:05.330008313 +0100
+++ ziproxy-3.2.0/src/image.c	2012-02-06 07:02:56.493341202 +0100
@@ -68,6 +68,7 @@
 #undef GLOBAL

 #include <png.h>
+#include <zlib.h>

 #ifdef JP2K
 #include <jasper/jasper.h>
@@ -556,7 +557,7 @@
 	bmp = new_raw_bitmap();
 	*out = bmp;

-	png_set_read_fn (png_ptr, (voidp) &desc, mem_to_png);
+	png_set_read_fn (png_ptr, (void*) &desc, mem_to_png);
 	
 	png_read_info (png_ptr,info_ptr);
 	png_get_IHDR (png_ptr, info_ptr, &width_png_uint_32, &height_png_uint_32,

[Ziproxy-devel] Adding Media content saving to Ziproxy

[Cyril R. <sta...@la...>]

Hi all,

   Ziproxy is very great already for saving bandwidth and removing most 
of the crap out of the ad companies.
However, there is still two points that would worth having in Ziproxy.

1) Content caching
When you go to a streaming website like grooveshark, deezer, jiwa, 
pandora, ..., you'll usually download the media stream each time you're 
listening to it.
I know about Squid proxy, but it doesn't really solve this, since Squid 
is not able to cache dynamic content.
So I've implemented my own solution (albeit, part of the solution), see 
the attached patch. You'll need to set up a directory in the 
ziproxy.conf file for the new option called "CachingDir"
It does save the streamed content, but I haven't figure out yet how to 
"identify & replay" the content. I think a kind of quick fingerprinting 
to identify the content would be required, and it's probably feasible 
with a MD5 or SHA1 of the first few KB of data. Then one need to use a 
transcode system to downsample/degrade the content size (like it's done 
for images) and stream that instead.

So it's still WIP but functional for the first part.


2) Adblocking is quite limited.
The adblocking system will crawl to death if used with a decent list, 
because it's doing O(N) searching in the text.
You might be interested in this page, to see how AdblockPlus did it and 
used a O(log N) algorithm.
http://adblockplus.org/blog/investigating-filter-matching-algorithms

I would love to be able to use Adblock filter list directly.

Let me know what you think.
Best regards,
Cyril

[Ziproxy-devel] Google SPDY protocol - of interest to Ziproxy?

[Andrew L. <al...@gm...>]

http://www.chromium.org/spdy

As part of the "Let's make the web faster"
<http://code.google.com/speed/> initiative,
we are experimenting with alternative protocols to help reduce the latency
of web pages. One of these experiments is SPDY (pronounced "SPeeDY"), an
application-layer protocol for transporting content over the web, designed
specifically for minimal latency.  In addition to a specification of the
protocol, we have developed a SPDY-enabled Google Chrome browser and
open-source web server. In lab tests, we have compared the performance of
these applications over HTTP and SPDY, and have observed up to 64%
reductions in page load times in SPDY. We hope to engage the open source
community to contribute ideas, feedback, code, and test results, to make
SPDY the next-generation application protocol for a faster web.
...
SPDY Proxy One potential use for SPDY is to provide fast, secure access to a
proxy.  For instance, if we had a forward proxy which took SPDY requests
from the client and could issue HTTP requests to the web, then we could
leverage the benefits of SPDY over the Client->Proxy link without needing to
upgrade the entire web.  Some environments where this could be particularly
useful include:

   - Mobile networks, where RTTs are very high
   - Public networks, where local WiFi or other protocols may be insecure,
   but SPDY connections to a secure proxy could add a layer of protection.

...


-- 
"Those who know, do not speak. Those who speak, do not know." (Lao Tzu)

My household GHG emissions from energy & transport ~4kg CO2e/day or
~1.4tonnes/year
Household daily use of Water 110L, Electricity 3.9kWh, Petrol 1.2L, Gas 0MJ

Re: [Ziproxy-devel] WebP for image compression?

[Daniel M. C. <da...@gm...>]

I've considered adding WebP support, but the real-world results are rather 
disappointing. 

I suggest you to read the following:
http://x264dev.multimedia.cx/archives/541

Still, I've decided to check that myself.
Months ago I've made some tests with JP2k (with proper tuning, which was kept 
the same for all test images) vs WebP at similar and very low bitrates.. and 
the results - as I remember - were:
in the worst cases (for JP2k) JP2k was slightly worse than WebP in certain 
parts/aspects of the pictures and slightly better in other parts/aspects.

About browser support, that's convenient that Chrome supports WebP (though 
there are/were JP2k plug-ins for Firefox), but that doesn't solve the problem 
when you provide single-link compression for multiple clients behind a 
shared, caching, proxy (what if someone is not using Chrome, and the cache 
has WebP data?).


It seems that recently a new, improved, WebP encoder was released... I may 
check that later, so this possibility is not dead yet.


.dan

Em Sunday 24 April 2011, Andrew Leahy escreveu:
> Hi - is Google's WebP image compression worth looking at in addition to
> JPEG-2000 for Ziproxy?
>
>  http://code.google.com/speed/webp/index.html
>
> At this point in time it's probably got a better chance of getting browser
> support than  JP2! :(
>
> Cheers, Andrew

[Ziproxy-devel] WebP for image compression?

[Andrew L. <al...@gm...>]

Hi - is Google's WebP image compression worth looking at in addition to
JPEG-2000 for Ziproxy?

 http://code.google.com/speed/webp/index.html

At this point in time it's probably got a better chance of getting browser
support than  JP2! :(

Cheers, Andrew
-- 
"Those who know, do not speak. Those who speak, do not know." (Lao Tzu)

My household GHG emissions from energy & transport ~4kg CO2e/day or
~1.4tonnes/year
Household daily use of Water 110L, Electricity 3.9kWh, Petrol 1.2L, Gas 0MJ

Re: [Ziproxy-devel] confirming ConnTimeout is functioning okay?

[Daniel M. C. <da...@gm...>]

I'm transferring this discussion to ziproxy-users.

Ziproxy should not timeout (by itself) before ConnTimeout elapses, but a 
remote HTTP server may do that and close its side of the connection and force 
Ziproxy to abort.

It would be interesting to see some related logs.
Is there a pattern for this problem you're aware of?


Em Tuesday 08 February 2011, Andrew Leahy escreveu:
> Hi - I'm getting a sense that ConnTimeout might be setting a
> hard-upper limit on the amount of time a connection can take,
> regardless of whether there is traffic on that connection or not.
> Rather than "ConnTimeout" seconds since idle/stall.
>
> The description ConnTimeout in the ziproxy.conf is..
>
> ## If processing of request exceeds specified time in seconds,
> ## or connection is idle beyond that time (stalled) it will abort.
> ## This avoids processes staying forever (or for a very long time)
> ## in case of a stalled connection or software bug.
> ## This will NOT necessarily abort the streaming of very big files,
> ## it will ONLY if the connection stalls or there's a software bug.
>
> On my system it seems to have problems determing if the connection is
> idle/stalled
>
> I use a 3G modem which sometimes runs down to GPRS (60kbps) type
> speeds, with a pair of ziproxy's (one at each end).
>
> What I've noticed in the local ziproxy logs is repeated serial
> requests all with time/duration at around ConnTimeout limit.
>
> It I tcpdump the outbound eth interface I can see traffic coming in
> the interface. But then I can see a connection reset, and the download
> appears to start again. This matches with a "PZ" in the ziproxy.log,
> and repeated attempts to download some content.
>
> I can send some logs of this. But I just wanted to raise it in case it
> was something other people had experienced?
>
> If you're on a fast link and not watching the logs you probably wouldn't
> notice. On slow links the browser may eventually get's the file (after
> repeated attempts) or a "blank" page with no content.
>
> With chained ziproxies is the ConnTimeout applied to both the upstream
> download (typically fast), as well as the downstream 'send' to the
> other ziproxy (typically slow link).
>
>
> Cheers,
> Andrew

[Ziproxy-devel] confirming ConnTimeout is functioning okay?

[Andrew L. <al...@gm...>]

Hi - I'm getting a sense that ConnTimeout might be setting a
hard-upper limit on the amount of time a connection can take,
regardless of whether there is traffic on that connection or not.
Rather than "ConnTimeout" seconds since idle/stall.

The description ConnTimeout in the ziproxy.conf is..

## If processing of request exceeds specified time in seconds,
## or connection is idle beyond that time (stalled) it will abort.
## This avoids processes staying forever (or for a very long time)
## in case of a stalled connection or software bug.
## This will NOT necessarily abort the streaming of very big files,
## it will ONLY if the connection stalls or there's a software bug.

On my system it seems to have problems determing if the connection is
idle/stalled

I use a 3G modem which sometimes runs down to GPRS (60kbps) type
speeds, with a pair of ziproxy's (one at each end).

What I've noticed in the local ziproxy logs is repeated serial
requests all with time/duration at around ConnTimeout limit.

It I tcpdump the outbound eth interface I can see traffic coming in
the interface. But then I can see a connection reset, and the download
appears to start again. This matches with a "PZ" in the ziproxy.log,
and repeated attempts to download some content.

I can send some logs of this. But I just wanted to raise it in case it
was something other people had experienced?

If you're on a fast link and not watching the logs you probably wouldn't notice.
On slow links the browser may eventually get's the file (after
repeated attempts) or a "blank" page with no content.

With chained ziproxies is the ConnTimeout applied to both the upstream
download (typically fast), as well as the downstream 'send' to the
other ziproxy (typically slow link).


Cheers,
Andrew

-- 
"Those who know, do not speak. Those who speak, do not know." (Lao Tzu)

My household GHG emissions from energy & transport ~4kg CO2e/day or
~1.4tonnes/year
Household daily use of Water 110L, Electricity 3.9kWh, Petrol 1.2L, Gas 0MJ

[Ziproxy-devel] Ziproxy 3.2.0 released

[Daniel M. C. <da...@gm...>]

--------------------------------------------------------------------------------
Ziproxy 3.2.0 released

2010-09-07

This version brings SASL support, licensing update to reflect the new 
licensing terms of legacy code (in practice GPL restrictions prevail, though) 
and some polishing.

New features:

    * Changed HTTP authentication code to a modular one.
    * Added SASL support for HTTP authentication.

Bugfixes/changes:

    * Fixed Nameservers-related code which prevented compilation in certain 
OSes/arch compinations (ex.: glibc+gcc+ARM).
    * Nameservers support may be enable/disabled at compilation time.
--------------------------------------------------------------------------------

[Ziproxy-devel] Legacy Ziproxy 1.3d_r2 release

[Daniel M. C. <da...@gm...>]

------------------------------------------------------------------
Legacy Ziproxy 1.3d_r2 release

2010-08-21

An old version of Ziproxy, version 1.3d, was released as 1.3d_r2 under a 
2-clause BSD license and is available for anyone interested (check the 
download section).
Ziproxy 1.3d was the last version developed by the former maintainer, Juraj 
Variny, who agreed to make his code available under this new licensing terms.

The pertinent public announcement e-mail follows:

[Ziproxy-users] Ziproxy license change
From: Juraj Variny <xxxxxx@xxxxxxxxx>
To: zip...@li...
Date: 21-08-2010 06:38:25
   
Hello ziproxy users and developers,

following announcement is result of agreement between me and Daniel
Mealha Cabrita and it means that my code in ziproxy is now dual
licensed under both GPL and NetBSD licenses, for everyone. If you
don't know what it means, it will not affect you.

I hereby testify that ziproxy-1.3d, released at 2004-09-30, contains
only source code and documentation written by me and source code
previously in public domain, without exception. I am re-releasing
ziproxy-1.3d under NetBSD 2-clause BSD license, under these terms:

 * Copyright (c) Juraj Variny, 2004
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 * 1. Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 * 2. Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
 * THIS SOFTWARE IS PROVIDED BY JURAJ VARINY
 * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
 * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
 * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
 * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
 * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
 * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
 * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 * POSSIBILITY OF SUCH DAMAGE.

Best regards,

Juraj Variny
------------------------------------------------------------------

[Ziproxy-devel] Ziproxy license change

[Juraj V. <ri...@gm...>]

Hello ziproxy users and developers,

following announcement is result of agreement between me and Daniel
Mealha Cabrita and it means that my code in ziproxy is now dual
licensed under both GPL and NetBSD licenses, for everyone. If you
don't know what it means, it will not affect you.

I hereby testify that ziproxy-1.3d, released at 2004-09-30, contains
only source code and documentation written by me and source code
previously in public domain, without exception. I am re-releasing
ziproxy-1.3d under NetBSD 2-clause BSD license, under these terms:

 * Copyright (c) Juraj Variny, 2004
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 * 1. Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 * 2. Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
 * THIS SOFTWARE IS PROVIDED BY JURAJ VARINY
 * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
 * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
 * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
 * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
 * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
 * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
 * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 * POSSIBILITY OF SUCH DAMAGE.

Best regards,

Juraj Variny

[Ziproxy-devel] Nameservers option

[Daniel M. C. <da...@gm...>]

hi there,


I there anyone using the "Nameservers" option?


At some point the related code stopped working with GLIBC. In practice that 
means it's not functional under Linux.
Not really a surprise, considering that the code is not exactly 
POSIX-compliant.
Apparently that option is very seldom used, since I've got no complaints until 
now.

More recently the code started to cause compilation problems under 
GCC+GLIBC+ARM combination.

Unfortunately I'm unaware of a way to reimplement that in a portable way.
If anyone here knows how to, this is the perfect time.

Also, if that option does not work in other OSes (like FreeBSD, Solaris etc) 
we have useless code here and chances are it will be disabled to avoid 
problems.


.dan

[Ziproxy-devel] Ziproxy 3.1.3 released

[Daniel M. C. <da...@gm...>]

--------------------------------------------------------------------------------
Ziproxy 3.1.3 released.

Bugfixes only:

    * Fixed non-POSIX behavior which brought problems with eglibc.
    * Fixed Safari HTTP authentication problem.
--------------------------------------------------------------------------------

[Ziproxy-devel] Ziproxy 3.1.2 released

[Daniel M. C. <da...@gm...>]

--------------------------------------------------------------------------------
Ziproxy 3.1.2 released.

Minor bugfixes and improvements:

    * Ziproxy avoids unnecessary image processing now, thus saving CPU.
    * Fixed daemonization code: no more stdin/stdout kludges.
--------------------------------------------------------------------------------

Re: [Ziproxy-devel] ziproxy 3.1.x CPU load problem... patch to not do PNG recompress.

[Andrew L. <al...@gm...>]

yup i see what you're doing.
the patch is working okay on my machine, though i've only just tested on a
few sites. but seems good.
i'll let you know if things go weird.

andrew


On 23 June 2010 01:05:11 UTC+10, Daniel Mealha Cabrita <da...@gm...>wrote:

>
> Try the attached patch and see how Ziproxy behaves in your machine.
>
>
> .dan
>
> Em Monday 21 June 2010, Andrew Leahy escreveu:
> > This particular ziproxy machine has an EPIA Eden ESP 600 "Samuel 2"
> > (600MHz, 64Kb L2 Cache). Which is over 5 years old. It's on the
> lean-side,
> > but it still grunts along, fan-less system & consuming only 7-10Watts.
> >
> > cpu MHz    : 601.471
> > cache size : 64 KB
> > flags        : fpu de tsc msr cx8 mtrr pge mmx 3dnow up
> >
> > I'll upgrade it eventually!
> >
> > I've implemented those DCT changes you suggested, not sure it's had much
> > affect.
> > Probably need to set up some test pages to compare/benchmark.
> >
> > Andrew
> >
> > On 21 June 2010 04:11, Daniel Mealha Cabrita <da...@gm...> wrote:
> > > That's peculiar. I didn't realize that libpng could be so heavy.
> > > AFAIR libpng has some kind of fuzzy analysis to guess which is the best
> > > compression strategy -- perhaps that's the reason.
> > >
> > > I will collect some compression statistics later and see what's the
> best
> > > to do
> > > about.
> > >
> > >
> > > Parallel to this, it seems that your CPU is one of the old models with
> a
> > > rather slow FPU (how much L2 cache does it have?).
> > > While keeping the changes you've made, could you try replacing the two
> > > occurences of:
> > > cinfo.dct_method = JDCT_FLOAT;
> > > to
> > > cinfo.dct_method = JDCT_ISLOW;
> > > And see if it improves performance?
> > >
> > > Em Sunday 20 June 2010, Andrew Leahy escreveu:
> > > > Dan, I've done a little digging to find why image 'decompression'
> (JP2
> > > > to JPEG) has been slow since 3.1.x on my tiny EPIA box.
> > > >
> > > > Things were slow enough that the 90second timeout was sometimes being
> > > > hit with a maximum of 4 simultaneous connections. CPU Load would sit
> > > > above 4 (~25% for each ziproxy). The machine was CPU flatlining on
> > > > image recompressing, no network I/O was happening while the image
> > > > recompress was busy number crunching.
> > > >
> > > > The low-end EPIA has no SSE flags. Recompiling with -O3 made almost
> no
> > > > difference. Limiting simultaneous connections to 1 or 2 actually made
> > > > things a little better... as it made more CPU resources available for
> > > > image recompress.
> > > >
> > > > Anyhow, the main culprit appears to be the dual recompression of JP2
> > > > into JPEG and PNG in image.c
> > > >
> > > > This is the ziproxy debug log, with a little extra added to show some
> > > > of the flags, before and after the 'decision logic' -
> > > >
> > > > [10036] Starting image decompression...
> > > > [10036] Image parms (non-palette) -- w: 600, h: 900, bpp: 3
> > > > [10036] Deciding image compression strategy...
> > > > [10036] Init flags: has_transparency=0 try_lossless=1 try_lossy=1
> > > > [10036] Final flags: has_transparency=0 try_lossless=1 try_lossy=1
> > > > [10036] Strategy defined. Continuing...
> > > > [10036] Attempting JPEG compression...
> > > > [10036] Attempting PNG compression...
> > > > [10036] Compression return codes -- JP2K:33 JPEG:0 PNG:0
> > > > [10036] Image modification/compression took 21537422 us.
> > > > [10036]   and returned 0.
> > > >
> > > > As you can see it took 21seconds to complete the recompress for a
> > > > 600x900 JP2 image (this is with  MaxConnections=1)
> > > >
> > > > What I've done is added an extra bit of logic to image.c -
> > > >
> > > > If source is JP2, has no transparency, the target is JPEG, and lossy
> > > > encode (PNG) is still enabled... then turn lossy off!
> > > >
> > > >
> > > > This has returned my client ziproxy to it's normal speed (big images
> > > > taking 4-5 seconds), increased MaxConnections to my "normal" 6.
> > > >
> > > >
> > > > I don't think this is a solution for everybody, and you may have a
> > > > better idea about when it is really appropriate to do a PNG
> > > > recompress?
> > > >
> > > >
> > > > image.c in compress_image()
> > > >
> > > > #ifdef JP2K
> > > >         if ((source_type == IMG_JP2K) && (has_transparency == 0) &&
> > > > (try_lossy == 0))
> > > >                 try_lossless = 0;
> > > >         if ((source_type == IMG_JP2K) && (has_transparency != 0) &&
> > > > (target_lossy == IMG_JP2K))
> > > >                 try_lossless = 0;
> > > >         if ((source_type == IMG_JP2K) && (has_transparency == 0) &&
> > > > (target_lossy = IMG_JPEG) && (try_lossy == 1)) /* Alf 20100620 */
> > > >                 try_lossless = 0;
> > > > #endif
> > > >
> > > >
> > > >
> > > > My debug output looks like this...
> > > >
> > > > [10070] Starting image decompression...
> > > > [10070] Image parms (non-palette) -- w: 600, h: 900, bpp: 3
> > > > [10070] Deciding image compression strategy...
> > > > [10070] Init flags: has_transparency=0 try_lossless=1 try_lossy=1
> > > > [10070] Final flags: has_transparency=0 try_lossless=0 try_lossy=1
> > > > [10070] Strategy defined. Continuing...
> > > > [10070] Attempting JPEG compression...
> > > > [10070] Compression return codes -- JP2K:33 JPEG:0 PNG:33
> > > > [10070] Image modification/compression took 3488304 us.
> > > > [10070]   and returned 0.
> > > >
> > > > So from 21seconds to 3seconds.
> > > >
> > > > Cheers, Andrew.
> > >
> > >
> -------------------------------------------------------------------------
> > >----- ThinkGeek and WIRED's GeekDad team up for the Ultimate
> > > GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> > > lucky parental unit.  See the prize list and enter to win:
> > > http://p.sf.net/sfu/thinkgeek-promo
> > > _______________________________________________
> > > Ziproxy-devel mailing list
> > > Zip...@li...
> > > https://lists.sourceforge.net/lists/listinfo/ziproxy-devel
>
>
>
>
> ------------------------------------------------------------------------------
> ThinkGeek and WIRED's GeekDad team up for the Ultimate
> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> lucky parental unit.  See the prize list and enter to win:
> http://p.sf.net/sfu/thinkgeek-promo
> _______________________________________________
> Ziproxy-devel mailing list
> Zip...@li...
> https://lists.sourceforge.net/lists/listinfo/ziproxy-devel
>
>


-- 
"The simplest way to deal with a need or want is to stop needing or wanting
it." (John Michael Greer)

My household GHG emissions from energy & transport ~4kg CO2e/day or
~1.4tonnes/year
Household daily use of Water 120L, Electricity 4.0kWh, Petrol 1.2L, Gas 0MJ

Re: [Ziproxy-devel] ziproxy 3.1.x CPU load problem... patch to not do PNG recompress.

[Daniel M. C. <da...@gm...>]

Try the attached patch and see how Ziproxy behaves in your machine.


.dan

Em Monday 21 June 2010, Andrew Leahy escreveu:
> This particular ziproxy machine has an EPIA Eden ESP 600 "Samuel 2"
> (600MHz, 64Kb L2 Cache). Which is over 5 years old. It's on the lean-side,
> but it still grunts along, fan-less system & consuming only 7-10Watts.
>
> cpu MHz    : 601.471
> cache size : 64 KB
> flags        : fpu de tsc msr cx8 mtrr pge mmx 3dnow up
>
> I'll upgrade it eventually!
>
> I've implemented those DCT changes you suggested, not sure it's had much
> affect.
> Probably need to set up some test pages to compare/benchmark.
>
> Andrew
>
> On 21 June 2010 04:11, Daniel Mealha Cabrita <da...@gm...> wrote:
> > That's peculiar. I didn't realize that libpng could be so heavy.
> > AFAIR libpng has some kind of fuzzy analysis to guess which is the best
> > compression strategy -- perhaps that's the reason.
> >
> > I will collect some compression statistics later and see what's the best
> > to do
> > about.
> >
> >
> > Parallel to this, it seems that your CPU is one of the old models with a
> > rather slow FPU (how much L2 cache does it have?).
> > While keeping the changes you've made, could you try replacing the two
> > occurences of:
> > cinfo.dct_method = JDCT_FLOAT;
> > to
> > cinfo.dct_method = JDCT_ISLOW;
> > And see if it improves performance?
> >
> > Em Sunday 20 June 2010, Andrew Leahy escreveu:
> > > Dan, I've done a little digging to find why image 'decompression' (JP2
> > > to JPEG) has been slow since 3.1.x on my tiny EPIA box.
> > >
> > > Things were slow enough that the 90second timeout was sometimes being
> > > hit with a maximum of 4 simultaneous connections. CPU Load would sit
> > > above 4 (~25% for each ziproxy). The machine was CPU flatlining on
> > > image recompressing, no network I/O was happening while the image
> > > recompress was busy number crunching.
> > >
> > > The low-end EPIA has no SSE flags. Recompiling with -O3 made almost no
> > > difference. Limiting simultaneous connections to 1 or 2 actually made
> > > things a little better... as it made more CPU resources available for
> > > image recompress.
> > >
> > > Anyhow, the main culprit appears to be the dual recompression of JP2
> > > into JPEG and PNG in image.c
> > >
> > > This is the ziproxy debug log, with a little extra added to show some
> > > of the flags, before and after the 'decision logic' -
> > >
> > > [10036] Starting image decompression...
> > > [10036] Image parms (non-palette) -- w: 600, h: 900, bpp: 3
> > > [10036] Deciding image compression strategy...
> > > [10036] Init flags: has_transparency=0 try_lossless=1 try_lossy=1
> > > [10036] Final flags: has_transparency=0 try_lossless=1 try_lossy=1
> > > [10036] Strategy defined. Continuing...
> > > [10036] Attempting JPEG compression...
> > > [10036] Attempting PNG compression...
> > > [10036] Compression return codes -- JP2K:33 JPEG:0 PNG:0
> > > [10036] Image modification/compression took 21537422 us.
> > > [10036]   and returned 0.
> > >
> > > As you can see it took 21seconds to complete the recompress for a
> > > 600x900 JP2 image (this is with  MaxConnections=1)
> > >
> > > What I've done is added an extra bit of logic to image.c -
> > >
> > > If source is JP2, has no transparency, the target is JPEG, and lossy
> > > encode (PNG) is still enabled... then turn lossy off!
> > >
> > >
> > > This has returned my client ziproxy to it's normal speed (big images
> > > taking 4-5 seconds), increased MaxConnections to my "normal" 6.
> > >
> > >
> > > I don't think this is a solution for everybody, and you may have a
> > > better idea about when it is really appropriate to do a PNG
> > > recompress?
> > >
> > >
> > > image.c in compress_image()
> > >
> > > #ifdef JP2K
> > >         if ((source_type == IMG_JP2K) && (has_transparency == 0) &&
> > > (try_lossy == 0))
> > >                 try_lossless = 0;
> > >         if ((source_type == IMG_JP2K) && (has_transparency != 0) &&
> > > (target_lossy == IMG_JP2K))
> > >                 try_lossless = 0;
> > >         if ((source_type == IMG_JP2K) && (has_transparency == 0) &&
> > > (target_lossy = IMG_JPEG) && (try_lossy == 1)) /* Alf 20100620 */
> > >                 try_lossless = 0;
> > > #endif
> > >
> > >
> > >
> > > My debug output looks like this...
> > >
> > > [10070] Starting image decompression...
> > > [10070] Image parms (non-palette) -- w: 600, h: 900, bpp: 3
> > > [10070] Deciding image compression strategy...
> > > [10070] Init flags: has_transparency=0 try_lossless=1 try_lossy=1
> > > [10070] Final flags: has_transparency=0 try_lossless=0 try_lossy=1
> > > [10070] Strategy defined. Continuing...
> > > [10070] Attempting JPEG compression...
> > > [10070] Compression return codes -- JP2K:33 JPEG:0 PNG:33
> > > [10070] Image modification/compression took 3488304 us.
> > > [10070]   and returned 0.
> > >
> > > So from 21seconds to 3seconds.
> > >
> > > Cheers, Andrew.
> >
> > -------------------------------------------------------------------------
> >----- ThinkGeek and WIRED's GeekDad team up for the Ultimate
> > GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> > lucky parental unit.  See the prize list and enter to win:
> > http://p.sf.net/sfu/thinkgeek-promo
> > _______________________________________________
> > Ziproxy-devel mailing list
> > Zip...@li...
> > https://lists.sourceforge.net/lists/listinfo/ziproxy-devel

Re: [Ziproxy-devel] ziproxy 3.1.x CPU load problem... patch to not do PNG recompress.

[Andrew L. <al...@gm...>]

This particular ziproxy machine has an EPIA Eden ESP 600 "Samuel 2" (600MHz,
64Kb L2 Cache). Which is over 5 years old. It's on the lean-side, but it
still grunts along, fan-less system & consuming only 7-10Watts.

cpu MHz    : 601.471
cache size : 64 KB
flags        : fpu de tsc msr cx8 mtrr pge mmx 3dnow up

I'll upgrade it eventually!

I've implemented those DCT changes you suggested, not sure it's had much
affect.
Probably need to set up some test pages to compare/benchmark.

Andrew


On 21 June 2010 04:11, Daniel Mealha Cabrita <da...@gm...> wrote:

>
> That's peculiar. I didn't realize that libpng could be so heavy.
> AFAIR libpng has some kind of fuzzy analysis to guess which is the best
> compression strategy -- perhaps that's the reason.
>
> I will collect some compression statistics later and see what's the best to
> do
> about.
>
>
> Parallel to this, it seems that your CPU is one of the old models with a
> rather slow FPU (how much L2 cache does it have?).
> While keeping the changes you've made, could you try replacing the two
> occurences of:
> cinfo.dct_method = JDCT_FLOAT;
> to
> cinfo.dct_method = JDCT_ISLOW;
> And see if it improves performance?
>
>
> Em Sunday 20 June 2010, Andrew Leahy escreveu:
> > Dan, I've done a little digging to find why image 'decompression' (JP2
> > to JPEG) has been slow since 3.1.x on my tiny EPIA box.
> >
> > Things were slow enough that the 90second timeout was sometimes being
> > hit with a maximum of 4 simultaneous connections. CPU Load would sit
> > above 4 (~25% for each ziproxy). The machine was CPU flatlining on
> > image recompressing, no network I/O was happening while the image
> > recompress was busy number crunching.
> >
> > The low-end EPIA has no SSE flags. Recompiling with -O3 made almost no
> > difference. Limiting simultaneous connections to 1 or 2 actually made
> > things a little better... as it made more CPU resources available for
> > image recompress.
> >
> > Anyhow, the main culprit appears to be the dual recompression of JP2
> > into JPEG and PNG in image.c
> >
> > This is the ziproxy debug log, with a little extra added to show some
> > of the flags, before and after the 'decision logic' -
> >
> > [10036] Starting image decompression...
> > [10036] Image parms (non-palette) -- w: 600, h: 900, bpp: 3
> > [10036] Deciding image compression strategy...
> > [10036] Init flags: has_transparency=0 try_lossless=1 try_lossy=1
> > [10036] Final flags: has_transparency=0 try_lossless=1 try_lossy=1
> > [10036] Strategy defined. Continuing...
> > [10036] Attempting JPEG compression...
> > [10036] Attempting PNG compression...
> > [10036] Compression return codes -- JP2K:33 JPEG:0 PNG:0
> > [10036] Image modification/compression took 21537422 us.
> > [10036]   and returned 0.
> >
> > As you can see it took 21seconds to complete the recompress for a
> > 600x900 JP2 image (this is with  MaxConnections=1)
> >
> > What I've done is added an extra bit of logic to image.c -
> >
> > If source is JP2, has no transparency, the target is JPEG, and lossy
> > encode (PNG) is still enabled... then turn lossy off!
> >
> >
> > This has returned my client ziproxy to it's normal speed (big images
> > taking 4-5 seconds), increased MaxConnections to my "normal" 6.
> >
> >
> > I don't think this is a solution for everybody, and you may have a
> > better idea about when it is really appropriate to do a PNG
> > recompress?
> >
> >
> > image.c in compress_image()
> >
> > #ifdef JP2K
> >         if ((source_type == IMG_JP2K) && (has_transparency == 0) &&
> > (try_lossy == 0))
> >                 try_lossless = 0;
> >         if ((source_type == IMG_JP2K) && (has_transparency != 0) &&
> > (target_lossy == IMG_JP2K))
> >                 try_lossless = 0;
> >         if ((source_type == IMG_JP2K) && (has_transparency == 0) &&
> > (target_lossy = IMG_JPEG) && (try_lossy == 1)) /* Alf 20100620 */
> >                 try_lossless = 0;
> > #endif
> >
> >
> >
> > My debug output looks like this...
> >
> > [10070] Starting image decompression...
> > [10070] Image parms (non-palette) -- w: 600, h: 900, bpp: 3
> > [10070] Deciding image compression strategy...
> > [10070] Init flags: has_transparency=0 try_lossless=1 try_lossy=1
> > [10070] Final flags: has_transparency=0 try_lossless=0 try_lossy=1
> > [10070] Strategy defined. Continuing...
> > [10070] Attempting JPEG compression...
> > [10070] Compression return codes -- JP2K:33 JPEG:0 PNG:33
> > [10070] Image modification/compression took 3488304 us.
> > [10070]   and returned 0.
> >
> > So from 21seconds to 3seconds.
> >
> > Cheers, Andrew.
>
>
>
>
> ------------------------------------------------------------------------------
> ThinkGeek and WIRED's GeekDad team up for the Ultimate
> GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
> lucky parental unit.  See the prize list and enter to win:
> http://p.sf.net/sfu/thinkgeek-promo
> _______________________________________________
> Ziproxy-devel mailing list
> Zip...@li...
> https://lists.sourceforge.net/lists/listinfo/ziproxy-devel
>



-- 
"The simplest way to deal with a need or want is to stop needing or wanting
it." (John Michael Greer)

My household GHG emissions from energy & transport ~4kg CO2e/day or
~1.4tonnes/year
Household daily use of Water 120L, Electricity 4.0kWh, Petrol 1.2L, Gas 0MJ