#4405 HTTP compression actually enlarging transfers

[Ryan Malayter]

HTTP compression seems to be enbaled on your Apache, but actually enlarges the data sent to the browser. This is costing you lots of bandwidth. Maybe checl your compression settings on Apache? Tested with the Fiddler debugging proxy.

See http://slashdot.org/~Thundersnatch/journal/235233?art_pos=1 for more info

[Ryan Malayter]

Here is a specif example, which downloaded at 827022 bytes, even though it is encoded with gzip (see headers below). The fiddler proxy shows the compressed content at 826826 bytes. When re-compressed using the fiddler tool's diagnostics (roughly equivalent to gzip -1), the file size is just 129744 bytes.

Clearly there is something wrong with how compression is configured on slashdot.org. I'm not sure if this is an issue at the Apache or Varnish layers (which does the compression?)

HTTP/1.1 200 OK
Server: Apache/1.3.41 (Unix) mod_perl/1.31-rc4
X-Powered-By: Slash 2.005001
X-Leela: He opened up relations with China. He doesn't want to hear about your ding-dong.
X-XRDS-Location: http://slashdot.org/slashdot.xrds
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
Date: Tue, 18 Aug 2009 02:20:28 GMT
X-Varnish: 1217353453
Age: 1
Connection: keep-alive
Vary: Accept-Encoding, User-Agent
Content-Encoding: gzip
Transfer-Encoding: chunked

[Ryan Malayter]

By the way, this investigation started for me when I noticed that slashdot was extraordinarily slow on my sprint 3G card. When your site is loading slower than adobe.com, you have issues ;-)

Let me know if you want me to do any further testing.

[Ryan Malayter]

Last post, I promise... here are the request headers for the response details posted above:

GET /article.pl?sid=09/08/12/1129230&tid=11 HTTP/1.1
Host: news.slashdot.org
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/530.5 (KHTML, like Gecko) Chrome/2.0.172.39 Safari/530.5
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Encoding: gzip,deflate,sdch
Cookie: <sanitized>
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3

[Ryan Malayter]

You may want to adjust the priority on this, depending on how you pay for bandwidth. My tests show the average slashdot dynamic HTML page compresses at 6-to-1. If you pay for bandwidth based on the 95/5 rule or per-byte, fixing this setting could immediately save gobs of money for Slashdot. I have had clients whose sites aren't as popular as Slashdot where implementing HTTP compression and caching properly for dynamic content has saved thousands of dollars per month. It is dead-simple, low-hanging fruit.

The other impact is that the end-user experience is vastly better, especially on wireless devices.