#17 HTTP Proxy Support

[Cortland Klein]

Dijjer should offer a localhost HTTP Proxy that can be
used by complient HTTP clients. This could be run on
port 9116, for example.

Reasons why Dijjer should offer a localhost HTTP Proxy:

* Content providers don't need to know about Dijjer
* Users don't have to manually modify URLs to use Dijjer
* Existing URLs loaded in clients can support Dijjer
* The proxy could fallback to downloading the content
directly if Dijjer fails to work with the URL
* The proxy could bypass Dijjer for small documents
less than 256k or 512k.

Proposed reasons why it should not and rebuttals:

Q. Users and content providers should manually decide
when content should be distributed through Dijjer or not.
A. Content is content. It saves content providers
bandwidth, and increases access speeds for users.

Q. A minimum size option would need to be modifiable by
the user.
A. Let the user modify it from a default of 256k or 512k.

Q. Many URLs don't support Range header.
A. Fallback to a regular download.

Q. A mozilla plugin will do.
A. The plugin would merely be doing the same thing. The
plugin might in fact just turn on the proxy.

The proxy could even be written seperate from Dijjer.
The proxy would then internally prefix
http://localhost:9115/ pending the URL's host supports
the Range header and it's Length is langer than 256k or
512k.

[Jay Knight]

Logged In: YES
user_id=259283

I had thought of this also... Since there is other interest,
I whipped this up real quick (actually, I waste half of my
afternoon at work):

#!/usr/bin/perl -w

use strict;
use HTTP::Proxy;

my $proxy = HTTP::Proxy->new( port => 9116 );

{
package DijjerFilter;
use base qw( HTTP::Proxy::HeaderFilter );

sub filter {
my ( $self, $headers, $message ) = @_;
my $ua = $self->proxy->agent;
my $response = $ua->head($message->uri);
if ($response->header('Accept-Ranges') &&
$response->header('Content-Length') &&
$response->header('Content-Length') > 100*1024
) {
$message->uri('http://127.0.0.1:9115/'.$message->uri());
print "dijjerized\n";
}
}
}
$proxy->push_filter( request => DijjerFilter->new() );
$proxy->start;

It's probably not perfect, but it seems to work okay.

[Daniel Keshet]

Logged In: YES
user_id=549952

Very cool. Using the perl proxyserver and everything works
quite well. Without it, I wouldn't bother using dijjer.

[Cortland Klein]

Logged In: YES
user_id=274439

Using rytsarsky's script from 2005-01-27 23:24, I get the
following error:

Can't locate object method "new" via package "DijjerFilter"
at ./dijjer-proxy line 25.

[Jay Knight]

Logged In: YES
user_id=259283

It kind of weird syntax, maybe doesn't work with some
versions/interpreters. Try this one:

#!/usr/bin/perl -w

use strict;
use HTTP::Proxy;
use HTTP::Proxy::HeaderFilter::simple;

my $proxy = HTTP::Proxy->new( port => 9116 );
my $dijjerfilter = HTTP::Proxy::HeaderFilter::simple->new(
sub {
my ( $self, $headers, $message ) = @_;
my $ua = $self->proxy->agent;
my $response = $ua->head($message->uri);
if ($response->header('Accept-Ranges') &&
$response->header('Content-Length') &&
$response->header('Content-Length') > 100*1024
) {
$message->uri('http://127.0.0.1:9115/'.$message->uri());
print "dijjerized\n";
}
}
);
$proxy->push_filter( request => $dijjerfilter );
$proxy->start;

[Mason]

Logged In: YES
user_id=258255

An additioanl proxy (on port 9116, maybe) would not be a bad
idea, and would be fairly simple to implement.

[Cortland Klein]

Logged In: YES
user_id=274439

rytsarsky: I found the problem (my HTTP::Proxy modules were
messed up), and thus both work. Thanks for your awesome
implementation; hopefully something like it will get into
Dijjer soon.

[Jay Knight]

Logged In: YES
user_id=259283

25 lines of code hardly qualifies as an "awesome
implementation." I've noticed that while I was using it,
browsing is quite a bit slower, even when nothing is
getting dijjerized. I set the number of children higher and
increased the timeout (because sometimes dijjer wouldn't
send any data for a long period, and the proxy would
timeout) That helped some, but not quite enough for me to
have this running all the time. The docs for HTTP::Dijjer
are at
http://search.cpan.org/~book/HTTP-Proxy-0.13/lib/HTTP/Proxy.pm
if anyone wishes to play some more... I imagine that even if
this is something Ian wants in Dijjer, it would have to be
written in Java.