rabbit-proxy-development

[Rabbit-proxy-development] Getting Contentlength

[<isc...@gm...>]

Hi to all,

i would like to know, how to get the content-length of incoming and 
outgoing traffic. The Fields in Connection are only filled, when an 
internal message should be sent to the client.

But how can i get the amount of data transfered by each connection?

I'm using the Version 3.6 of Rabbit.

Greetings
Ingo

Re: [Rabbit-proxy-development] Getting Contentlength

[Robert O. <ro...@kh...>]

Ingo Schr=F6der wrote:
> i would like to know, how to get the content-length of incoming and=20
> outgoing traffic. The Fields in Connection are only filled, when an=20
> internal message should be sent to the client.

You can check to see if that header is set or not.
In many cases you will not know in advance:
*) http/1.1 with chunking
*) http/0.9
*) http/1.0 or http/1.1 with "Connection: close" set.

If you check the header then be sure that you count the correct amount,
when rabbit filters a resource it may change the content length.

> But how can i get the amount of data transfered by each connection?

All data that is transferred is counted so you can implement a
TrafficLogger and count the data as it happen.

Rabbit usually just keeps simple statistics, check the status page
to see the normal statistics.

/robo

Re: [Rabbit-proxy-development] Getting Contentlength

[<isc...@gm...>]

Hello Robert,

thanks for the quick answer. So the TrafficLogger would be my friend. 
After searching the source, I found 2 places, where the traffic is 
logged (ResouceHandlerBase and WebConnectionResourceSource).  As I 
understood the source, the WebConnectionResourceSource is only used for 
getting information from the net and so the ResourceHandlerBase would be 
better. Is this correct?

I want to assign the traffic to the connection. Would the 
ResourceHandlerBase the right place to do this?

Greetings

Ingo
Robert Olofsson schrieb:
> Ingo Schröder wrote:
>> But how can i get the amount of data transfered by each connection?
>
> All data that is transferred is counted so you can implement a
> TrafficLogger and count the data as it happen.
>
> Rabbit usually just keeps simple statistics, check the status page
> to see the normal statistics.
>
> /robo
>

Re: [Rabbit-proxy-development] Getting Contentlength

[Robert O. <ro...@kh...>]

Ingo Schr=F6der wrote:
> thanks for the quick answer. So the TrafficLogger would be my friend.=20
> After searching the source, I found 2 places, where the traffic is=20
> logged (ResouceHandlerBase and WebConnectionResourceSource). =20

The traffic is logged in more places.
Tunnel: logs two ways.
WebConnectionResourceSource: logs data read from the net.
TransferHandler: logs data transferred (cache->client).
BlockSender: logs data sent, can be either client or server.
HttpHeaderReader: logs data read, can be either client or server.
BaseMetaHandler: logs proxy->client data, probably not interesting.

I may have missed one or two cases...

If you are ok to update the counters only at the end of each
request you can look at: Connection.logConnection(), it updates
the proxy statistics after each handled request.

> I want to assign the traffic to the connection. Would the=20
> ResourceHandlerBase the right place to do this?

Probably not, depends on what you mean by "connection" and what
you want to use the data for.

/robo

Re: [Rabbit-proxy-development] Getting Contentlength

[<ISc...@gm...>]

Hello Robert,

my first interest would be getting all the traffic from the client to the proxy. I would like to filter all the clients sending more than 512 kB as a fileupload for example.

Greetings

Ingo
-------- Original-Nachricht --------
Datum: Sun, 07 Jan 2007 22:17:45 +0100
Von: Robert Olofsson <ro...@kh...>
An: "Ingo Schröder" <isc...@gm...>
Betreff: Re: [Rabbit-proxy-development] Getting Contentlength

> Ingo Schröder wrote:
> > thanks for the quick answer. So the TrafficLogger would be my friend. 
> > After searching the source, I found 2 places, where the traffic is 
> > logged (ResouceHandlerBase and WebConnectionResourceSource).  
> 
> The traffic is logged in more places.
> Tunnel: logs two ways.
> WebConnectionResourceSource: logs data read from the net.
> TransferHandler: logs data transferred (cache->client).
> BlockSender: logs data sent, can be either client or server.
> HttpHeaderReader: logs data read, can be either client or server.
> BaseMetaHandler: logs proxy->client data, probably not interesting.
> 
> I may have missed one or two cases...
> 
> If you are ok to update the counters only at the end of each
> request you can look at: Connection.logConnection(), it updates
> the proxy statistics after each handled request.
> 
> > I want to assign the traffic to the connection. Would the 
> > ResourceHandlerBase the right place to do this?
> 
> Probably not, depends on what you mean by "connection" and what
> you want to use the data for.
> 
> /robo

-- 
Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! 
Ideal für Modem und ISDN: http://www.gmx.net/de/go/smartsurfer

Re: [Rabbit-proxy-development] Getting Contentlength

[<isc...@gm...>]

Hello Robert,

for my assurance. Calling the method from the connection when filtering 
it would bring the part of bytes transferred by this connection. And 
afterwards this counter would be set to null and by the next time only 
the new amount would be available.

So by saving the connection to a database and using some sql i would get 
the slightly higher amount.

Is that correct so far. This would solve my problem.

Ingo
Robert Olofsson schrieb:
> Ingo Schröder wrote:
>> I will try to do the suggested workaround tomorrow. So by 
>> implementing the TrafficLogger I would get some more traffic then the 
>> real size of the resource. This would be ok. I would test and maybe 
>> get a feeling for the difference.
>
> You may get a larger size, yes. Uploads that have a known content 
> length (and it usually is) will send the correct amount. Chunked 
> transfers
> are usually done in something like 512b/2k/larger buffers and then you
> have a chunk header that usually is something like: "\r\n200\r\n"
>
>> Having not used the TrafficLogger, how would i get it 
>> (con.getTrafficHandler().getClient().getAmount() for example?) And 
>> what would it show me if i use it from a filter (httpInFilter in my 
>> case). Is the complete resource / byte count available at that 
>> moment? Or should i get the right data later? And from what starting 
>> point?
>
> IF you follow my idea you will get calls to read(long) with the amount
> of data just read. You have to keep track of how much you have read
> (similar to how SimpleTrafficLogger does it).
> It will be the amount of data the connection have seen this far in the
> handling of this resource. If you check the Connection.logConnection
> method you will see that after each call the proxy statistics are
> updated and then the connection clears its statistics (to be ready for
> the next request).
>
> /robo
>

Re: [Rabbit-proxy-development] Getting Contentlength

[Robert O. <ro...@kh...>]

Ingo Schr=F6der wrote:
> for my assurance. Calling the method from the connection when filtering=
=20
> it would bring the part of bytes transferred by this connection. And=20
> afterwards this counter would be set to null and by the next time only=20
> the new amount would be available.
>=20
> So by saving the connection to a database and using some sql i would ge=
t=20
> the slightly higher amount.

Saving the connection to a database? a connection may be used to handle
many requests. Depending on the user a connection _average_ something
between 2 and 3 requests for the data I have seen.

But yes, the TrafficLogger is cleared for each request. The variable is
not set to null, it is cleared and reused.

/robo

Re: [Rabbit-proxy-development] Getting Contentlength

[Robert O. <ro...@kh...>]

Ingo Schr=F6der wrote:
> my first interest would be getting all the traffic from the client to
> the proxy. I would like to filter all the clients sending more than
> 512 kB as a fileupload for example.

Ok, there is no easy way to do that at the moment, you have to add some
code. Here is what I would do:
1) create a class that implements TrafficLogger, maybe taking a
    Connection in its constructor.
2) Add Connection as a parameter to TrafficLoggerHandler
3) Change the client-parameter to be the class from 1.

Does that help?

There are still a few things to think about here:
1) Chunked uploads will count the bytes transferred, not the resource
    size.
2) Multipart uploads will count the bytes transferred, there may be many
    resources in the upload.

Does that help?

/robo

Re: [Rabbit-proxy-development] Getting Contentlength

[<isc...@gm...>]

Hello Robert,

I will try to do the suggested workaround tomorrow. So by implementing 
the TrafficLogger I would get some more traffic then the real size of 
the resource. This would be ok. I would test and maybe get a feeling for 
the difference.

Having not used the TrafficLogger, how would i get it 
(con.getTrafficHandler().getClient().getAmount() for example?) And what 
would it show me if i use it from a filter (httpInFilter in my case). Is 
the complete resource / byte count available at that moment? Or should i 
get the right data later? And from what starting point?

By the way, thanks for the quick help. I was suprised how quick the 
answers came.

Ingo
Robert Olofsson schrieb:
> Ingo Schröder wrote:
>   
>> my first interest would be getting all the traffic from the client to
>> the proxy. I would like to filter all the clients sending more than
>> 512 kB as a fileupload for example.
>>     
>
> Ok, there is no easy way to do that at the moment, you have to add some
> code. Here is what I would do:
> 1) create a class that implements TrafficLogger, maybe taking a
>     Connection in its constructor.
> 2) Add Connection as a parameter to TrafficLoggerHandler
> 3) Change the client-parameter to be the class from 1.
>
> Does that help?
>
> There are still a few things to think about here:
> 1) Chunked uploads will count the bytes transferred, not the resource
>     size.
> 2) Multipart uploads will count the bytes transferred, there may be many
>     resources in the upload.
>
> Does that help?
>
> /robo
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> Rabbit-proxy-development mailing list
> Rab...@li...
> https://lists.sourceforge.net/lists/listinfo/rabbit-proxy-development
>
>   

Re: [Rabbit-proxy-development] Getting Contentlength

[Robert O. <ro...@kh...>]

Ingo Schr=F6der wrote:
> I will try to do the suggested workaround tomorrow. So by implementing=20
> the TrafficLogger I would get some more traffic then the real size of=20
> the resource. This would be ok. I would test and maybe get a feeling fo=
r=20
> the difference.

You may get a larger size, yes. Uploads that have a known content length=20
(and it usually is) will send the correct amount. Chunked transfers
are usually done in something like 512b/2k/larger buffers and then you
have a chunk header that usually is something like: "\r\n200\r\n"

> Having not used the TrafficLogger, how would i get it=20
> (con.getTrafficHandler().getClient().getAmount() for example?) And what=
=20
> would it show me if i use it from a filter (httpInFilter in my case). I=
s=20
> the complete resource / byte count available at that moment? Or should =
i=20
> get the right data later? And from what starting point?

IF you follow my idea you will get calls to read(long) with the amount
of data just read. You have to keep track of how much you have read
(similar to how SimpleTrafficLogger does it).
It will be the amount of data the connection have seen this far in the
handling of this resource. If you check the Connection.logConnection
method you will see that after each call the proxy statistics are
updated and then the connection clears its statistics (to be ready for
the next request).

/robo