Re: [S3tools-general] Regarding Amazon S3 logs

[Michal L. <mi...@lo...>]

  On 08/16/2010 10:18 PM, Hina Tabassum wrote:
> Hi,
>
> Read about your work at www.sourgeforge.net 
> <http://www.sourgeforge.net> regarding Amazon S3 logs. Great work I 
> must say.
>
> I have a problem & really know nothing about S3 facility. I heard it 
> first time.
>
> I want to track bandwidth use for each bucket/user.  At the end of the 
> month, we need to generate a report for each bucket and report the 
> bandwidth used.
>
> So I need to know how to start the work or what I am supposed to start 
> from. I am able to get a list of buckets but  thats it. Except that I 
> am able to do nothing.
>
> How to download daily logs for a bucket is the place where I am lost. 
> Pls help.
>
> ---
> Hina
>


Hi Hina,

there are two sorts of access logs - one is the log of the S3 storage 
itself (generated per-bucket) and one is usage log for CloudFront, 
obviously only available if you deliver content from S3 over CloudFront, 
i.e. when you use S3 as a storage for web content as oppose to, say, for 
backup purposes.

The former (S3 logs) are enabled with:
s3cmd accesslog 
--access-logging-target-prefix=s3://your-log-bucket/target-bucket/s3/ 
s3://target-bucket
That will enable access logging for 's3://target-bucket' and store the 
logs to 's3://your-log-bucket/target-bucket/s3/' (obviously you should 
create s3://your-log-bucket first).

Similarly for CloudFront logging can be enabled with:
s3cmd cfmodify 
--access-logging-target-prefix=s3://your-log-bucket/target-bucket/cf/ 
cf://THEDISTID

At the end of the month simply download all the logs from 
s3://your-log-bucket/ and process to get whatever reports you need.

It is recommended to store the S3 logs in a separate bucket, not in the 
one being monitored. Otherwise writing the logfiles into the same bucket 
generates access requests and is logged too, which creates kind of a loop.

Hope that helps

Michal