On 08/16/2010 10:18 PM, Hina Tabassum wrote:

Read about your work at www.sourgeforge.net regarding Amazon S3 logs. Great work I must say.

I have a problem & really know nothing about S3 facility. I heard it first time.

I want to track bandwidth use for each bucket/user.  At the end of the month, we need to generate a report for each bucket and report the bandwidth used.

So I need to know how to start the work or what I am supposed to start from. I am able to get a list of buckets but  thats it. Except that I am able to do nothing.

How to download daily logs for a bucket is the place where I am lost. Pls help.


Hi Hina,

there are two sorts of access logs - one is the log of the S3 storage itself (generated per-bucket) and one is usage log for CloudFront, obviously only available if you deliver content from S3 over CloudFront, i.e. when you use S3 as a storage for web content as oppose to, say, for backup purposes.

The former (S3 logs) are enabled with:
s3cmd accesslog --access-logging-target-prefix=s3://your-log-bucket/target-bucket/s3/ s3://target-bucket
That will enable access logging for 's3://target-bucket' and store the logs to 's3://your-log-bucket/target-bucket/s3/' (obviously you should create s3://your-log-bucket first).

Similarly for CloudFront logging can be enabled with:
s3cmd cfmodify --access-logging-target-prefix=s3://your-log-bucket/target-bucket/cf/ cf://THEDISTID

At the end of the month simply download all the logs from s3://your-log-bucket/ and process to get whatever reports you need.

It is recommended to store the S3 logs in a separate bucket, not in the one being monitored. Otherwise writing the logfiles into the same bucket generates access requests and is logged too, which creates kind of a loop.

Hope that helps