#108 unnecessary memory usage by the du command

open
nobody
None
5
2014-08-25
2011-12-29
Anonymous
No

I've managed to crash a server by running s3cmd du s3://giganticbucket
This is due to the du command building a list of all objects in the bucket, and then iterating over the list to get the size.

The attached patch against s3cmd 1.0.0 calculates the size as the truncated responses are coming in, and doesn't build a list of all objects thus using constant memory, and saving a little user time.

Discussion


  • Anonymous
    2011-12-29

    Patch for du command to reduce memory usage

     
    Attachments

  • Anonymous
    2011-12-29

    With debugging print removed. Use this one please.

     
    Attachments