From: Jean J. <jea...@gm...> - 2009-08-17 08:40:14
|
Hi there I'm deleting a bucket with many objects like this: $ ./s3cmd --verbose --progress rb --force s3://BUCKETNAME Memory use is at 1.8GB resident now. s3cmd version 0.9.9 -- jean . .. .... //\\\oo///\\ |
From: Michal L. <mi...@lo...> - 2009-08-17 10:05:12
|
Hi Jean, > I'm deleting a bucket with many objects like this: > $ ./s3cmd --verbose --progress rb --force s3://BUCKETNAME > > Memory use is at 1.8GB resident now. I know this is an issue for users with extremly large buckets. For now, before the memory utilisation is optimised, I suggest to delete the objects in chunks, for example if "s3cmd ls s3://BUCKETNAME/" gives you s3://BUCKETNAME/folder1 s3://BUCKETNAME/folder2 s3://BUCKETNAME/folder3 you can do: s3cmd --recursive del s3://BUCKETNAME/folder1 once it's done: s3cmd --recursive del s3://BUCKETNAME/folder2 etc. That should make it much faster. Also give s3cmd-0.9.9-rc3-speedup release a try. It' in SourceForge -> Files -> testing. It should significantly trim down the runtime but doesn't work from behind a proxy. Michal |
From: Jean J. <jea...@gm...> - 2009-08-17 10:19:08
|
Hi Michal > I know this is an issue for users with extremly large buckets. Ah, good to know others are also feeling the pain ;-) > For now, > before the memory utilisation is optimised, I suggest to delete the > objects in chunks, Everything is in one massive bucket. The objects are smallish files created by the quillen backup-to-S3 tool. Is there any good way to delete those in chunks? E.g. saving all object ids to a textfile and then iterating through that, deleting one at a time? When s3cmd reports File s3://BUCKETNAME/1cc78b9209b129e9ab52a7a532ec4213dc7a7b05-667535 deleted is that file is really deleted, or is it pending commit? > Also give s3cmd-0.9.9-rc3-speedup release a try. It' in SourceForge -> > Files -> testing. It should significantly trim down the runtime but > doesn't work from behind a proxy. I'm not behind a proxy. I'll give it a try. Regards, -- jean . .. .... //\\\oo///\\ |
From: Michal L. <mi...@lo...> - 2009-08-17 10:39:10
|
Jean Jordaan wrote: > When s3cmd reports > File s3://BUCKETNAME/1cc78b9209b129e9ab52a7a532ec4213dc7a7b05-667535 deleted You can still use the method I suggested: for PREFIX_INT in $(seq 0 255); do PREFIX=$(printf "%02x" ${PREFIX_INT}) s3cmd del -r s3://BUCKETNAME/${PREFIX}* done That'll remove the bucket content in 256 runs. First only filenames starting with 00, then 01, ... 1c, ... ff and it's done. Hopefully ;-) > is that file is really deleted, or is it pending commit? Once it says deleted it's deleted. At least some good news. Michal |