#20 [PATCH] files are not closed properly

Malfunction
closed-fixed
Michal Ludvig
s3cmd (117)
5
2008-11-11
2008-10-21
Patrick Linskey
No

Currently (0.9.8.3), cmd_sync_remote2local() does not close the files it opens. As a result, large syncs from S3 to a local client can run into ulimit problems. You can confirm this issue by running a sync from an S3 bucket to your client and examining the output of 'lsof' to watch the number of open files associated with the process.

The attached patch closes the file after processing is complete. I've validated that the number of open files no longer increases during the course of a remote sync.

Discussion

  •  
    Attachments
  • Michal Ludvig
    Michal Ludvig
    2008-11-05

    Unfortunately we can't use 'finally:' statement because of a compatibility with python 2.4

    As far as I can tell the culprit is:
    os.open(dst_file, open_flags)
    We can close that one right after it's open since we don't do any I/O on the returned file_handle. The call is there just to raise an exception if the file already exists.

     
  • Michal Ludvig
    Michal Ludvig
    2008-11-11

    Fixed in 0.9.8.4. Thanks for the report.

     
  • Michal Ludvig
    Michal Ludvig
    2008-11-11

    • assigned_to: nobody --> ludvigm
    • status: open --> closed-fixed