Write speeds?

Darren G.
2009-01-24
2013-02-19
  • Darren G.
    Darren G.
    2009-01-24

    Hi,
      I'm curious what the typical write speeds are for folks using this nice package. I understand its related to many things other than s3cmd, but I'm seeing rather poor write speeds from within EC2. Just curious if this is typical for others using s3cmd to S3.

    s3cmd reports 2-6MB/sec, but it seems it takes about 1 clock second per file of 400 bytes or so - which is slow. I'm trying to write many small files.

    thanks,

     
    • Michal Ludvig
      Michal Ludvig
      2009-01-24

      Hi, do you have HTTPS or GPG turned on? That may explain the slow speed. I personally don't use it on EC2 and don't have any relevant numbers for comparison.

      However there's a significant speed-up rewrite underway. As soon as it's usable I'll post an update to s3tools-general mailing list.

       
    • Darren G.
      Darren G.
      2009-01-25

      Hi, I believe so. I guess I should have looked into that. Look forward to the next release. Great job on this useful tool.

      Is it accessible from python like an API?

      Many thanks.

       
      • Michal Ludvig
        Michal Ludvig
        2009-01-25

        Hi,
        no it isn't a callable API yet. Only the low-level stuff like "put object" is in a callable module but the high level operations (like "sync this directory to that bucket") is not. It's on my TODO list though. CloudFront support is already designed that way.

         
    • Darren G.
      Darren G.
      2009-01-25

      Hi, I believe so. I guess I should have looked into that. Look forward to the next release. Great job on this useful tool.

      Is it accessible from python like an API?

      Many thanks.

       
    • Michal Ludvig
      Michal Ludvig
      2009-01-30

      Hi, can you try s3cmd from s3cmd-speedup SVN branch? It has some significant speed improvements. For instance doing 'setacl' on a bucket with 150 files originally took 6:30 minutes while now with the speedup branch it's less then 2 mins, using https over my ADSL connection.