Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

mget patch

2008-05-21
2013-02-19
  • Laurence Rowe
    Laurence Rowe
    2008-05-21

    This patch adds recursive get support as a new command mget.

    Laurence

    Index: s3cmd

    --- s3cmd    (revision 179)
    +++ s3cmd    (working copy)
    @@ -298,6 +298,60 @@
                 else:
                     raise

    +def cmd_mget(args):
    +    s3 = S3(Config())
    +
    +    src = args.pop(0)
    +    src_uri = S3Uri(src)
    +    if S3Uri(src).type != "s3":
    +        raise ParameterError("Source must be a S3 URI instead of: %s" % src)   
    +    dst = args.pop(0)
    +    if dst.endswith('/'):
    +        dst = dst[:-1]
    +    if os.path.exists(dst):
    +        if not os.path.isdir(dst):
    +            raise ParameterError("Destination must be a local directory instead of: %s" % dst)
    +    else:
    +        os.mkdir(dst)
    +    if (len(args)):
    +        raise ParameterError("Too many parameters! Expected: %s" % commands['mget']['param'])
    +
    +   
    +    bucket = src_uri.bucket()
    +    prefix = src_uri.object()
    +
    +    if prefix.endswith('*'):
    +        prefix = prefix[:-1]
    +    if prefix and not prefix.endswith('/'):
    +        prefix += '/'
    +    try:
    +        response = s3.bucket_list(bucket, prefix = prefix)
    +    except S3Error, e:
    +        if S3.codes.has_key(e.info["Code"]):
    +            error(S3.codes[e.info["Code"]] % bucket)
    +            return
    +        else:
    +            raise
    +    for object in response["list"]:
    +        uri = S3Uri(src_uri.compose_uri(bucket, object["Key"]))
    +
    +        destination = dst + "/" + uri.object()[len(prefix):]
    +        if not Config().force and os.path.exists(destination):
    +            raise ParameterError("File %s already exists. Use --force to overwrite it" % destination)
    +        dest_dir, dest_file = os.path.split(destination)
    +        if not os.path.exists(dest_dir):
    +            os.makedirs(dest_dir) # XXX this is chmod 0777
    +       
    +        response = s3.object_get_uri(uri, destination)
    +        if response["headers"].has_key("x-amz-meta-s3tools-gpgenc"):
    +            gpg_decrypt(destination, response["headers"]["x-amz-meta-s3tools-gpgenc"])
    +            response["size"] = os.stat(destination)[6]
    +        if destination != "-":
    +            speed_fmt = formatSize(response["speed"], human_readable = True, floating_point = True)
    +            output("Object %s saved as '%s' (%d bytes in %0.1f seconds, %0.2f %sB/s)" %
    +                (uri, destination, response["size"], response["elapsed"], speed_fmt[0], speed_fmt[1]))
    +
    +
    def cmd_sync(args):
         def _build_attr_header(src):
             attrs = {}
    @@ -629,6 +683,7 @@
         {"cmd":"la", "label":"List all object in all buckets", "param":"", "func":cmd_buckets_list_all_all, "argc":0},
         {"cmd":"put", "label":"Put file into bucket", "param":"FILE [FILE...] s3://BUCKET[/PREFIX]", "func":cmd_object_put, "argc":2},
         {"cmd":"get", "label":"Get file from bucket", "param":"s3://BUCKET/OBJECT LOCAL_FILE", "func":cmd_object_get, "argc":1},
    +    {"cmd":"mget", "label":"Get file from bucket", "param":"s3://BUCKET[/prefix] LOCAL_DIR", "func":cmd_mget, "argc":1},
         {"cmd":"del", "label":"Delete file from bucket", "param":"s3://BUCKET/OBJECT", "func":cmd_object_del, "argc":1},
         #{"cmd":"mkdir", "label":"Make a virtual S3 directory", "param":"s3://BUCKET/path/to/dir", "func":cmd_mkdir, "argc":1},
         {"cmd":"sync", "label":"Synchronize a directory tree to S3", "param":"LOCAL_DIR s3://BUCKET[/PREFIX]", "func":cmd_sync, "argc":2},

     
    • Michal Ludvig
      Michal Ludvig
      2008-05-22

      Hi Laurence,

      thanks for the patch. The name "mget" however isn't optimal. I thing it should be merged to "sync" command and detect whether the first argument is S3Url to copy out from S3 to a local disk (you mget routine) or if the last one is S3Url in which case the current sync routine will run.

      Are you keen to update the patch or should I?

      Thanks

      Michal

       
    • Laurence Rowe
      Laurence Rowe
      2008-05-23

      please feel free to update the patch, I didn't really understand the sync code. mget is not as silly as it sounds if you've used s command line ftp client ;-)

       
      • Michal Ludvig
        Michal Ludvig
        2008-06-05

        Hi Laurence,

        I'm aware of FTP mget command of course, I just didn't like the name as we have 'sync' for uploads and not mput.

        Anyway, I have now implemented s3cmd sync in the direction S3->local filesystem and released it in s3cmd 0.9.7. That should sort out the need for mget. Let me know whether this new sync does all you need.

        Michal

         
        • dsbcpas
          dsbcpas
          2008-06-05

          Re: s3cmd 0.9.7
          For clarification, is the download sync command, ie  "Synchronize a directory tree from S3"

          s3cmd sync s3://BUCKET[/PREFIX] LOCAL_DIR

           
          • Michal Ludvig
            Michal Ludvig
            2008-06-05

            Yes, that's the one. All objects with name beginning with PREFIX will be downloaded to a directory LOCAL_DIR with the PREFIX part stripped off.

            For example:

            s3cmd sync s3://test-bucket/some/prefix/ /tmp/text-sync

            may download s3://test-bucket/some/prefix/project/data/file.cfg  to  /tmp/text-sync/project/data/file.cfg