mget patch

  • Laurence Rowe

    Laurence Rowe - 2008-05-21

    This patch adds recursive get support as a new command mget.


    Index: s3cmd

    --- s3cmd    (revision 179)
    +++ s3cmd    (working copy)
    @@ -298,6 +298,60 @@

    +def cmd_mget(args):
    +    s3 = S3(Config())
    +    src = args.pop(0)
    +    src_uri = S3Uri(src)
    +    if S3Uri(src).type != "s3":
    +        raise ParameterError("Source must be a S3 URI instead of: %s" % src)   
    +    dst = args.pop(0)
    +    if dst.endswith('/'):
    +        dst = dst[:-1]
    +    if os.path.exists(dst):
    +        if not os.path.isdir(dst):
    +            raise ParameterError("Destination must be a local directory instead of: %s" % dst)
    +    else:
    +        os.mkdir(dst)
    +    if (len(args)):
    +        raise ParameterError("Too many parameters! Expected: %s" % commands['mget']['param'])
    +    bucket = src_uri.bucket()
    +    prefix = src_uri.object()
    +    if prefix.endswith('*'):
    +        prefix = prefix[:-1]
    +    if prefix and not prefix.endswith('/'):
    +        prefix += '/'
    +    try:
    +        response = s3.bucket_list(bucket, prefix = prefix)
    +    except S3Error, e:
    +        if["Code"]):
    +            error([["Code"]] % bucket)
    +            return
    +        else:
    +            raise
    +    for object in response["list"]:
    +        uri = S3Uri(src_uri.compose_uri(bucket, object["Key"]))
    +        destination = dst + "/" + uri.object()[len(prefix):]
    +        if not Config().force and os.path.exists(destination):
    +            raise ParameterError("File %s already exists. Use --force to overwrite it" % destination)
    +        dest_dir, dest_file = os.path.split(destination)
    +        if not os.path.exists(dest_dir):
    +            os.makedirs(dest_dir) # XXX this is chmod 0777
    +        response = s3.object_get_uri(uri, destination)
    +        if response["headers"].has_key("x-amz-meta-s3tools-gpgenc"):
    +            gpg_decrypt(destination, response["headers"]["x-amz-meta-s3tools-gpgenc"])
    +            response["size"] = os.stat(destination)[6]
    +        if destination != "-":
    +            speed_fmt = formatSize(response["speed"], human_readable = True, floating_point = True)
    +            output("Object %s saved as '%s' (%d bytes in %0.1f seconds, %0.2f %sB/s)" %
    +                (uri, destination, response["size"], response["elapsed"], speed_fmt[0], speed_fmt[1]))
    def cmd_sync(args):
         def _build_attr_header(src):
             attrs = {}
    @@ -629,6 +683,7 @@
         {"cmd":"la", "label":"List all object in all buckets", "param":"", "func":cmd_buckets_list_all_all, "argc":0},
         {"cmd":"put", "label":"Put file into bucket", "param":"FILE [FILE...] s3://BUCKET[/PREFIX]", "func":cmd_object_put, "argc":2},
         {"cmd":"get", "label":"Get file from bucket", "param":"s3://BUCKET/OBJECT LOCAL_FILE", "func":cmd_object_get, "argc":1},
    +    {"cmd":"mget", "label":"Get file from bucket", "param":"s3://BUCKET[/prefix] LOCAL_DIR", "func":cmd_mget, "argc":1},
         {"cmd":"del", "label":"Delete file from bucket", "param":"s3://BUCKET/OBJECT", "func":cmd_object_del, "argc":1},
         #{"cmd":"mkdir", "label":"Make a virtual S3 directory", "param":"s3://BUCKET/path/to/dir", "func":cmd_mkdir, "argc":1},
         {"cmd":"sync", "label":"Synchronize a directory tree to S3", "param":"LOCAL_DIR s3://BUCKET[/PREFIX]", "func":cmd_sync, "argc":2},

    • Michal Ludvig

      Michal Ludvig - 2008-05-22

      Hi Laurence,

      thanks for the patch. The name "mget" however isn't optimal. I thing it should be merged to "sync" command and detect whether the first argument is S3Url to copy out from S3 to a local disk (you mget routine) or if the last one is S3Url in which case the current sync routine will run.

      Are you keen to update the patch or should I?



    • Laurence Rowe

      Laurence Rowe - 2008-05-23

      please feel free to update the patch, I didn't really understand the sync code. mget is not as silly as it sounds if you've used s command line ftp client ;-)

      • Michal Ludvig

        Michal Ludvig - 2008-06-05

        Hi Laurence,

        I'm aware of FTP mget command of course, I just didn't like the name as we have 'sync' for uploads and not mput.

        Anyway, I have now implemented s3cmd sync in the direction S3->local filesystem and released it in s3cmd 0.9.7. That should sort out the need for mget. Let me know whether this new sync does all you need.


        • dsbcpas

          dsbcpas - 2008-06-05

          Re: s3cmd 0.9.7
          For clarification, is the download sync command, ie  "Synchronize a directory tree from S3"

          s3cmd sync s3://BUCKET[/PREFIX] LOCAL_DIR

          • Michal Ludvig

            Michal Ludvig - 2008-06-05

            Yes, that's the one. All objects with name beginning with PREFIX will be downloaded to a directory LOCAL_DIR with the PREFIX part stripped off.

            For example:

            s3cmd sync s3://test-bucket/some/prefix/ /tmp/text-sync

            may download s3://test-bucket/some/prefix/project/data/file.cfg  to  /tmp/text-sync/project/data/file.cfg


Log in to post a comment.

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

No, thanks