Re: [S3tools-general] files-from

[Matt D. <Mat...@de...>]

On Tue, Feb 19, 2013 at 11:49:23PM -0600, Michael Ludvig wrote:
> Can we have --files-from enabled for stdin? I think it would be a common
> scenario:
> 
> find . -name \*.gpg | s3cmd --files-from=- put s3://...

Done.  I put this in my files-from branch, which is represented in
pull request #116.  Patch below.

Note, this works with sync local2remote, and put.  When used with sync
remote2local, at present it causes the local (destination) side to
think the only files present are those specified in files-from, which
is not the behavior we want.  We want it to operate on the source
side, not the destination side.  However, where we get the file lists
(local or remote), we don't know know if it's source or destination.

Since we want to know that, we should pass src or dest information
into fetch_{remote,local}_list() as an argument, and then make the
corresponding change to add processing to fetch_remote_list().  Unless
there's a better way I'm not seeing...

Also note, this is scary to do with any --delete options enabled, as
the source list will be exactly the list passed.

Thanks,
Matt

-- 
Matt Domsch
Technology Strategist
Dell | Office of the CTO


>From b76c5b38457caf93e1c3c16fa5dcbbb1dd9b1709 Mon Sep 17 00:00:00 2001
From: Matt Domsch <Mat...@de...>
Date: Wed, 20 Feb 2013 14:11:37 -0600
Subject: [PATCH] accept --files-from=- to read from stdin

This allows shell syntax:

find . -name \*.gpg | s3cmd sync --files-from=- src dst

to take the list of files to transfer from stdin.

Be careful, as using with a --delete option will cause files on the
remote side not listed in stdin to be deleted too.
---
 S3/FileLists.py | 14 ++++++++++++--
 s3cmd           |  2 +-
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/S3/FileLists.py b/S3/FileLists.py
index fae9004..4b84b08 100644
--- a/S3/FileLists.py
+++ b/S3/FileLists.py
@@ -14,6 +14,7 @@ from HashCache import HashCache
 from logging import debug, info, warning, error
 
 import os
+import sys
 import glob
 import copy
 
@@ -150,14 +151,23 @@ def _get_filelist_from_file(cfg, local_path):
 
     filelist = {}
     for fname in cfg.files_from:
-        f = open(fname, 'r')
+        if fname == u'-':
+            f = sys.stdin
+        else:
+            try:
+                f = open(fname, 'r')
+            except IOError, e:
+                warning(u"--files-from input file %s could not be opened for reading (%s), skipping." % (fname, e.strerror))
+                continue
+                
         for line in f:
             line = line.strip()
             line = os.path.normpath(os.path.join(local_path, line))
             dirname = os.path.dirname(line)
             basename = os.path.basename(line)
             _append(filelist, dirname, basename)
-        f.close()
+        if f != sys.stdin:
+            f.close()
 
     # reformat to match os.walk()
     result = []
diff --git a/s3cmd b/s3cmd
index c1a1a28..0804db8 100755
--- a/s3cmd
+++ b/s3cmd
@@ -1738,7 +1738,7 @@ def main():
     optparser.add_option(      "--rinclude", dest="rinclude", action="append", metavar="REGEXP", help="Same as --include but uses REGEXP (regular expression) instead of GLOB")
     optparser.add_option(      "--rinclude-from", dest="rinclude_from", action="append", metavar="FILE", help="Read --rinclude REGEXPs from FILE")
 
-    optparser.add_option(      "--files-from", dest="files_from", action="append", metavar="FILE", help="Read list of source-file names from FILE")
+    optparser.add_option(      "--files-from", dest="files_from", action="append", metavar="FILE", help="Read list of source-file names from FILE. Use - to read from stdin.")
     optparser.add_option(      "--bucket-location", dest="bucket_location", help="Datacentre to create bucket in. As of now the datacenters are: US (default), EU, ap-northeast-1, ap-southeast-1, sa-east-1, us-west-1 and us-west-2")
     optparser.add_option(      "--reduced-redundancy", "--rr", dest="reduced_redundancy", action="store_true", help="Store object with 'Reduced redundancy'. Lower per-GB price. [put, cp, mv]")
 
-- 
1.8.1.2