Activity for s3cmd
-
Ravi posted a comment on discussion Open Discussion
Hello Support Team, I have two S3 buckets where versioning and replication are enabled. However, there are still about 500,000 files missing in the replication bucket that exist in our main bucket. I tried running the following s3cmd command: s3cmd sync --skip-existing -r s3://main-data s3://main-recovery When I run this command, it copies all items from main-data to main-recovery, creating a version of each item and synchronizing them. It seems that only after deleting all versions from the main-recovery...
-
s3cmd released /s3cmd/2.4.0/s3cmd-2.4.0.tar.gz
-
-
-
-
-
-
Daniel Looby posted a comment on discussion Open Discussion
New to s3cmd. Ran the s3cmd --configuration and it created the .s3cfg file in my $HOME (Linux) directory. I want to automate a download of files daily via a another user but don't want the configuration file in that user's $HOME. I tried s3cmd --config /app/banner/bdevlc/home/gtsis/misc/.s3cfg and ensured that the permissions on the file were set so group had read/write and user was in the group. But when I tried to then get a listing of files in the bucket it complained that it couldn't access the...
-
Erik posted a comment on discussion Open Discussion
I started to use s3 for backups ..... our old config has 4 storage NFS mount points to spread out the write load. What about s3 ? Do I gain any performance by ie creating multiple buckets like (4) Rather than just using one backup bucket and pour all my backups into the same destination I have like 8 different backup servers that write to the same s3 at the same time To me it sounds like a 'bottle-neck' issue but I don't have the experice or insight in what s3 can do or not do Please advice thx
-
Hi I am new to s3 - we moving from NFS to s3 but I am not happy with the write preformance HOW can I speed it up ? some config parameter that move largers packages ? or is it how you setup the hardware - ie hardware parameters ? Now it's around 10MB/s vs our regular NFS filesystem 200MB/s ie 20 times slower !!!! ugh TEST upload: '<stdin>' -> 's3://yetest/backups/si106-251/si106-251.20221219202432.tgz' [part 299, 15MB] 15728640 of 15728640 100% in 1s 8.45 MB/s done</stdin>
-
Ho Tien Hung modified a comment on discussion Open Discussion
Hi All, I'm using the command below to upload a file to S3 with the file encrypted. s3cmd -e put /local/to-bk-file/file.txt s3://data/backup During the upload process, the temporary encrypted file is generated and stored at the /tmp/ directory e.g: upload: '/tmp/tmpfile-scTm46AYxJuiWJ1XcVEI' -> 's3://data/backup/file.zip' [1 of 1] Is possible to change the default /tmp to another like /mnt/tmp? Since I need to reduce the hard drive space so the tmp-file should be stored on another disk. My env: Ubuntu...
-
Hi All, I'm using the command below to upload a file to S3 with the file encrypted. s3cmd -e put /local/to-bk-file/file.txt s3://data/backup During the upload process, the temporary encrypted file is generated and stored at the /tmp/ directory e.g: upload: '/tmp/tmpfile-scTm46AYxJuiWJ1XcVEI' -> 's3://data/backup/file.zip' [1 of 1] Is possible to change the default /tmp to another like /mnt/tmp? My env: Ubuntu Thanks in advance. Hung
-
Hi All, I'm using the command below to upload a file to S3 with the file encrypted. s3cmd -e put /local/to-bk-file/file.txt s3://data/backup During the upload process, the temporary encrypted file is generated and stored at the /tmp/ directory e.g: upload: '/tmp/tmpfile-scTm46AYxJuiWJ1XcVEI' -> 's3://data/backup/file.zip' [1 of 1] How to change the default /tmp to another like /mnt/tmp? My env: Ubuntu Thanks in advance. Hung
-
Hi All, I'm using the command below to upload a file to S3 with the file encrypted. s3cmd -e put /local/to-bk-file/file.txt s3://data/backup During the upload process, the temporary encrypt file is generated and stored at the /tmp/ directory e.g: upload: '/tmp/tmpfile-scTm46AYxJuiWJ1XcVEI' -> 's3://data/backup/file.zip' [1 of 1] How to change the default /tmp to another like /mnt/tmp? My env: Ubuntu Thanks in advance. Hung
-
Hi All, I'm using the command below to upload a file to S3 with the file encrypted. s3cmd -e put /local/to-bk-file/file.txt s3://data/backup During the upload process, the temporary encrypt file is generated and stored at the /tmp/ directory e.g: upload: '/tmp/tmpfile-scTm46AYxJuiWJ1XcVEI' -> 's3://data/backup/file.zip' [1 of 1] How to change the default /tmp to another like /mnt/tmp? My env: Ubuntu
-
-
-
-
-
-
-
John Newman posted a comment on discussion Open Discussion
Hi, I was wondering when the likely release date or predicted release date would be for making s3cmd compatible with IMDSv2? Thanks in advance John
-
-
-
-
-
-
-
Sharadha K posted a comment on discussion Open Discussion
Hi, I am trying to delete files inside a particular folder in the s3 bucket. For this I am using the command s3cmd del s3://bucket_name/folder1/folder2/file_name. When I execute this, the entire directory structure gets deleted, meaning folder1 and folder2 also get deleted along with the file. So can someone let me know how to solve this and delete only the file but preserve the directories above it?
-
Florent Viard posted a comment on discussion Open Discussion
Hi, Your issue is that you are not supposed to put the "domain name" in your "s3:// " element but directly the s3 bucket name. So, for example, in your case, you have to use directly 's3:// group-bucket/My http://s3.amazonaws.com/group-bucket/My Name/MyDirectory/' instead of 's3://s3.amazonaws.com/group-bucket/My Name/MyDirectory/'. Florent On Fri, Oct 30, 2020 at 10:43 PM SimonHGR simonhgr@users.sourceforge.net wrote: Hi all, I've been using "duck" in the command line version to communicate with...
-
SimonHGR posted a comment on discussion Open Discussion
Hi all, I've been using "duck" in the command line version to communicate with AWS S3, so I know my access and secret keys are good, plus when I ran through "s3cmd --configure" it tested my keys and reported "Success. Your access key and secret key worked fine :-)" However, when I try to do anything myself, it fails, claiming access denied. I'm wondering if there's something I don't understand about s3 that duck was hiding, or perhaps I'm fundamentally misunderstanding how s3cmd command lines should...
-
Gopikrishna modified a comment on discussion Open Discussion
Hi All I am using latest s3cmd 2.1.0 and modified s3cfg file with host_base=servera <mymachine> host_buckets=%(bucket)s.servera </mymachine>
-
Hi All I am using latest s3cmd 2.1.0 and modified s3cfg file with host_base=servera <mymachine> host_buckets=%(bucket)s.servera when I try to list the buckets info  when I tried to create the bucket I am getting the below error  can you...
-
Hi All I am using latest s3cmd 2.1.0 and modified s3cfg file with host_base=servera <mymachine> host_buckets=%(bucket)s.servera when I try to list the buckets info  when I tried to create the bucket I am getting the below error can you please help to resolve this issue</mymachine>
-
Martin posted a comment on discussion Open Discussion
Hello, I need help uploading a txt file to an internal S3 style bucket. It works in graphical mode but I need to launch it in command line mode using PowerShell with the AWS module tools that I installed. I have the following information: file name: releasenote.txt file location: c: \ temp bucket name: platon-test Endpoint: gtm-vdr-bureautique Access Key: xxxxx1xxxxx Secretkey: xxxxx2xxxxx Thanks if anyone could help me over this.
-
ning li posted a comment on discussion Open Discussion
Hello, I want to sync two buckets using the s3cmd sync command for files that match a certain pattern. Both buckets already exist. My code snippet is: s3cmd sync --rinclude '^[a-zA-Z_]*._2019_09_\d*.avro' \ s3://bucket_a/path/aa \ s3://bucket_b/ I expected the command to include only files that have the '2019_09' string, but it ends up syncing everything. I've tested my regex elsewhere and it seems to work. Wondering if I should use a different variant of regex? Or am I miss-using the rinclude parameter?...
-
Roland Dunker posted a comment on ticket #182
thank you very much. Von: Florent Viard fviard@users.sourceforge.net Gesendet: Mittwoch, 8. April 2020 02:15 An: [s3tools:bugs] 182@bugs.s3tools.p.re.sourceforge.net Betreff: [s3tools:bugs] #182 Error in s3cfg parser for add_headers status: open --> closed-fixed Comment: Thank you for your report. Indeed, there was a bug. I duplicated your issue on the following ticket of github: https://github.com/s3tools/s3cmd/issues/1087 (fyi, it is on the github platform that the main developpement is going on...
-
Error in s3cfg parser for add_headers
-
Thank you for your report. Indeed, there was a bug. I duplicated your issue on the following ticket of github: https://github.com/s3tools/s3cmd/issues/1087 (fyi, it is on the github platform that the main developpement is going on now) I have pushed a fix for this issue that is in the "MASTER" version in github, but if you want to use a release with the issue fixed, please wait for the next release to come, that will probably be 2.2.0.
-
s3cmd falls back to sigv2 when underscore is in bucket name
-
Issue solved in release 2.1.0.
-
Alec Taylor posted a comment on ticket #165
My /etc/magic is empty on Ubuntu 14.10 x64. I am using s3cmd 1.5.2. Ended up using aws s3 sync which worked right off the mark: usage: aws s3 sync <LocalPath> <S3Path> or <S3Path> <LocalPath> or <S3Path> <S3Path>
-
-
-
-
-
-
-
L J created ticket #183
s3cmd falls back to sigv2 when underscore is in bucket name
-
When using s3cmd to upload to a bucket where the name has an underscore in it, it falls back to signature_v2, and when the bucket has no underscore it uses signature_v4. I dont think buckets with underscores and capitals need to be changed or not used, its just that new buckets can not be created with those in the names. I have tested this on the latest version (2.0.2). I see under 2.0.1 a fix for upper-case: Fixed signature issues due to upper-case in hostname I wonder if fixing the underscore was...
-
Error in s3cfg parser for add_headers
-
Hi Vimal, I guess that you are running an old version of s3cmd as we have added fixes to not fail the info command in the case that such things are not supported. You can try the most up to date version from github: https://github.com/s3tools/s3cmd If that doesn't solve your issue, please open an issue there, with a backtrace that you would generate calling s3cmd with the "-d" flag. Regards, Florent On Fri, Mar 8, 2019 at 10:56 PM Vimal Kaneria vkaneria@users.sourceforge.net wrote: I am running the...
-
Vimal Kaneria posted a comment on discussion Open Discussion
I am running the s3cmd info command against Hitachi's HCP which supports S3 functionality. However, it is failing to return the proper metadata information. On the HCP side we see the s3cmd running the followinq queries GET /testNS1/?location HTTP/1.1" 200 124 TestTenant@hs3 35 220 0 GET /testNS1/?requestPayment HTTP/1.1"501 281 TestTenant@hs3 6 220 0 GET /testNS1/?lifecycle HTTP/1.1"404 257 TestTenant@hs3 7 220 0 GET /testNS1/?policy HTTP/1.1"501 281 TestTenant@hs3 6 220 0 GET /testNS1/?cors HTTP/1.1"403...
-
Giuseppe Lo Re posted a comment on discussion Open Discussion
Hi, I would like to use the temporary urls to allow public/temporary access to objects. Coud anyone send me an example of out to use the signurl command? Is it the right command to use? Thanks in advance, Giuseppe
-
Hi, I would like to use the temporary urls to allow public/temporary access to objects. Coud anyone send me an example of out to use the signurl command? Is it the right command to use? Thanks in advance, Giuseppe
-
Petr Slansky posted a comment on ticket #181
This is a patch. Could you review it and apply it? I checked the code and I see that file mode is set only in "sync" but not in "get" command. Maybe that "file mode" should not be set for files in "sync" but set only to empty directories. One more point, when I try "s3cmd get s3://s3cmd-bug-demo-us-east-1 --recursive", directory permissions are set correctly, so I guess that all code that plays with "mode" can be removed from "sync" section because that part is handled by "os" or python library......
-
"s3cmd get" sets permissions correctly (to 664), I like it: $ s3cmd get s3://s3cmd-bug-demo-us-east-1/hello-s3cmd.txt ./hello-get.txt download: 's3://s3cmd-bug-demo-us-east-1/hello-s3cmd.txt' -> './hello-get.txt' [1 of 1] 111 of 111 100% in 0s 191.82 B/s done $ ls -l hello-get.txt -rw-rw-r-- 1 user user 111 Oct 3 09:18 hello-get.txt $ umask 0002
-
s3cmd sync, permissions are not set correctly
-
seller mobile modified a comment on discussion Open Discussion
E-Commerce software is a tool that drives all online store processes, helping owners manage inventory, add or remove products, process payments, compute taxes and fulfill orders, among many others. It offers all the things needed to run a retail site, effectively simplifying online store management. If you want ecommerce software service and tools for amazon seller we at seller mobile provide you this service. You may have a look at our website in order to get to know about our service. SellerMobile...
-
E-Commerce software is a tool that drives all online store processes, helping owners manage inventory, add or remove products, process payments, compute taxes and fulfill orders, among many others. It offers all the things needed to run a retail site, effectively simplifying online store management. If you want ecommerce software service and tools for amazon seller we at seller mobile provide you this service. You may have a look at our website in order to get to know about our service. SellerMobile...
-
-
-
-
-
-
-
-
Hi Greg, For that you don't have to worry. s3cmd is already used in a few embedded linux devices and especially "NAS". In addition with python, the only dependency is "python-dateutil" and eventually python magic. Regards, Florent On Fri, Jun 22, 2018 at 7:57 PM, Greg Schaub gschaub@users.sourceforge.net wrote: I am considering creating s3cmd as a package for OpenWrt (open source router firmware). Since this is an embedded linux device, storage is at a premium; I want to install only as much of Python...
-
Tim Baverstock created ticket #180
Top-level large bucket listing is very slow.
-
Greg Schaub posted a comment on discussion Open Discussion
I am considering creating s3cmd as a package for OpenWrt (open source router firmware). Since this is an embedded linux device, storage is at a premium; I want to install only as much of Python as absolutely required. OpenWrt currently has several Python packages available that are less than "Full". I've built s3cmd previously and love the product, but system resources weren't an issue at that time. Regards...Greg
-
Benoit Poirey posted a comment on discussion Open Discussion
Thank you for your answer :-)
-
Hi, Unfortunately, s3cmd doesn't support any operation related to "versions" of files. That is on the todo list of nice features to add. Regards, Florent On Fri, Jun 1, 2018 at 11:58 AM, Benoit Poirey onclebendusud@users.sourceforge.net wrote: Hello We have a S3 compatible bucket. I could successfully manage its content with s3cmd command lines. But I didn’t find any option to get a specific version of a file from the bucket. How could I manage the “version id” with s3cmd ? Thanks [http://corporate.airfrance.com/sites/default/images/af_...
-
Hello We have a S3 compatible bucket with versionning activated. I could successfully manage its content with s3cmd command lines. But I didn’t find any option to get a specific version of a file from the bucket. How could I manage the “version id” with s3cmd ? Thanks
-
Hello We have a S3 compatible bucket. I could successfully manage its content with s3cmd command lines. But I didn’t find any option to get a specific version of a file from the bucket. How could I manage the “version id” with s3cmd ? Thanks
-
Hello We have a S3 compatible bucket. I could successfully manage its content with s3cmd command lines. But I didn’t find any option to get a specific version of a file from the bucket. How could I manage the “version id” with s3cmd ? Thanks [http://corporate.airfrance.com/sites/default/images/af_logo_france-is-in-the-air.gif]http://www.airfrance.com Accédez aux meilleurs tarifs Air France, gérez vos réservations et enregistrez-vous en ligne sur http://www.airfrance.com Find best Air France fares,...
-
Hi Hakan, For the regular failures, it is probably transient aws or network failures. I'm not sure if it is related, but i'm currently trying to improve the retry code. For the issue with resuming upload multiple times, can you open an issue on github: https://github.com/s3tools/s3cmd So we can investigate it. Thank you, Florent On Thu, Feb 22, 2018 at 6:30 PM, Hakan Duran <hakayova@users.sourceforge.net wrote: Hi all, I have a 85 GB file that I am uploading to AWS S3 via s3cmd. It is automatically...
-
Hakan Duran posted a comment on discussion Open Discussion
Hi all, I have a 85 GB file that I am uploading to AWS S3 via s3cmd. It is automatically converted to a multipart upload as expected but it has failed several times over the course of last 10-15 days. The total number parts is around 5000 and the first failure happened while uploading the part number 1000 or so. When I resumed the upload using the upload ID for the first time, it resumed from that part (1000) but the upload again failed at part 3800 or so. Unfortunately, the upload ID did not change,...
-
Djangofan posted a comment on discussion Open Discussion
Is s3cmd compatible with a local dockerized S3 ? If so, how do I configure .s3cfg ? My docker image is registered like so: docker run -p 4569:4569 --name s3-local --restart unless-stopped -d lphoward/fake-s3 I'm on a Mac and my Java code connects to my S3 at address 192.168.99.100:4569 I looked around on Google and couldn't find any existing software that allows browsing a local S3 for development. Is there one?
-
Christie Craig posted a comment on discussion Open Discussion
I was doing a quick look around, and wasn't able to find out concretely that there is no way to utilize an IAM role rather than a user. Am I correct in that roles are not supported? Is there any future aspirations to include IAM role support instead of baked-in credentials for a user?
-
-
-
-
-
-
-
-
-
Martin Schlatter posted a comment on ticket #81
Same here. Excluding directories does not work.
-
Hi, The service is probably not supporting the last signature version of Amazon (v4). Edit your configuration file, (by default generated in ~/.s3cmd ) and Change the 'signature_v2' option to 'True'. This will force to only use the old version signature scheme and not the most recent one. Regards, Florent Le 20 sept. 2017 8:40 AM, "massimo" mgabby@users.sf.net a écrit : Hi, i have installed s3cmd on my ubuntu 16.04 server. I configured it for use Aruba object storage, italian compatible S3 service....
-
massimo posted a comment on discussion Open Discussion
Hi, i have installed s3cmd on my ubuntu 16.04 server. I configured it for use Aruba object storage, italian compatible S3 service. The wizard setup finished ok, and when I try comand "s3cmd ls" i received the buckets list. When i try to use "s3cmd la" I receive, "ERROR: S3 error: 403 (AuthMethodNotRecognized)". In debug mode I receive: DEBUG: s3cmd version 1.6.1 DEBUG: ConfigParser: Reading file '/root/.s3cfg' DEBUG: ConfigParser: access_key->st...12_chars...p DEBUG: ConfigParser: access_token->...
-
Erike Magegere created ticket #179
first file of a put (singular or recursive) is uploaded twice
-
Sudip Saha posted a comment on discussion Open Discussion
Hi, I am trying to modify my bucket policy, so that any object in it are by default publicly readable by http Get. I am trying to use the command: s3cmd setpolicy "policyfile" "s3://bucket". Could you give me an example of a simple policyfile? Note that, my hostbase is not amazon. I tried this policyfile, { "Statement": [{ "Sid": "MakeItPublic", "Effect": "Allow", "Principal": "", "Action": "s3:GetObject", "Resource": "" }] } But, it failed with the error: S3 error: 400 (InvalidArgument).
-
-
-
-
-
Eric Brandl modified a comment on discussion Open Discussion
Good afternoon, I have a set of files I need to keep synced up to S3 that need to...
-
Good afternoon, I have a set of files I need to keep synced up to S3 that need to...
-
Vikas posted a comment on discussion Open Discussion
s3cmd must not be depricated as of now, as it has some features which are still not...
-
dsbcpas posted a comment on discussion Open Discussion
Vikas; Your issue sounds a lot like my posted issue #846. Also see Florent last reply...