From: Joe A. <jo...@ne...> - 2010-11-12 22:26:14
|
Hello, Is there a way to get s3cmd to not fuss over matching checksums on live log files and files that are constantly being written to? I realize that I can exclude them, but there are many of them that are dynamically created, I'd rather not list them individually. What I'm finding is that because more information is being written to the source file by the time the upload is complete, I get stuck in an endless loop of uploading the file over and over again until checksums match. If giving up after x number of failures is a possibility and just accepting what has been uploaded that is cool too, but what would be awesome is to perhaps skip the comparison of final checksums for certain directories. Has anybody else run into this problem? Any suggestions for techniques for working around this sort of problem? -- Joe Auty, NetMusician NetMusician helps musicians, bands and artists create beautiful, professional, custom designed, career-essential websites that are easy to maintain and to integrate with popular social networks. www.netmusician.org <http://www.netmusician.org> jo...@ne... <mailto:jo...@ne...> |
From: Michal L. <ml...@lo...> - 2010-11-13 20:13:13
|
Hi Joe, currently it isn't possible to skip checksums comparison. If all your log files are in the same directory or have some common name pattern, e.g. *.log, you can exclude them with --exclude \*.log or some other variant of --exclude-from/--rexclude/--rexclude-from. In the future there will be an option to not compare the checksums on sync, but I'm not sure if it's a good idea to skip comparing them on upload... Michal On 11/13/2010 11:26 AM, Joe Auty wrote: > Hello, > > Is there a way to get s3cmd to not fuss over matching checksums on > live log files and files that are constantly being written to? I > realize that I can exclude them, but there are many of them that are > dynamically created, I'd rather not list them individually. > > What I'm finding is that because more information is being written to > the source file by the time the upload is complete, I get stuck in an > endless loop of uploading the file over and over again until checksums > match. If giving up after x number of failures is a possibility and > just accepting what has been uploaded that is cool too, but what would > be awesome is to perhaps skip the comparison of final checksums for > certain directories. > > Has anybody else run into this problem? Any suggestions for techniques > for working around this sort of problem? > > > -- > Joe Auty, NetMusician > NetMusician helps musicians, bands and artists create beautiful, > professional, custom designed, career-essential websites that are easy > to maintain and to integrate with popular social networks. > www.netmusician.org <http://www.netmusician.org> > jo...@ne... <mailto:jo...@ne...> > > > ------------------------------------------------------------------------------ > Centralized Desktop Delivery: Dell and VMware Reference Architecture > Simplifying enterprise desktop deployment and management using > Dell EqualLogic storage and VMware View: A highly scalable, end-to-end > client virtualization framework. Read more! > http://p.sf.net/sfu/dell-eql-dev2dev > > > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general |
From: Joe A. <jo...@ne...> - 2010-11-15 08:06:51
|
Thanks for this! Another problem I'm running into is that if a file is deleted on the source before it is sent, the sync fails and spews out an error message. This is a problem with data such as email where an email message might be deleted in real time, or perhaps a log file turned over or something. I don't see a way to force the sync job to continue on these sorts of errors, but I thought I would double check first? Michal Ludvig wrote: > Hi Joe, > > currently it isn't possible to skip checksums comparison. > > If all your log files are in the same directory or have some common > name pattern, e.g. *.log, you can exclude them with --exclude \*.log > or some other variant of --exclude-from/--rexclude/--rexclude-from. > > In the future there will be an option to not compare the checksums on > sync, but I'm not sure if it's a good idea to skip comparing them on > upload... > > Michal > > On 11/13/2010 11:26 AM, Joe Auty wrote: >> Hello, >> >> Is there a way to get s3cmd to not fuss over matching checksums on >> live log files and files that are constantly being written to? I >> realize that I can exclude them, but there are many of them that are >> dynamically created, I'd rather not list them individually. >> >> What I'm finding is that because more information is being written to >> the source file by the time the upload is complete, I get stuck in an >> endless loop of uploading the file over and over again until >> checksums match. If giving up after x number of failures is a >> possibility and just accepting what has been uploaded that is cool >> too, but what would be awesome is to perhaps skip the comparison of >> final checksums for certain directories. >> >> Has anybody else run into this problem? Any suggestions for >> techniques for working around this sort of problem? >> >> >> -- >> Joe Auty, NetMusician >> NetMusician helps musicians, bands and artists create beautiful, >> professional, custom designed, career-essential websites that are >> easy to maintain and to integrate with popular social networks. >> www.netmusician.org <http://www.netmusician.org> >> jo...@ne... <mailto:jo...@ne...> >> >> >> ------------------------------------------------------------------------------ >> Centralized Desktop Delivery: Dell and VMware Reference Architecture >> Simplifying enterprise desktop deployment and management using >> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >> client virtualization framework. Read more! >> http://p.sf.net/sfu/dell-eql-dev2dev >> >> >> _______________________________________________ >> S3tools-general mailing list >> S3t...@li... >> https://lists.sourceforge.net/lists/listinfo/s3tools-general > > > ------------------------------------------------------------------------------ > Centralized Desktop Delivery: Dell and VMware Reference Architecture > Simplifying enterprise desktop deployment and management using > Dell EqualLogic storage and VMware View: A highly scalable, end-to-end > client virtualization framework. Read more! > http://p.sf.net/sfu/dell-eql-dev2dev > > > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general -- Joe Auty, NetMusician NetMusician helps musicians, bands and artists create beautiful, professional, custom designed, career-essential websites that are easy to maintain and to integrate with popular social networks. www.netmusician.org <http://www.netmusician.org> jo...@ne... <mailto:jo...@ne...> |
From: Joe A. <jo...@ne...> - 2010-11-15 08:17:41
|
Joe Auty wrote: > Thanks for this! > > Another problem I'm running into is that if a file is deleted on the > source before it is sent, the sync fails and spews out an error > message. This is a problem with data such as email where an email > message might be deleted in real time, or perhaps a log file turned > over or something. I don't see a way to force the sync job to continue > on these sorts of errors, but I thought I would double check first? > Sorry, this may not have been clear... What I meant here is if the file is deleted on the source *during the sync* process, i.e. prior to actually sending the file from the source to destination, a fatal error is generated. > > > > Michal Ludvig wrote: >> Hi Joe, >> >> currently it isn't possible to skip checksums comparison. >> >> If all your log files are in the same directory or have some common >> name pattern, e.g. *.log, you can exclude them with --exclude \*.log >> or some other variant of --exclude-from/--rexclude/--rexclude-from. >> >> In the future there will be an option to not compare the checksums on >> sync, but I'm not sure if it's a good idea to skip comparing them on >> upload... >> >> Michal >> >> On 11/13/2010 11:26 AM, Joe Auty wrote: >>> Hello, >>> >>> Is there a way to get s3cmd to not fuss over matching checksums on >>> live log files and files that are constantly being written to? I >>> realize that I can exclude them, but there are many of them that are >>> dynamically created, I'd rather not list them individually. >>> >>> What I'm finding is that because more information is being written >>> to the source file by the time the upload is complete, I get stuck >>> in an endless loop of uploading the file over and over again until >>> checksums match. If giving up after x number of failures is a >>> possibility and just accepting what has been uploaded that is cool >>> too, but what would be awesome is to perhaps skip the comparison of >>> final checksums for certain directories. >>> >>> Has anybody else run into this problem? Any suggestions for >>> techniques for working around this sort of problem? >>> >>> >>> -- >>> Joe Auty, NetMusician >>> NetMusician helps musicians, bands and artists create beautiful, >>> professional, custom designed, career-essential websites that are >>> easy to maintain and to integrate with popular social networks. >>> www.netmusician.org <http://www.netmusician.org> >>> jo...@ne... <mailto:jo...@ne...> >>> >>> >>> ------------------------------------------------------------------------------ >>> Centralized Desktop Delivery: Dell and VMware Reference Architecture >>> Simplifying enterprise desktop deployment and management using >>> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >>> client virtualization framework. Read more! >>> http://p.sf.net/sfu/dell-eql-dev2dev >>> >>> >>> _______________________________________________ >>> S3tools-general mailing list >>> S3t...@li... >>> https://lists.sourceforge.net/lists/listinfo/s3tools-general >> >> >> ------------------------------------------------------------------------------ >> Centralized Desktop Delivery: Dell and VMware Reference Architecture >> Simplifying enterprise desktop deployment and management using >> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >> client virtualization framework. Read more! >> http://p.sf.net/sfu/dell-eql-dev2dev >> >> >> _______________________________________________ >> S3tools-general mailing list >> S3t...@li... >> https://lists.sourceforge.net/lists/listinfo/s3tools-general > > > -- > Joe Auty, NetMusician > NetMusician helps musicians, bands and artists create beautiful, > professional, custom designed, career-essential websites that are easy > to maintain and to integrate with popular social networks. > www.netmusician.org <http://www.netmusician.org> > jo...@ne... <mailto:jo...@ne...> > > > ------------------------------------------------------------------------------ > Centralized Desktop Delivery: Dell and VMware Reference Architecture > Simplifying enterprise desktop deployment and management using > Dell EqualLogic storage and VMware View: A highly scalable, end-to-end > client virtualization framework. Read more! > http://p.sf.net/sfu/dell-eql-dev2dev > > > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general -- Joe Auty, NetMusician NetMusician helps musicians, bands and artists create beautiful, professional, custom designed, career-essential websites that are easy to maintain and to integrate with popular social networks. www.netmusician.org <http://www.netmusician.org> jo...@ne... <mailto:jo...@ne...> |
From: Michal L. <ml...@lo...> - 2010-11-15 11:40:53
|
Hi Joe, this problem has been fixed in s3cmd 1.0.0(-rc1). I hope you're not observing the crashes with this newest s3cmd, are you? See http://s3tools.org/s3cmd-100rc1-released for more info on what has been changed/fixed. Michal On 11/15/2010 09:06 PM, Joe Auty wrote: > Thanks for this! > > Another problem I'm running into is that if a file is deleted on the > source before it is sent, the sync fails and spews out an error > message. This is a problem with data such as email where an email > message might be deleted in real time, or perhaps a log file turned > over or something. I don't see a way to force the sync job to continue > on these sorts of errors, but I thought I would double check first? > > > > > Michal Ludvig wrote: >> Hi Joe, >> >> currently it isn't possible to skip checksums comparison. >> >> If all your log files are in the same directory or have some common >> name pattern, e.g. *.log, you can exclude them with --exclude \*.log >> or some other variant of --exclude-from/--rexclude/--rexclude-from. >> >> In the future there will be an option to not compare the checksums on >> sync, but I'm not sure if it's a good idea to skip comparing them on >> upload... >> >> Michal >> >> On 11/13/2010 11:26 AM, Joe Auty wrote: >>> Hello, >>> >>> Is there a way to get s3cmd to not fuss over matching checksums on >>> live log files and files that are constantly being written to? I >>> realize that I can exclude them, but there are many of them that are >>> dynamically created, I'd rather not list them individually. >>> >>> What I'm finding is that because more information is being written >>> to the source file by the time the upload is complete, I get stuck >>> in an endless loop of uploading the file over and over again until >>> checksums match. If giving up after x number of failures is a >>> possibility and just accepting what has been uploaded that is cool >>> too, but what would be awesome is to perhaps skip the comparison of >>> final checksums for certain directories. >>> >>> Has anybody else run into this problem? Any suggestions for >>> techniques for working around this sort of problem? >>> >>> >>> -- >>> Joe Auty, NetMusician >>> NetMusician helps musicians, bands and artists create beautiful, >>> professional, custom designed, career-essential websites that are >>> easy to maintain and to integrate with popular social networks. >>> www.netmusician.org <http://www.netmusician.org> >>> jo...@ne... <mailto:jo...@ne...> >>> >>> >>> ------------------------------------------------------------------------------ >>> Centralized Desktop Delivery: Dell and VMware Reference Architecture >>> Simplifying enterprise desktop deployment and management using >>> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >>> client virtualization framework. Read more! >>> http://p.sf.net/sfu/dell-eql-dev2dev >>> >>> >>> _______________________________________________ >>> S3tools-general mailing list >>> S3t...@li... >>> https://lists.sourceforge.net/lists/listinfo/s3tools-general >> >> >> ------------------------------------------------------------------------------ >> Centralized Desktop Delivery: Dell and VMware Reference Architecture >> Simplifying enterprise desktop deployment and management using >> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >> client virtualization framework. Read more! >> http://p.sf.net/sfu/dell-eql-dev2dev >> >> >> _______________________________________________ >> S3tools-general mailing list >> S3t...@li... >> https://lists.sourceforge.net/lists/listinfo/s3tools-general > > > -- > Joe Auty, NetMusician > NetMusician helps musicians, bands and artists create beautiful, > professional, custom designed, career-essential websites that are easy > to maintain and to integrate with popular social networks. > www.netmusician.org <http://www.netmusician.org> > jo...@ne... <mailto:jo...@ne...> > > > ------------------------------------------------------------------------------ > Centralized Desktop Delivery: Dell and VMware Reference Architecture > Simplifying enterprise desktop deployment and management using > Dell EqualLogic storage and VMware View: A highly scalable, end-to-end > client virtualization framework. Read more! > http://p.sf.net/sfu/dell-eql-dev2dev > > > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general |
From: Joe A. <jo...@ne...> - 2010-11-15 18:22:25
|
Michal Ludvig wrote: > Hi Joe, > > this problem has been fixed in s3cmd 1.0.0(-rc1). I hope you're not > observing the crashes with this newest s3cmd, are you? > > See http://s3tools.org/s3cmd-100rc1-released for more info on what has > been changed/fixed. I actually am using the 1.0.0 rc1 version :( > > Michal > > On 11/15/2010 09:06 PM, Joe Auty wrote: >> Thanks for this! >> >> Another problem I'm running into is that if a file is deleted on the >> source before it is sent, the sync fails and spews out an error >> message. This is a problem with data such as email where an email >> message might be deleted in real time, or perhaps a log file turned >> over or something. I don't see a way to force the sync job to >> continue on these sorts of errors, but I thought I would double check >> first? >> >> >> >> >> Michal Ludvig wrote: >>> Hi Joe, >>> >>> currently it isn't possible to skip checksums comparison. >>> >>> If all your log files are in the same directory or have some common >>> name pattern, e.g. *.log, you can exclude them with --exclude \*.log >>> or some other variant of --exclude-from/--rexclude/--rexclude-from. >>> >>> In the future there will be an option to not compare the checksums >>> on sync, but I'm not sure if it's a good idea to skip comparing them >>> on upload... >>> >>> Michal >>> >>> On 11/13/2010 11:26 AM, Joe Auty wrote: >>>> Hello, >>>> >>>> Is there a way to get s3cmd to not fuss over matching checksums on >>>> live log files and files that are constantly being written to? I >>>> realize that I can exclude them, but there are many of them that >>>> are dynamically created, I'd rather not list them individually. >>>> >>>> What I'm finding is that because more information is being written >>>> to the source file by the time the upload is complete, I get stuck >>>> in an endless loop of uploading the file over and over again until >>>> checksums match. If giving up after x number of failures is a >>>> possibility and just accepting what has been uploaded that is cool >>>> too, but what would be awesome is to perhaps skip the comparison of >>>> final checksums for certain directories. >>>> >>>> Has anybody else run into this problem? Any suggestions for >>>> techniques for working around this sort of problem? >>>> >>>> >>>> -- >>>> Joe Auty, NetMusician >>>> NetMusician helps musicians, bands and artists create beautiful, >>>> professional, custom designed, career-essential websites that are >>>> easy to maintain and to integrate with popular social networks. >>>> www.netmusician.org <http://www.netmusician.org> >>>> jo...@ne... <mailto:jo...@ne...> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Centralized Desktop Delivery: Dell and VMware Reference Architecture >>>> Simplifying enterprise desktop deployment and management using >>>> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >>>> client virtualization framework. Read more! >>>> http://p.sf.net/sfu/dell-eql-dev2dev >>>> >>>> >>>> _______________________________________________ >>>> S3tools-general mailing list >>>> S3t...@li... >>>> https://lists.sourceforge.net/lists/listinfo/s3tools-general >>> >>> >>> ------------------------------------------------------------------------------ >>> Centralized Desktop Delivery: Dell and VMware Reference Architecture >>> Simplifying enterprise desktop deployment and management using >>> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >>> client virtualization framework. Read more! >>> http://p.sf.net/sfu/dell-eql-dev2dev >>> >>> >>> _______________________________________________ >>> S3tools-general mailing list >>> S3t...@li... >>> https://lists.sourceforge.net/lists/listinfo/s3tools-general >> >> >> -- >> Joe Auty, NetMusician >> NetMusician helps musicians, bands and artists create beautiful, >> professional, custom designed, career-essential websites that are >> easy to maintain and to integrate with popular social networks. >> www.netmusician.org <http://www.netmusician.org> >> jo...@ne... <mailto:jo...@ne...> >> >> >> ------------------------------------------------------------------------------ >> Centralized Desktop Delivery: Dell and VMware Reference Architecture >> Simplifying enterprise desktop deployment and management using >> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >> client virtualization framework. Read more! >> http://p.sf.net/sfu/dell-eql-dev2dev >> >> >> _______________________________________________ >> S3tools-general mailing list >> S3t...@li... >> https://lists.sourceforge.net/lists/listinfo/s3tools-general > > > ------------------------------------------------------------------------------ > Centralized Desktop Delivery: Dell and VMware Reference Architecture > Simplifying enterprise desktop deployment and management using > Dell EqualLogic storage and VMware View: A highly scalable, end-to-end > client virtualization framework. Read more! > http://p.sf.net/sfu/dell-eql-dev2dev > > > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general -- Joe Auty, NetMusician NetMusician helps musicians, bands and artists create beautiful, professional, custom designed, career-essential websites that are easy to maintain and to integrate with popular social networks. www.netmusician.org <http://www.netmusician.org> jo...@ne... <mailto:jo...@ne...> |
From: Joe A. <jo...@ne...> - 2010-11-15 18:25:37
|
Joe Auty wrote: > Michal Ludvig wrote: >> Hi Joe, >> >> this problem has been fixed in s3cmd 1.0.0(-rc1). I hope you're not >> observing the crashes with this newest s3cmd, are you? >> >> See http://s3tools.org/s3cmd-100rc1-released for more info on what >> has been changed/fixed. > > I actually am using the 1.0.0 rc1 version :( > Please let me know how I can be of help in providing additional information, testing stuff, or what have you... >> >> Michal >> >> On 11/15/2010 09:06 PM, Joe Auty wrote: >>> Thanks for this! >>> >>> Another problem I'm running into is that if a file is deleted on the >>> source before it is sent, the sync fails and spews out an error >>> message. This is a problem with data such as email where an email >>> message might be deleted in real time, or perhaps a log file turned >>> over or something. I don't see a way to force the sync job to >>> continue on these sorts of errors, but I thought I would double >>> check first? >>> >>> >>> >>> >>> Michal Ludvig wrote: >>>> Hi Joe, >>>> >>>> currently it isn't possible to skip checksums comparison. >>>> >>>> If all your log files are in the same directory or have some common >>>> name pattern, e.g. *.log, you can exclude them with --exclude >>>> \*.log or some other variant of >>>> --exclude-from/--rexclude/--rexclude-from. >>>> >>>> In the future there will be an option to not compare the checksums >>>> on sync, but I'm not sure if it's a good idea to skip comparing >>>> them on upload... >>>> >>>> Michal >>>> >>>> On 11/13/2010 11:26 AM, Joe Auty wrote: >>>>> Hello, >>>>> >>>>> Is there a way to get s3cmd to not fuss over matching checksums on >>>>> live log files and files that are constantly being written to? I >>>>> realize that I can exclude them, but there are many of them that >>>>> are dynamically created, I'd rather not list them individually. >>>>> >>>>> What I'm finding is that because more information is being written >>>>> to the source file by the time the upload is complete, I get stuck >>>>> in an endless loop of uploading the file over and over again until >>>>> checksums match. If giving up after x number of failures is a >>>>> possibility and just accepting what has been uploaded that is cool >>>>> too, but what would be awesome is to perhaps skip the comparison >>>>> of final checksums for certain directories. >>>>> >>>>> Has anybody else run into this problem? Any suggestions for >>>>> techniques for working around this sort of problem? >>>>> >>>>> >>>>> -- >>>>> Joe Auty, NetMusician >>>>> NetMusician helps musicians, bands and artists create beautiful, >>>>> professional, custom designed, career-essential websites that are >>>>> easy to maintain and to integrate with popular social networks. >>>>> www.netmusician.org <http://www.netmusician.org> >>>>> jo...@ne... <mailto:jo...@ne...> >>>>> >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> Centralized Desktop Delivery: Dell and VMware Reference Architecture >>>>> Simplifying enterprise desktop deployment and management using >>>>> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >>>>> client virtualization framework. Read more! >>>>> http://p.sf.net/sfu/dell-eql-dev2dev >>>>> >>>>> >>>>> _______________________________________________ >>>>> S3tools-general mailing list >>>>> S3t...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/s3tools-general >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Centralized Desktop Delivery: Dell and VMware Reference Architecture >>>> Simplifying enterprise desktop deployment and management using >>>> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >>>> client virtualization framework. Read more! >>>> http://p.sf.net/sfu/dell-eql-dev2dev >>>> >>>> >>>> _______________________________________________ >>>> S3tools-general mailing list >>>> S3t...@li... >>>> https://lists.sourceforge.net/lists/listinfo/s3tools-general >>> >>> >>> -- >>> Joe Auty, NetMusician >>> NetMusician helps musicians, bands and artists create beautiful, >>> professional, custom designed, career-essential websites that are >>> easy to maintain and to integrate with popular social networks. >>> www.netmusician.org <http://www.netmusician.org> >>> jo...@ne... <mailto:jo...@ne...> >>> >>> >>> ------------------------------------------------------------------------------ >>> Centralized Desktop Delivery: Dell and VMware Reference Architecture >>> Simplifying enterprise desktop deployment and management using >>> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >>> client virtualization framework. Read more! >>> http://p.sf.net/sfu/dell-eql-dev2dev >>> >>> >>> _______________________________________________ >>> S3tools-general mailing list >>> S3t...@li... >>> https://lists.sourceforge.net/lists/listinfo/s3tools-general >> >> >> ------------------------------------------------------------------------------ >> Centralized Desktop Delivery: Dell and VMware Reference Architecture >> Simplifying enterprise desktop deployment and management using >> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >> client virtualization framework. Read more! >> http://p.sf.net/sfu/dell-eql-dev2dev >> >> >> _______________________________________________ >> S3tools-general mailing list >> S3t...@li... >> https://lists.sourceforge.net/lists/listinfo/s3tools-general > > > -- > Joe Auty, NetMusician > NetMusician helps musicians, bands and artists create beautiful, > professional, custom designed, career-essential websites that are easy > to maintain and to integrate with popular social networks. > www.netmusician.org <http://www.netmusician.org> > jo...@ne... <mailto:jo...@ne...> > > > ------------------------------------------------------------------------------ > Centralized Desktop Delivery: Dell and VMware Reference Architecture > Simplifying enterprise desktop deployment and management using > Dell EqualLogic storage and VMware View: A highly scalable, end-to-end > client virtualization framework. Read more! > http://p.sf.net/sfu/dell-eql-dev2dev > > > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general -- Joe Auty, NetMusician NetMusician helps musicians, bands and artists create beautiful, professional, custom designed, career-essential websites that are easy to maintain and to integrate with popular social networks. www.netmusician.org <http://www.netmusician.org> jo...@ne... <mailto:jo...@ne...> |
From: Michal L. <ml...@lo...> - 2010-11-16 10:45:54
|
On 11/16/2010 07:25 AM, Joe Auty wrote: > Joe Auty wrote: >> Michal Ludvig wrote: >>> Hi Joe, >>> >>> this problem has been fixed in s3cmd 1.0.0(-rc1). I hope you're not >>> observing the crashes with this newest s3cmd, are you? >>> >>> See http://s3tools.org/s3cmd-100rc1-released for more info on what >>> has been changed/fixed. >> >> I actually am using the 1.0.0 rc1 version :( >> > Please let me know how I can be of help in providing additional > information, testing stuff, or what have you... Hi Joe. The actual crash info that s3cmd prints out would actually be useful... Thanks! Michal |
From: Joe A. <jo...@ne...> - 2010-11-16 13:36:39
|
Here is the error message and trace info: > Your "cron" job on myserver > /usr/bin/s3cmd sync --rr --delete-removed --exclude 'maillogs/maillog' /nm/dorian/ s3://nmbackup/doriandata/ > > produced the following output: > > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > An unexpected error has occurred. > Please report the following lines to: > s3t...@li... > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > > Problem: IOErr: [Errno 2] No such file or directory: '/nm/dorian/Maildirs/christian/Maildir/.Deleted Messages/cur/1287414714.M371894P65638V0400FF01I00127FD0_1.dorian.netmusician.org,S=9171:2,S' > S3cmd: 1.0.0-rc1 > > Traceback (most recent call last): > File "/usr/bin/s3cmd", line 1899, in ? > main() > File "/usr/bin/s3cmd", line 1843, in main > cmd_func(args) > File "/usr/bin/s3cmd", line 1118, in cmd_sync > return cmd_sync_local2remote(args) > File "/usr/bin/s3cmd", line 1034, in cmd_sync_local2remote > local_list, remote_list, existing_list = _compare_filelists(local_list, remote_list, True) > File "/usr/bin/s3cmd", line 803, in _compare_filelists > src_md5 = Utils.hash_file_md5(src_list[file]['full_name']) > File "/usr/lib/python2.4/site-packages/S3/Utils.py", line 210, in hash_file_md5 > f = open(filename, "rb") > IOError: [Errno 2] No such file or directory: '/nm/dorian/Maildirs/christian/Maildir/.Deleted Messages/cur/1287414714.M371894P65638V0400FF01I00127FD0_1.dorian.netmusician.org,S=9171:2,S' > > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > An unexpected error has occurred. > Please report the above lines to: > s3t...@li... > !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > Joe Auty wrote: > Joe Auty wrote: >> Michal Ludvig wrote: >>> Hi Joe, >>> >>> this problem has been fixed in s3cmd 1.0.0(-rc1). I hope you're not >>> observing the crashes with this newest s3cmd, are you? >>> >>> See http://s3tools.org/s3cmd-100rc1-released for more info on what >>> has been changed/fixed. >> >> I actually am using the 1.0.0 rc1 version :( >> > Please let me know how I can be of help in providing additional > information, testing stuff, or what have you... > > >>> >>> Michal >>> >>> On 11/15/2010 09:06 PM, Joe Auty wrote: >>>> Thanks for this! >>>> >>>> Another problem I'm running into is that if a file is deleted on >>>> the source before it is sent, the sync fails and spews out an error >>>> message. This is a problem with data such as email where an email >>>> message might be deleted in real time, or perhaps a log file turned >>>> over or something. I don't see a way to force the sync job to >>>> continue on these sorts of errors, but I thought I would double >>>> check first? >>>> >>>> >>>> >>>> >>>> Michal Ludvig wrote: >>>>> Hi Joe, >>>>> >>>>> currently it isn't possible to skip checksums comparison. >>>>> >>>>> If all your log files are in the same directory or have some >>>>> common name pattern, e.g. *.log, you can exclude them with >>>>> --exclude \*.log or some other variant of >>>>> --exclude-from/--rexclude/--rexclude-from. >>>>> >>>>> In the future there will be an option to not compare the checksums >>>>> on sync, but I'm not sure if it's a good idea to skip comparing >>>>> them on upload... >>>>> >>>>> Michal >>>>> >>>>> On 11/13/2010 11:26 AM, Joe Auty wrote: >>>>>> Hello, >>>>>> >>>>>> Is there a way to get s3cmd to not fuss over matching checksums >>>>>> on live log files and files that are constantly being written to? >>>>>> I realize that I can exclude them, but there are many of them >>>>>> that are dynamically created, I'd rather not list them individually. >>>>>> >>>>>> What I'm finding is that because more information is being >>>>>> written to the source file by the time the upload is complete, I >>>>>> get stuck in an endless loop of uploading the file over and over >>>>>> again until checksums match. If giving up after x number of >>>>>> failures is a possibility and just accepting what has been >>>>>> uploaded that is cool too, but what would be awesome is to >>>>>> perhaps skip the comparison of final checksums for certain >>>>>> directories. >>>>>> >>>>>> Has anybody else run into this problem? Any suggestions for >>>>>> techniques for working around this sort of problem? >>>>>> >>>>>> >>>>>> -- >>>>>> Joe Auty, NetMusician >>>>>> NetMusician helps musicians, bands and artists create beautiful, >>>>>> professional, custom designed, career-essential websites that are >>>>>> easy to maintain and to integrate with popular social networks. >>>>>> www.netmusician.org <http://www.netmusician.org> >>>>>> jo...@ne... <mailto:jo...@ne...> >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------------------ >>>>>> Centralized Desktop Delivery: Dell and VMware Reference Architecture >>>>>> Simplifying enterprise desktop deployment and management using >>>>>> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >>>>>> client virtualization framework. Read more! >>>>>> http://p.sf.net/sfu/dell-eql-dev2dev >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> S3tools-general mailing list >>>>>> S3t...@li... >>>>>> https://lists.sourceforge.net/lists/listinfo/s3tools-general >>>>> >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> Centralized Desktop Delivery: Dell and VMware Reference Architecture >>>>> Simplifying enterprise desktop deployment and management using >>>>> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >>>>> client virtualization framework. Read more! >>>>> http://p.sf.net/sfu/dell-eql-dev2dev >>>>> >>>>> >>>>> _______________________________________________ >>>>> S3tools-general mailing list >>>>> S3t...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/s3tools-general >>>> >>>> >>>> -- >>>> Joe Auty, NetMusician >>>> NetMusician helps musicians, bands and artists create beautiful, >>>> professional, custom designed, career-essential websites that are >>>> easy to maintain and to integrate with popular social networks. >>>> www.netmusician.org <http://www.netmusician.org> >>>> jo...@ne... <mailto:jo...@ne...> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Centralized Desktop Delivery: Dell and VMware Reference Architecture >>>> Simplifying enterprise desktop deployment and management using >>>> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >>>> client virtualization framework. Read more! >>>> http://p.sf.net/sfu/dell-eql-dev2dev >>>> >>>> >>>> _______________________________________________ >>>> S3tools-general mailing list >>>> S3t...@li... >>>> https://lists.sourceforge.net/lists/listinfo/s3tools-general >>> >>> >>> ------------------------------------------------------------------------------ >>> Centralized Desktop Delivery: Dell and VMware Reference Architecture >>> Simplifying enterprise desktop deployment and management using >>> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >>> client virtualization framework. Read more! >>> http://p.sf.net/sfu/dell-eql-dev2dev >>> >>> >>> _______________________________________________ >>> S3tools-general mailing list >>> S3t...@li... >>> https://lists.sourceforge.net/lists/listinfo/s3tools-general >> >> >> -- >> Joe Auty, NetMusician >> NetMusician helps musicians, bands and artists create beautiful, >> professional, custom designed, career-essential websites that are >> easy to maintain and to integrate with popular social networks. >> www.netmusician.org <http://www.netmusician.org> >> jo...@ne... <mailto:jo...@ne...> >> >> >> ------------------------------------------------------------------------------ >> Centralized Desktop Delivery: Dell and VMware Reference Architecture >> Simplifying enterprise desktop deployment and management using >> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end >> client virtualization framework. Read more! >> http://p.sf.net/sfu/dell-eql-dev2dev >> >> >> _______________________________________________ >> S3tools-general mailing list >> S3t...@li... >> https://lists.sourceforge.net/lists/listinfo/s3tools-general > > > -- > Joe Auty, NetMusician > NetMusician helps musicians, bands and artists create beautiful, > professional, custom designed, career-essential websites that are easy > to maintain and to integrate with popular social networks. > www.netmusician.org <http://www.netmusician.org> > jo...@ne... <mailto:jo...@ne...> > > > ------------------------------------------------------------------------------ > Centralized Desktop Delivery: Dell and VMware Reference Architecture > Simplifying enterprise desktop deployment and management using > Dell EqualLogic storage and VMware View: A highly scalable, end-to-end > client virtualization framework. Read more! > http://p.sf.net/sfu/dell-eql-dev2dev > > > _______________________________________________ > S3tools-general mailing list > S3t...@li... > https://lists.sourceforge.net/lists/listinfo/s3tools-general -- Joe Auty, NetMusician NetMusician helps musicians, bands and artists create beautiful, professional, custom designed, career-essential websites that are easy to maintain and to integrate with popular social networks. www.netmusician.org <http://www.netmusician.org> jo...@ne... <mailto:jo...@ne...> |
From: Michal L. <ml...@lo...> - 2010-11-16 21:45:03
|
Hi Joe, thanks for that. Looks like it crashed during checksumming and not during the actual upload, that's what I fixed in rc1. I'll have a look at this one too. Michal |