Thread: Daily sync of hard disk without additional hard disk ?
Brought to you by:
thesun
From: renaud <re...@fa...> - 2007-06-11 18:57:21
|
Hello, I plan to use rsyncrypto in a production environment to sync data every night trough an internet link. Data are stored on a 400 GB disk and around 250 MB of new data is generated every day. I would like to avoid the use of an additional hard disk necessary to store encrypted data generated by rsyncrypto before they are synchronized with rsync. The simple idea would be : 1. compare the list of files on the source and on the destination based on name and time stamp of files, and generate a list of deleted and changed files on source 2. encrypt the changed files on source in a separate directory on the data hard disk (250 MB each day) 3. delete on destination the files who need to 4. sync the encrypted changed files to the destination with rsync 5. delete encrypted files on source keeping keyfiles files in keydir In this scheme we avoid the use of an additional hard disk in exchange of re-encrypting 250 MB of data every day. Name consistency between source and destination is also necessary but it is not a problem in my case. I guess that everybody who plan to backup an entire hard disk with rsyncrypto is facing this problem but I have not been able to find any resource on this subject. Mi question is : is there any tested and stable tool or script who already do this kind of things ? Drsync seems of no help for this. I'm using debian etch and rsyncrypto 0.12-1 (sorry, this is an old version, but i need debian stable) Best regards, Renaud Cabrol Hope you will decrypt my english ;-) |
From: Shachar S. <sh...@sh...> - 2007-06-11 20:16:39
|
renaud wrote: > The simple idea would be : > 1. compare the list of files on the source and on the destination based > on name and time stamp of files, and generate a list of deleted and > changed files on source > 2. encrypt the changed files on source in a separate directory on the > data hard disk (250 MB each day) > 3. delete on destination the files who need to > You mean files that were deleted on the source, right? > 4. sync the encrypted changed files to the destination with rsync > 5. delete encrypted files on source keeping keyfiles files in keydir > > In this scheme we avoid the use of an additional hard disk in exchange > of re-encrypting 250 MB of data every day. > I don't see how you re-encrypt beyond what rsyncrypto would do anyways. All that is different with this scheme is that you save on disk space, in return for manual decision on what has changed. > I guess that everybody who plan to backup an entire hard disk with > rsyncrypto is facing this problem but I have not been able to find any > resource on this subject. > Hey, I'm still in the "wow, people are using it" stage. Don't push me into "everybody use it a certain way" avenue just yet :-) > Mi question is : is there any tested and stable tool or script who > already do this kind of things ? > Not to my knowledge. > Drsync seems of no help for this. > (goes to check what drsync is) Seems like a tool that is a wrapper around rsync, but does things that rsync can do on its own. Strange. Either way, no, it does not seem like it's doing what you want, but it does seem like it's something that can be, relatively easily, adapted to do what you want. Out of curiosity, though, how are you going to manage the initial transfer? In chunks? > I'm using debian etch and rsyncrypto 0.12-1 (sorry, this is an old > version, but i need debian stable) > Last time I checked, Debian Etch had rsyncrypto 0.19 (http://packages.qa.debian.org/r/rsyncrypto.html). I really really really recommend you use 0.19, as one of the major changes was speed improvements, and you sound like you intend to back up really lots of stuff. Thanks, Shachar |
From: renaud <re...@fa...> - 2007-06-11 22:34:45
|
Shachar Shemesh wrote: > renaud wrote: > >> The simple idea would be : >> 1. compare the list of files on the source and on the destination based >> on name and time stamp of files, and generate a list of deleted and >> changed files on source >> 2. encrypt the changed files on source in a separate directory on the >> data hard disk (250 MB each day) >> 3. delete on destination the files who need to >> >> > You mean files that were deleted on the source, right? > Right. >> 4. sync the encrypted changed files to the destination with rsync >> 5. delete encrypted files on source keeping keyfiles files in keydir >> >> In this scheme we avoid the use of an additional hard disk in exchange >> of re-encrypting 250 MB of data every day. >> >> > I don't see how you re-encrypt beyond what rsyncrypto would do anyways. > All that is different with this scheme is that you save on disk space, > in return for manual decision on what has changed. > Right again, 250 MB of data have to be encrypted every day in both scheme... my mistake. To be more clear I plan to install this feature (rsyncrypto + rsync) on several server so i would like to avoid adding an additional disk dedicated only to rsyncrypto on each one, saving time and investment. >> I guess that everybody who plan to backup an entire hard disk with >> rsyncrypto is facing this problem but I have not been able to find any >> resource on this subject. >> >> > Hey, I'm still in the "wow, people are using it" stage. Don't push me > into "everybody use it a certain way" avenue just yet :-) > Rsyncrypto seems a very useful tool to me, massive use is on the way... >> Mi question is : is there any tested and stable tool or script who >> already do this kind of things ? >> >> > Not to my knowledge. > >> Drsync seems of no help for this. >> >> > (goes to check what drsync is) > Seems like a tool that is a wrapper around rsync, but does things that > rsync can do on its own. Strange. > Either way, no, it does not seem like it's doing what you want, but it > does seem like it's something that can be, relatively easily, adapted to > do what you want. > I'm not very at ease with perl and scripting in general, so I would like to avoid playing with this in a production environment. But I think I will have a look at it anyway and see what can be done. > Out of curiosity, though, how are you going to manage the initial > transfer? In chunks? > I first plan to do the initial complete sync locally with rsyncrypto onto a hard disk mounted in an external case and attached directly to the source server, then move physically this disk to it's final position in the destination server which is in the same town (Paris, France, to name it ;-). This is time consuming and it would be interesting to do the whole job remotely since I'm based at the remote point. Chunks would be a great idea to achieve this but again it needs a customized script. Generally I don't like to rely on tools that have not been well tested and validated (Specially the one I could create ;-( >> I'm using debian etch and rsyncrypto 0.12-1 (sorry, this is an old >> version, but i need debian stable) >> >> > Last time I checked, Debian Etch had rsyncrypto 0.19 > (http://packages.qa.debian.org/r/rsyncrypto.html). I really really > really recommend you use 0.19, as one of the major changes was speed > improvements, and you sound like you intend to back up really lots of stuff. > > That's true, I looked at version on the wrong machine who still runs sarge... > Thanks, > Shachar > I must say that I'm a freelance worker here in Paris, providing linux based server to small companies since a year and I'm actually testing this rsyncrypto + rsync solution to start offering also a simple remote backup service to my clients. I just realize that you are offering the same services through your Lingnu company in Israel. Hope this will not be a problem to continue exchanging on this list (I also saw your recommendations on your blog..) Anyway I will give feedback when the whole thing is up and running. Thank's a lot for your work on rsyncrypto, your help and fast answer. Renaud Cabrol |