rsyncrypto-devel — Development discussions, bug reports, and announcements for rsyncrypto

Re: destination cannot be drive other than C: on windows?

[Julian <jul...@gm...>]

I just wanted a quick code fix to get me out of a corner and to be honest i
I feel more comfortable knowing that this has now been properly dealt with
by Shachar (thanks). In any case, glad to help.


2008/5/7 Shachar Shemesh <sh...@sh...>:

> David V. wrote:
>
> > And Julian !
> > Thank you also to have worked on that and provided a patch ! :-)
> >
> > David V.
> >
>
> ACtually, there is a lot to learn from this.
>
> I didn't use Julian's patch. I wrote my own version of something not all
> too different (better handling of error codes, but otherwise similar).
>
> But that is not to say that Julian's work was meaningless. The fact he did
> send a list of what needs to change pointed me in the right direction,
> helping much in fixing the problem.
>
> Lesson: Send what you got, even if it's incomplete.
>
> Shachar
>

Re: destination cannot be drive other than C: on windows?

[David V. <dav...@gm...>]

And Julian !
Thank you also to have worked on that and provided a patch ! :-)

David V.

On Wed, May 7, 2008 at 11:38 AM, David V. <dav...@gm...> wrote:

> Shachar,
>
> Thank you very much for solving this bug.
>
> David V.
>
>
> On Wed, May 7, 2008 at 11:15 AM, Shachar Shemesh <sh...@sh...>
> wrote:
>
>> Julian wrote:
>> > Hi Shachar,
>> >
>> > Hope you are OK!!
>> >
>> > I have run into a small (but for me, big!) problem whereby on windows,
>> > the following works:
>> >
>> > C:\> rsyncrypto -rcv --delete "C:\FILES\XYZ\Documents\Malta 123"
>> > "C:\Encrypted\files" "C:\Encrypted\keys" backup.crt
>> >
>> > but the following does not (just the dest drive is differnent):
>> >
>> > C:\> rsyncrypto -rcv --delete "C:\FILES\XYZ\Documents\Malta 123"
>> > "E:\Encrypted\files" "E:\Encrypted\keys" backup.crt
>> >
>> > and I get:
>> > mkdir failed(E:): Input/output error
>> >
>> > I have tried different drives, drive letters, permissions etc... but
>> > to no avail. It seems that any letter other than "C:" is rejected.
>> >
>> > At this point it would be really important for me to be able to backup
>> > to an external drive.
>> >
>> > Any clue why this may be happenning?
>> Because MS has not heard of "documenting what you do" or "API
>> consistency". Then they complain that people do odd stuff in their
>> programs, which MS has to support in future versions. They brought it on
>> themselves.
>>
>> Anyways, the problem is now solved in SVN. I'm hoping to release a new
>> version fairly soon.
>>
>> Shachar
>>
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
>> Don't miss this year's exciting event. There's still time to save $100.
>> Use priority code J8TL2D2.
>>
>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
>> _______________________________________________
>> Rsyncrypto-devel mailing list
>> Rsy...@li...
>> https://lists.sourceforge.net/lists/listinfo/rsyncrypto-devel
>>
>
>

Re: destination cannot be drive other than C: on windows?

[David V. <dav...@gm...>]

Shachar,

Thank you very much for solving this bug.

David V.

On Wed, May 7, 2008 at 11:15 AM, Shachar Shemesh <sh...@sh...>
wrote:

> Julian wrote:
> > Hi Shachar,
> >
> > Hope you are OK!!
> >
> > I have run into a small (but for me, big!) problem whereby on windows,
> > the following works:
> >
> > C:\> rsyncrypto -rcv --delete "C:\FILES\XYZ\Documents\Malta 123"
> > "C:\Encrypted\files" "C:\Encrypted\keys" backup.crt
> >
> > but the following does not (just the dest drive is differnent):
> >
> > C:\> rsyncrypto -rcv --delete "C:\FILES\XYZ\Documents\Malta 123"
> > "E:\Encrypted\files" "E:\Encrypted\keys" backup.crt
> >
> > and I get:
> > mkdir failed(E:): Input/output error
> >
> > I have tried different drives, drive letters, permissions etc... but
> > to no avail. It seems that any letter other than "C:" is rejected.
> >
> > At this point it would be really important for me to be able to backup
> > to an external drive.
> >
> > Any clue why this may be happenning?
> Because MS has not heard of "documenting what you do" or "API
> consistency". Then they complain that people do odd stuff in their
> programs, which MS has to support in future versions. They brought it on
> themselves.
>
> Anyways, the problem is now solved in SVN. I'm hoping to release a new
> version fairly soon.
>
> Shachar
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
> Don't miss this year's exciting event. There's still time to save $100.
> Use priority code J8TL2D2.
>
> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
> _______________________________________________
> Rsyncrypto-devel mailing list
> Rsy...@li...
> https://lists.sourceforge.net/lists/listinfo/rsyncrypto-devel
>

Re: destination cannot be drive other than C: on windows?

[Shachar S. <sh...@sh...>]

Julian wrote:
> Hi Shachar,
>
> Hope you are OK!!
>
> I have run into a small (but for me, big!) problem whereby on windows, 
> the following works:
>
> C:\> rsyncrypto -rcv --delete "C:\FILES\XYZ\Documents\Malta 123" 
> "C:\Encrypted\files" "C:\Encrypted\keys" backup.crt
>
> but the following does not (just the dest drive is differnent):
>
> C:\> rsyncrypto -rcv --delete "C:\FILES\XYZ\Documents\Malta 123" 
> "E:\Encrypted\files" "E:\Encrypted\keys" backup.crt
>
> and I get:
> mkdir failed(E:): Input/output error
>
> I have tried different drives, drive letters, permissions etc... but 
> to no avail. It seems that any letter other than "C:" is rejected.
>
> At this point it would be really important for me to be able to backup 
> to an external drive.
>
> Any clue why this may be happenning?
Because MS has not heard of "documenting what you do" or "API 
consistency". Then they complain that people do odd stuff in their 
programs, which MS has to support in future versions. They brought it on 
themselves.

Anyways, the problem is now solved in SVN. I'm hoping to release a new 
version fairly soon.

Shachar

Re: Nesting bugs; with actual reproduction steps this time.

[Shachar S. <sh...@sh...>]

Robin Lee Powell wrote:
> This is not the same stuff I was talking about before.
>
> Summary: I've got 3 different segfaults here, and quite a lot of
> documentation failures.  If you want me to send an updated man page,
> I'd be happy to, just let me know.
>
> I made a very simple structure like so:
>
> mkdir orig
> cd orig
> mkdir -p a/b/c
> mkdir -p d/e/f
> touch a/b/c/foo
> touch d/e/f/bar
>
> Encrypted like so:
>
> rsyncrypto --ne-nesting=5 --name-encrypt=filemap -c -v -r orig orig.enc orig.keys /var/tmp/rcb/backup.crt
>
> Now, I would like to be able to do cold decryption.  The man page
> doesn't say how the keys directory is supposed to be defined when
> one is doing this, so I just changed the keys directory name to a
> directory that doesn't exist, and deleted the filemap, like so:
>
> rlpowell@chain> ls -l                                      
> total 16
> drwxr-xr-x 4 rlpowell users 4096 2008-02-17 10:11 orig/
> drwxr-xr-x 4 rlpowell users 4096 2008-02-17 10:12 orig.enc/
>
> rlpowell@chain> rsyncrypto --ne-nesting=5 --name-encrypt=filemap -d -v -r orig.enc plain plain.keys /var/tmp/rcb/backup.crt
> [1]    13122 segmentation fault (core dumped)  rsyncrypto --ne-nesting=5 --name-encrypt=filemap -d -v -r orig.enc plain
>
> Well, that's a bit uncool.  :)  It'd be nice if --name-encrypt=
> understood that in -d mode, I mean the filemap in the encrypted
> directory,
It does. The problem here (and it's documented in the "bugs" section of 
the manual page) is that you tried using the public key for cold 
decryption. As you can well understand, if I had a good idea how to give 
a meaningful error message instead of segfaulting, I would.
>  but oh well, let's try again:
>
> rlpowell@chain> rsyncrypto --ne-nesting=5 --name-encrypt=orig.enc/filemap -d -v -r orig.enc plain plain.keys /var/tmp/rcb/backup.crt
> [1]    13190 bus error (core dumped)  rsyncrypto --ne-nesting=5 --name-encrypt=orig.enc/filemap -d -v -r orig.enc
>
> Oh, but wait, it gets better!  In this case, it *deleted the
> filemap*:
>   
Of course it did, precisely because of your question above - it tried to 
reconstruct it.

The proper command line should have been:
> rlpowell@chain> rsyncrypto --ne-nesting=5 --name-encrypt=filemap -d -v -r orig.enc plain plain.keys /var/tmp/rcb/backup.key
Exactly the same command line as your first attempt, only using the 
private key. Just tried it. It automatically decrypted "filemap" from 
the encrypted directory and saved it to "filemap" in the local one, and 
created the entire decrypted tree + symmetric keys.

If I misunderstood you, please clarify. If not, feel free to send 
suggestions as to what should be changed in the manual page to make this 
point clearer.

Shachar

Re: --delete not actually deleting with encrypted names.

[Shachar S. <sh...@sh...>]

Hi Robin,

Robin Lee Powell wrote:
> rsyncrypto is *so* awesome; if only it didn't hate me.  :(
>
> $ find a.enc/ -name 5974EC4026990DE16FAD43681769DC72 -ls
> 12359639    4 -rw-r--r--   1 rlpowell users         244 Feb 21 10:03 a.enc/5/59/597/5974/5974E/5974EC4026990DE16FAD43681769DC72
>
> $ rsyncrypto --trim=1 --ne-nesting=5 --name-encrypt=filemap --delete -c -v -r a a.enc a.keys backup.crt
> Delete 5974EC4026990DE16FAD43681769DC72 (b/c/bar)
>
> $ find a.enc/ -name 5974EC4026990DE16FAD43681769DC72 -ls
> 12359639    4 -rw-r--r--   1 rlpowell users         244 Feb 21 10:03 a.enc/5/59/597/5974/5974E/5974EC4026990DE16FAD43681769DC72
>
> $ bash -c "cat filemap | tr '\000' '\012'"
>   
I use "tr '\0' '\n'" myself.
> /1314AF0918C5FA40C1CBB763CF9BFD3D b/c/frobnitz
> /5974EC4026990DE16FAD43681769DC72 b/c/bar
> /D537DB78571FBCF53B9417A3A5B47FF5 b/d/foo
>
> So the output says the file was deleted, but it in fact was deleted
> from neither the *encrypted space* nor the filemap.
>   
Are you sure about the encrypted space part of that claim? Nothing in 
the output you showed here suggests that the actual encrypted file was 
not deleted.

Unless you also specify "--delete-keys", not deleting the key and file 
mapping is what rsyncrypto is supposed to do. Only the encrypted file 
itself is deleted in such a case. On checking out the man page, I just 
noticed that "--delete-keys" was not documented there (though it did 
appear when you run rsyncrypto --help). I've amended this in latest SVN.

If the above explanation does not appear to explain the symptoms you 
describe (I know it's been a while, but if you can recheck it would be 
great), then please do check out the head of SVN and test again. If the 
problem is still not solved, please let me know.

Thanks,
Shachar

Re: File type change == fail

[Shachar S. <sh...@sh...>]

Robin Lee Powell wrote:
> I've got to say, between this and the delete bug (deletes with a
> filemap don't actually delete anything)
Actually, it was "delete with filemap and --trim=0 don't delete absolute 
paths", if you want the bug description to be accurate.

Either way, this was bug #1759887, and it is now fixed in SVN. Just 
thought you'd be interested to know....

This also provides an easier workaround for your current problem. If you 
run rsyncrypto with --delete-keys between erasing the directory and 
creating the file with the same name, it will get deleted from the filemap.

Shachar

Re: File type change == fail

[Shachar S. <sh...@sh...>]

Robin Lee Powell wrote:
>
> And rsyncrypto does, in fact, do far better, fwiw.
>
>   
Yes, that's what it was designed to do, after all :-)

Shachar

Re: File type change == fail

[Robin L. P. <rlp...@di...>]

On Mon, Apr 28, 2008 at 08:59:46AM +0300, Shachar Shemesh wrote:
> Robin Lee Powell wrote:
> >
> > How about 73?  Surely not a block aligment.  :D
> >
> > dd if=/dev/urandom of=test_rand_wierd bs=1 count=73
> > cat test_rand_wierd test_long1 >test_wierd1
> > cat test_rand_wierd test_long1 >test_wierd2
> > cat test_rand_wierd test_long1 test_rand_wierd >test_wierd3
> >   
> All files begin and end the same way, so I'm not sure what you are
> trying to test here. 

That's because I mis-typed; the first one was:

cat test_long1 test_rand_wierd >test_wierd1

> Try this:
>
> dd if=/dev/urandom of=random1 bs=1024 count=16
> dd if=/dev/urandom of=random2 bs=1024 count=1024
> dd if=/dev/urandom of=pad bs=1 count=73
> 
> cat random1 random2 > before
> cat random1 pad random2 > after
> 
> Now see if rsync manages to save much when it has "before" and it
> needs "after" (or vice versa)

Made two copies, so I could do both:

sent 1049756 bytes  received 6229 bytes  2111970.00 bytes/sec
total size is 1064960  speedup is 1.01

sent 1048929 bytes  received 6271 bytes  2110400.00 bytes/sec
total size is 1065033  speedup is 1.01

So, no.

With larger files, it's better, but still not great:

dd if=/dev/urandom of=random1 bs=1M count=16
dd if=/dev/urandom of=random2 bs=1M count=17
dd if=/dev/urandom of=pad bs=1 count=73

cat random1 random2 > before
cat random1 pad random2 > after
cat random1 random2 > before2
cat random1 pad random2 > after2

sent 17841045 bytes  received 41226 bytes  5109220.29 bytes/sectotal
size is 34603008  speedup is 1.94

sent 17841118 bytes  received 41226 bytes  3973854.22 bytes/sec
total size is 34603081  speedup is 1.94

So about half the file is sent; presumably the random1 part.

And rsyncrypto does, in fact, do far better, fwiw.

cat random1 random2 > before
cat random1 pad random2 > after
cat random1 random2 > before2
cat random1 pad random2 > after2

rsyncrypto plain/before cipher/before keys/before backup.key  
rsyncrypto plain/before2 cipher/before2 keys/before backup.key
rsyncrypto plain/after cipher/after keys/before backup.key    
rsyncrypto plain/after2 cipher/after2 keys/before backup.key  

sent 81235 bytes  received 41296 bytes  35008.86 bytes/sec
total size is 34661412  speedup is 282.88

sent 81299 bytes  received 41296 bytes  49038.00 bytes/sec
total size is 34661476  speedup is 282.73

-Robin

Re: File type change == fail

[Robin L. P. <rlp...@di...>]

On Sun, Apr 27, 2008 at 12:18:45PM -0700, Robin Lee Powell wrote:
> As a workaround, you can remove the file from the filemap.
> 
> Here's some Perl to remove a file from the filemap; I'm afraid I've
> made no attempt to make it user friendly.

Here's a better version.  It walks the entire filemap, and if any
file fails to stat, it deletes that line *and* the encrypted version
of the file.

Run it like this:

/var/tmp/rsyncrypto_filemap_fix.pl [orig filemap] [new filemap] \
    [source dir for unencrypted files] [dest dir for encrypted files]

(As you can see, I'm willing to help out with code :)

-Robin

-- 
Lojban Reason #17: http://en.wikipedia.org/wiki/Buffalo_buffalo
Proud Supporter of the Singularity Institute - http://singinst.org/
http://www.digitalkingdom.org/~rlpowell/ *** http://www.lojban.org/

Re: File type change == fail

[Shachar S. <sh...@sh...>]

Robin Lee Powell wrote:
>
> How about 73?  Surely not a block aligment.  :D
>
> dd if=/dev/urandom of=test_rand_wierd bs=1 count=73
> cat test_rand_wierd test_long1 >test_wierd1
> cat test_rand_wierd test_long1 >test_wierd2
> cat test_rand_wierd test_long1 test_rand_wierd >test_wierd3
>   
All files begin and end the same way, so I'm not sure what you are 
trying to test here. Try this:
dd if=/dev/urandom of=random1 bs=1024 count=16
dd if=/dev/urandom of=random2 bs=1024 count=1024
dd if=/dev/urandom of=pad bs=1 count=73

cat random1 random2 > before
cat random1 pad random2 > after

Now see if rsync manages to save much when it has "before" and it needs 
"after" (or vice versa)

Shachar

Re: File type change == fail

[Robin L. P. <rlp...@di...>]

On Mon, Apr 28, 2008 at 07:18:05AM +0300, Shachar Shemesh wrote:
> Robin Lee Powell wrote:
> > On Mon, Apr 28, 2008 at 12:44:10AM +0300, Shachar Shemesh wrote:
> >   
> >>           o Using the later, I find it hard to believe that it
> >>           actually provides rsync friendliness. In particular,
> >>           does it provide rsync friendliness if the change (not at
> >>           the end of the file) caused a modification to the file
> >>           size?
> >>     
> >
> > OK, here's what I did:
> >
> > dd if=/dev/zero of=test_long1 bs=1M count=16
> > dd if=/dev/urandom of=test_rand_1m bs=1M count=1
> > cat test_long1 test_rand_1m >test_long2             
> > cat test_rand_1m test_long1 >test_long3             
> > cat test_rand_1m test_long1 test_rand_1m >test_long4
> >
> >   
> 
> > So yeah, it appears to work.  If you want me to test something else,
> > let me know.
> >   
> Your change is block aligned. Try a change that is one byte long.

How about 73?  Surely not a block aligment.  :D

dd if=/dev/urandom of=test_rand_wierd bs=1 count=73
cat test_rand_wierd test_long1 >test_wierd1
cat test_rand_wierd test_long1 >test_wierd2
cat test_rand_wierd test_long1 test_rand_wierd >test_wierd3

-rw-r--r-- 1 rlpowell users 16777216 Apr 27 17:06 SfNM,dlfBajg2NiV0YzRJQz4
-rw-r--r-- 1 root     root  16777289 Apr 27 22:29 PBSDF8bMYicaImaAyx5XmPZc
-rw-r--r-- 1 root     root  16777289 Apr 27 22:29 1m3WqxLQu5JtWXHswzHwKeRI
-rw-r--r-- 1 root     root  16777362 Apr 27 22:29 I8DD-PtO14RhUyy6CBB5rWtj

cp SfNM,dlfBajg2NiV0YzRJQz4 /tmp/target
rsync --stats --no-whole-file PBSDF8bMYicaImaAyx5XmPZc /tmp/target
[snip]
sent 16546 bytes  received 24607 bytes  82306.00 bytes/sec
total size is 16777289  speedup is 407.68

For 1m3WqxLQu5JtWXHswzHwKeRI:
sent 20642 bytes  received 24607 bytes  90498.00 bytes/sec
total size is 16777289  speedup is 370.78

For I8DD-PtO14RhUyy6CBB5rWtj:

sent 20715 bytes  received 24607 bytes  90644.00 bytes/sec
total size is 16777362  speedup is 370.18

-Robin

Re: Questions on Certificate Criteria

[Shachar S. <sh...@sh...>]

Chuck Okerstrom wrote:
> or even tell me what parameters of the 
> cert generation itself are critical to this whole process?
Oh dear, and here I was trying to be nice to people.

The PKCS certificates are used for one purpose, and one purpose only. As 
a standard way of storing the public key. They are not tested for any 
validity (by rsyncrypto) beyond that, at all. They can have whatever 
name you want, be signed, unsigned or self signed, and can even be 
expired for all rsyncrypto cares. The only thing that matters is the 
public key stored in them.
>   If I'd 
> incorrectly specified the cert generation parameters would it still 
> work, but maybe not optimally?
>   
No. If it has the right key, it will work. If it hasn't, it won't. The 
only reason PKCS certificates were used at all was that there is no 
other standard way to store a public key.

Hope that quieten your concerns.

Shachar

Re: File type change == fail

[Shachar S. <sh...@sh...>]

Robin Lee Powell wrote:
> On Mon, Apr 28, 2008 at 12:44:10AM +0300, Shachar Shemesh wrote:
>   
>>           o Using the later, I find it hard to believe that it
>>           actually provides rsync friendliness. In particular,
>>           does it provide rsync friendliness if the change (not at
>>           the end of the file) caused a modification to the file
>>           size?
>>     
>
> OK, here's what I did:
>
> dd if=/dev/zero of=test_long1 bs=1M count=16
> dd if=/dev/urandom of=test_rand_1m bs=1M count=1
> cat test_long1 test_rand_1m >test_long2             
> cat test_rand_1m test_long1 >test_long3             
> cat test_rand_1m test_long1 test_rand_1m >test_long4
>
>   

> So yeah, it appears to work.  If you want me to test something else,
> let me know.
>   
Your change is block aligned. Try a change that is one byte long.
> It's worth noting that I prioritized rsyncability over security in
> my setup; I used block encoding, and no IV at all.
>   
Like I said in my earlier email, I suspect that block encoding actually 
increases security.

Also, if rsyncrypto provides rsyncability enough for you, there should 
be no problem with adding the IV back. The IV will make different file 
names have different session keys, but rsyncrypto has that too.

(In actuality, that is not a correct statement. The IV will cause 
different files to have different encodings for the same data. As far as 
cryptographic strength goes, they actually WILL have the same key, which 
means that rsyncrypto still has more secure encoding in this case as well).
>   
> Stream mode only affects files names, AFAICT.
>   
I don't know. I haven't seen the actual algorithm explained. What I 
gathered from the presentation I read was that block mode meant 
encrypting the file in CBC mode, reverting to a known IV every "block" 
while stream mode meant using CFB. For comparison sake, rsyncrypto uses 
CBC, reverting to a known IV in a self synchronizing way, so that block 
sizes are dynamic by track specific points in the file. This, itself, is 
a potential information leak about the plain text file, but an extremely 
minor one (and will be even smaller for rsyncrypto 2).

All ciphers suffer when almost the same data is re-encrypted using the 
same key. CBC is rather robust. CFB, however, has a huge problem. Any 
change means that two different plain texts are XORed with the same 
pseudo random stream. By simply XORing the two versions, Eve has a XOR 
of the two plain texts encrypted. Assuming they are not very random, a 
simple double frequency analysis will give her a good chance of cracking 
the plain text.

What's worse, once cracked, this applies to ALL future encryptions using 
the same key.
> -Robin
>
>   
Shachar

Re: File type change == fail

[Robin L. P. <rlp...@di...>]

On Mon, Apr 28, 2008 at 12:44:10AM +0300, Shachar Shemesh wrote:
>           o Using the later, I find it hard to believe that it
>           actually provides rsync friendliness. In particular,
>           does it provide rsync friendliness if the change (not at
>           the end of the file) caused a modification to the file
>           size?

OK, here's what I did:

dd if=/dev/zero of=test_long1 bs=1M count=16
dd if=/dev/urandom of=test_rand_1m bs=1M count=1
cat test_long1 test_rand_1m >test_long2             
cat test_rand_1m test_long1 >test_long3             
cat test_rand_1m test_long1 test_rand_1m >test_long4

Then I unmounted.

Then I figured out which was which by file size, more or less:

-rw-r--r-- 1 rlpowell users 16777216 2008-04-27 17:06 SfNM,dlfBajg2NiV0YzRJQz4
-rw-r--r-- 1 rlpowell users 17825792 2008-04-27 17:38 NZ92YbAoRtXWp29DypUz6g,n
-rw-r--r-- 1 rlpowell users 17825792 2008-04-27 17:38 3XnzDYA9ebc1ZNTuxQjsyVw8
-rw-r--r-- 1 rlpowell users 18874368 2008-04-27 17:38 X3bNiy35p439CRPql54uJ7I9

cp SfNM,dlfBajg2NiV0YzRJQz4 /tmp/target                           
rsync --stats --no-whole-file NZ92YbAoRtXWp29DypUz6g,n /tmp/target
[snip]
sent 1065173 bytes  received 24607 bytes  726520.00 bytes/sec
total size is 17825792  speedup is 16.36

cp SfNM,dlfBajg2NiV0YzRJQz4 /tmp/target                           
rsync --stats --no-whole-file 3XnzDYA9ebc1ZNTuxQjsyVw8 /tmp/target
[snip]
sent 2112853 bytes  received 24607 bytes  854984.00 bytes/sec
total size is 17825792  speedup is 8.34

cp SfNM,dlfBajg2NiV0YzRJQz4 /tmp/target                           
rsync --stats --no-whole-file X3bNiy35p439CRPql54uJ7I9 /tmp/target
[snip]
sent 3161557 bytes  received 24607 bytes  6372328.00 bytes/sec
total size is 18874368  speedup is 5.92

So yeah, it appears to work.  If you want me to test something else,
let me know.

It's worth noting that I prioritized rsyncability over security in
my setup; I used block encoding, and no IV at all.

>     * I have not looked at the actual implementation, only at the
>     presentation, but it seems to me that the "stream mode" offers
>     grave security issues if an attacker ever has access to the
>     same encrypted image before and after a modification that
>     changes the file length. This is unrelated to rsyncrypto. This
>     is, by the way, an issue with many file encrypting programs.

Stream mode only affects files names, AFAICT.

-Robin

-- 
Lojban Reason #17: http://en.wikipedia.org/wiki/Buffalo_buffalo
Proud Supporter of the Singularity Institute - http://singinst.org/
http://www.digitalkingdom.org/~rlpowell/ *** http://www.lojban.org/

Questions on Certificate Criteria

[Chuck O. <cok...@ea...>]

Due to my requirements for separate certs for each of my customers, I'm 
attempting to automate this cert creation process and am using PHP to 
create and store the private & public keys.  Unfortunately the method 
for doing this within PHP is quite different than the openssl command 
line method that's used in the example, so I'm attempting to emulate 
that given example within PHP and want to ensure that I'm not 
overlooking something.

I've created a very simple test script (shown below) to simply and 
quickly create the cert pairs. Tests using these generated certs all 
seem to show that everything is working correctly and rsync does indeed 
find "Matched data" and only sends a chunk of the file and not the whole 
thing.  And, of course, all back and forth encryption/decryptions are 
successful.

But, I suspect that I really haven't generated these certs with the 
exact parameters as specified in the example and was wondering if anyone 
could see an obvious problem, or even tell me what parameters of the 
cert generation itself are critical to this whole process?  If I'd 
incorrectly specified the cert generation parameters would it still 
work, but maybe not optimally?

Below are my test script and the 2 generated test certs (which I've 
attempted to escape so that an e-mail client wouldn't try to use 'em -- 
though I'm unsure whether it would even matter).  Comments anyone?


Thanx,
Chuck



Test Script
=========================
<?php

// Setup CSR info
$dn = array('countryName' 			=> 'US',
			'stateOrProvinceName'	=> 'FL',
			'localityName'		=> 'Orlando',
			'organizationName'	=> 'Business',
			'organizationalUnitName' => 'none',
			'commonName'		=> 'Business',
			'emailAddress'		=> 'co...@ea...',
			);
			
// Specify Key parameters
$key_conf = array(	'private_key_bits'	=> 1536,
			'private_key_type'	=> OPENSSL_KEYTYPE_RSA,
			'encrypt_key'		=> FALSE,
				);
				
$pw = '';
$numdays = 1085;			


$privkey = openssl_pkey_new($key_conf);
$csr = openssl_csr_new($dn, $privkey);
$cert = openssl_csr_sign($csr, null, $privkey, $numdays);
openssl_pkey_export_to_file($privkey, 'test.key');
openssl_x509_export_to_file($cert, 'test.pub');
?>


Private Cert - test.key
===========================================
#-----BEGIN RSA PRIVATE KEY-----
#MIIDfAIBAAKBwQCdYQFngtGBQ+FmwgEIo565UNuHYuBUEGXwb0T/76Cg4UTlMwqq
#1u9RLTeYuWzRMq+rJJmr+6ZfdP2vySw4HL/XzfvpekSC3CYku3CJ7wg/FH0GIxiz
#SPlLAtPsDR73j3yTOIJgMmgUJY3FQyTyRpQ48XJNfpIsDeUgki6fC/Vk6Lqd1uOw
#D1pVLMN6e5GUwevyaGpqFQaTx21Z2ANWqc8gjoJv3pKeHbfVfHzUGOxSSKdC0B3P
#sQYMIEp1yTE9ItsCAwEAAQKBwH7oaql95FPI2Upzx0GgL65gdaaHJT6kuo9YKtv3
#8B/LiDMLJd79054ySFLvs8A+j0oDCaiWFWOEg83s+6uEA2+Su0FbR0P/IwMb43RX
#PN8hNnBsfM6WWfETJrGDIyWeniJ+JVmmGbW04B897mkzPOIQH1LHR1ESbnbQfACZ
#sWue9Fnz1GCGiZOw3ojT+CM2Kz94sy6ZsX242KjK1nWv75qUwqacgs5UZdz8xSEi
#NCmEVU+CAvyuzn4IynF/iRhngQJhAM6gJeERCSn7ZU+JQBw+XAnPLVx0XNqDMUQJ
#uCGrPUJaLG3lkw3PIxq9qyefP5dQPoZgyf103C+H7OafaicEyw3Wl6rc7pkpT1vK
#unqmbiqY2uyLP45288TvLvGV058ITQJhAML8Th7QNNmCBVR8XbW0VEr5B57uBmMq
#cXjdyFSBQv+teuw9qnD7a2rS/RKE474hY6/nwlYWNKnz5zbT2Fmy8PSz5nOhQYj9
#JVlcpepE5N7KY2BPKg+xOG0Of8IoE3vrxwJgGTgkx26r3qrnd6i54XifBTd7QuCV
#ALqohbRl+/4JkRKuf49YvoO8tiPWQxTFzzMlHoOrw7rCsS529MMaUr7cBcleY6Vp
#ndoT7JE254duxNY5SkvIqxvLrwq+gRAXbz61AmAWBDct84SEKtI/P+u04K/D52qc
#33OJLvmxFBnSsOXHyObgVfYw27K9VSWOOcMdbNe8vQaMgeVga1HoNvNu7W2Xs9iJ
#peOofC0Dchqp4S2WmnOuJEIzk2czqTdzzOKmU3ECYQCXMKcLI3vVptiirzBPsjbI
#7UAya/gFGtLXXFdNcDAVIVA+rDdWwSWBmN1QglYIdGrdu+t1gjw9awZ/EsjGa90p
#SOkout0eX9jqXp5PJjTocYgxPZtE45fwOcJoW9xqa34=
#-----END RSA PRIVATE KEY-----


Public Cert - test.pub
============================
#-----BEGIN CERTIFICATE-----
#MIIDzDCCAvWgAwIBAgIBADANBgkqhkiG9w0BAQQFADB9MQswCQYDVQQGEwJVUzEL
#MAkGA1UECBMCRkwxEjAQBgNVBAcTCU1lbGJvdXJuZTERMA8GA1UEChMIQnVzaW5l
#c3MxDTALBgNVBAsTBG5vbmUxETAPBgNVBAMTCEJ1c2luZXNzMRgwFgYJKoZIhvcN
#AQkBFgljb0BlYS5uZXQwHhcNMDgwNDI3MjAzOTMwWhcNMTEwNDE3MjAzOTMwWjB9
#MQswCQYDVQQGEwJVUzELMAkGA1UECBMCRkwxEjAQBgNVBAcTCU1lbGJvdXJuZTER
#MA8GA1UEChMIQnVzaW5lc3MxDTALBgNVBAsTBG5vbmUxETAPBgNVBAMTCEJ1c2lu
#ZXNzMRgwFgYJKoZIhvcNAQkBFgljb0BlYS5uZXQwgd8wDQYJKoZIhvcNAQEBBQAD
#gc0AMIHJAoHBAJ1hAWeC0YFD4WbCAQijnrlQ24di4FQQZfBvRP/voKDhROUzCqrW
#71EtN5i5bNEyr6skmav7pl90/a/JLDgcv9fN++l6RILcJiS7cInvCD8UfQYjGLNI
#+UsC0+wNHvePfJM4gmAyaBQljcVDJPJGlDjxck1+kiwN5SCSLp8L9WToup3W47AP
#WlUsw3p7kZTB6/JoamoVBpPHbVnYA1apzyCOgm/ekp4dt9V8fNQY7FJIp0LQHc+x
#BgwgSnXJMT0i2wIDAQABo4HbMIHYMB0GA1UdDgQWBBR0+jj2imA8nyB9UwBIjJO2
#QuuDnDCBqAYDVR0jBIGgMIGdgBR0+jj2imA8nyB9UwBIjJO2QuuDnKGBgaR/MH0x
#CzAJBgNVBAYTAlVTMQswCQYDVQQIEwJGTDESMBAGA1UEBxMJTWVsYm91cm5lMREw
#DwYDVQQKEwhCdXNpbmVzczENMAsGA1UECxMEbm9uZTERMA8GA1UEAxMIQnVzaW5l
#c3MxGDAWBgkqhkiG9w0BCQEWCWNvQGVhLm5ldIIBADAMBgNVHRMEBTADAQH/MA0G
#CSqGSIb3DQEBBAUAA4HBAEYocJY76tGHJYf/FcKGCfFQkXD5R2Xnh6eGI6IT6pOZ
#uyhDhn5dsmkgS57JZbcvRETLuvkLnrCLSU40f9e0E6bhfo6po8CM3eUylStDykOQ
#h8f5Kph2junSGv7ay3DHbt6Ou/ExLPA0dwsD7olrHWnRg52NuuakPHDPPFydPVVn
#1FnTRKmmKJ6c2+pScnb8HMVhMhGuppVFIlOw10ALLx8+tC1zUeZSnJlFxSikGnYc
#rnYI1Kt+97PbBvx2ouLMlA==
#-----END CERTIFICATE-----

Re: File type change == fail

[Shachar S. <sh...@sh...>]

Robin Lee Powell wrote:
>
> I'm sorry, but I tried at one point.  I couldn't deal with the code.
> :(
>   
And I actually tried.....

> That wasn't my intention at all.
No harm done.
>   My intention was to point out
> something useful in similar situations to those who didn't know
> about it (like me, until a couple of hours ago).
In  that case, lets start with a link:
http://sourceforge.net/projects/ecryptfs or http://www.arg0.net/encfs

I downloaded a presentation about it, and I have the following insights:

    * This is an encrypted file system. This has nothing to do with
      rsyncrypto (i.e. - not the same design goals)
    * If you attempt to use EncFS for rsync, there are two ways to do
      it. Either show rsync the decrypted version (through EncFS) or the
      encrypted version (through the original file system):
          o Using the former is just like using any other file system.
          o Using the later, I find it hard to believe that it actually
            provides rsync friendliness. In particular, does it provide
            rsync friendliness if the change (not at the end of the
            file) caused a modification to the file size?
    * I have not looked at the actual implementation, only at the
      presentation, but it seems to me that the "stream mode" offers
      grave security issues if an attacker ever has access to the same
      encrypted image before and after a modification that changes the
      file length. This is unrelated to rsyncrypto. This is, by the way,
      an issue with many file encrypting programs.

In other words, EncFS may or may not be a good idea on its own, but I 
wouldn't use it as a rsyncrypto replacement if rsyncability is important 
to you (and if it's not, I wouldn't use rsyncrypto either, though I have 
to admit that there is a shortage of whole tree encryptors of a 
reasonable quality).
>   Sorry you felt attacked.
>   
Forgotten.
> -Robin
>
>   
Shachar

Re: File type change == fail

[Robin L. P. <rlp...@di...>]

On Mon, Apr 28, 2008 at 12:01:24AM +0300, Shachar Shemesh wrote:
> In other words, I'm sorry you feel unhappy with rsyncrypto. I will
> definitely look at those bugs when I get a chance (and sooner if
> you send a patch), 

I'm sorry, but I tried at one point.  I couldn't deal with the code.
:(

> but a simple bug report would have been just as effective, if not
> more. I tend to ignore people who try to employ emotional
> blackmail on me.

*blink*

That wasn't my intention at all.  My intention was to point out
something useful in similar situations to those who didn't know
about it (like me, until a couple of hours ago).  Sorry you felt
attacked.

-Robin

-- 
Lojban Reason #17: http://en.wikipedia.org/wiki/Buffalo_buffalo
Proud Supporter of the Singularity Institute - http://singinst.org/
http://www.digitalkingdom.org/~rlpowell/ *** http://www.lojban.org/

Re: File type change == fail

[Shachar S. <sh...@sh...>]

Before anything else, thanks for the bug report. I really do appreciate it.

Robin Lee Powell wrote:
> You'll have to take my word for it that rsyncrypto exits at that
> point.
>   
Funny, I was certain that it's possible to test that....
>
> I've got to say, between this and the delete bug (deletes with a
> filemap don't actually delete anything), EncFS is looking mighty
> tasty.
If you send me the receipt number I'll be happy to give you a full refund.
>   I've tested, and rsync *does* work with it (in the sense of
> saving transfer time/data).
>   
Rsyncrypto is made for a specific purpose. that purpose is to allow you 
to sync (using rsync) without sharing the key. Unless I'm missing on 
something, EncFS does not allow you that. I should point out that if 
that is not what you are after, then other solutions are much more space 
efficient, as they do not require an extra encrypted copy. Plans for 
making that unnecessary are in place, but they will take some time.

In other words, I'm sorry you feel unhappy with rsyncrypto. I will 
definitely look at those bugs when I get a chance (and sooner if you 
send a patch), but a simple bug report would have been just as 
effective, if not more. I tend to ignore people who try to employ 
emotional blackmail on me.
> -Robin
>   
Shachar

File type change == fail

[Robin L. P. <rlp...@di...>]

If a file changes from a directory to a regular file, rsyncrypto
errors out and does not continue, which is annoying when it's in the
middle of my ~million file backup.  -_-

Example: starting with:

$ find orig -ls                                                            
6455565    4 drwxr-xr-x   3 rlpowell users        4096 Apr 27 11:10 orig
6455574    4 drwxr-xr-x   2 rlpowell users        4096 Apr 27 11:20 orig/testdir
6455575    4 -rw-r--r--   1 rlpowell users           4 Apr 27 11:10 orig/testdir/testfile

I run:

$ rsyncrypto --trim=1 --ne-nesting=5 --name-encrypt=filemap --delete -c -v -r orig enc keys backup.crt

I then turn testdir into a file, so it looks like this:

$ find orig -ls
6455597    4 drwxr-xr-x   2 rlpowell users        4096 Apr 27 11:22 orig
6455598    0 -rw-r--r--   1 rlpowell users           0 Apr 27 11:22 orig/testdir

$ rsyncrypto --trim=1 --ne-nesting=5 --name-encrypt=filemap --delete -c -v -r orig enc keys backup.crt
Encrypting orig/testdir
lstat failed(orig/testdir/testfile): Not a directory

You'll have to take my word for it that rsyncrypto exits at that
point.

As a workaround, you can remove the file from the filemap.

Here's some Perl to remove a file from the filemap; I'm afraid I've
made no attempt to make it user friendly.

perl -n0e 'BEGIN { open( FM, ">[NEWFILEMAP]"); } m{^/[0-9A-F]+ [FILENAME_TO_REMOVE]} || do { print FM; } ' <[FILEMAP]

To just see the contents of the filemap so you know what to m{...}
for, try:

tr '\000' '\012' <[FILEMAP] | less

I've got to say, between this and the delete bug (deletes with a
filemap don't actually delete anything), EncFS is looking mighty
tasty.  I've tested, and rsync *does* work with it (in the sense of
saving transfer time/data).

-Robin

-- 
Lojban Reason #17: http://en.wikipedia.org/wiki/Buffalo_buffalo
Proud Supporter of the Singularity Institute - http://singinst.org/
http://www.digitalkingdom.org/~rlpowell/ *** http://www.lojban.org/

Re: Problem with configure on OpenBSD cannot find -largtable2

[Chuck O. <cok...@ea...>]

Just wanted to respond back that this did indeed solve the problem.  So 
in my case, the method used to solve the configure script issue and then 
be able to run make was....

$ LDFLAGS=-L/usr/local/lib  CPPFLAGS=-I/usr/local/include  ./configure


Many thanks for your response.  Have a wonderful weekend!


Chuck



Shachar Shemesh wrote:
> Chuck Okerstrom wrote:
>> First off, my compliments on this little jewel.  I've been looking 
>> for a good encryption option for rysnc for quite some time and this 
>> sure looks like it'll fit the bill.
>>   
> Thanks.
>> I've tried setting an LDFLAGS environment var....
>>
>> $ echo $LDFLAGS
>> /usr/local/lib
>>   
> First of all, it is important to note that the time to set the 
> environment variable is when invoking configure, not when invoking 
> "make". configure saves a snapshot of the environment is finds inside 
> the generated make file. If you are using bash (and also, if memory 
> serves me correctly, bourne shell), you can do "LDFLAGS=whatever 
> ./configure" to temporarily set the correct environment variables.
>
> As for the actual problem - LDFLAGS are the command line parameters 
> passed to ld. Your problem seems to be that you did not specify a full 
> parameter.
>
> There are two solutions to your problem. If you want to use argtable 
> in its uninstalled state, you need to use the "--with-argtable2=path" 
> configure command line option. This does not appear to be your case. 
> In your case, you probably simply need to run:
> LDFLAGS=-L/usr/local/lib ./configure
>
> to solve your problem.
>
> Depending on your setup, I suspect you may also need to specify:
> CPPFLAGS=-I/usr/local/include
>
> Otherwise the header files for argtable may also not be found.
>
> Hope this solves your problem
>
>> I've tried setting the configure option --libdir=/usr/local/lib
>>   
> That's because libdir controls where libraries generated by the 
> current project are sent during 'make install', and does not affect 
> where libraries needed by the current project are searched for.
>
> Also, make sure that /etc/ld.so.conf has /usr/local/lib in it, and 
> make sure that you have run ldconfig since you installed argtable. If 
> you do not, rsyncrypto may not run, complaining that it cannot find 
> the library.
>
> Shachar
>
>

Re: Problem with configure on OpenBSD cannot find -largtable2

[Shachar S. <sh...@sh...>]

Chuck Okerstrom wrote:
> First off, my compliments on this little jewel.  I've been looking for a 
> good encryption option for rysnc for quite some time and this sure looks 
> like it'll fit the bill.
>   
Thanks.
> I've tried setting an LDFLAGS environment var....
>
> $ echo $LDFLAGS
> /usr/local/lib
>   
First of all, it is important to note that the time to set the 
environment variable is when invoking configure, not when invoking 
"make". configure saves a snapshot of the environment is finds inside 
the generated make file. If you are using bash (and also, if memory 
serves me correctly, bourne shell), you can do "LDFLAGS=whatever 
./configure" to temporarily set the correct environment variables.

As for the actual problem - LDFLAGS are the command line parameters 
passed to ld. Your problem seems to be that you did not specify a full 
parameter.

There are two solutions to your problem. If you want to use argtable in 
its uninstalled state, you need to use the "--with-argtable2=path" 
configure command line option. This does not appear to be your case. In 
your case, you probably simply need to run:
LDFLAGS=-L/usr/local/lib ./configure

to solve your problem.

Depending on your setup, I suspect you may also need to specify:
CPPFLAGS=-I/usr/local/include

Otherwise the header files for argtable may also not be found.

Hope this solves your problem

> I've tried setting the configure option --libdir=/usr/local/lib
>   
That's because libdir controls where libraries generated by the current 
project are sent during 'make install', and does not affect where 
libraries needed by the current project are searched for.

Also, make sure that /etc/ld.so.conf has /usr/local/lib in it, and make 
sure that you have run ldconfig since you installed argtable. If you do 
not, rsyncrypto may not run, complaining that it cannot find the library.

Shachar

Problem with configure on OpenBSD cannot find -largtable2

[Chuck O. <cok...@ea...>]

First off, my compliments on this little jewel.  I've been looking for a 
good encryption option for rysnc for quite some time and this sure looks 
like it'll fit the bill.

My problem occurs when attempting to initially run ./configure on an 
OpenBSD 4.0 box.  The configure script refuses to find my installed 
argtables2 library.  I'm no make/build genius, but I can generally hack 
around enough to get by configure errors, but this one is absolutely 
killing me.  So any hints would be appreciated.

I've got more details listed below, but it appears to me that the crux 
of the issue is the following test in the configure script...

   g++ -o conftest -g -O2   conftest.cpp -largtable2  -lcrypto    // 
This Fails

When this run on the cmd line, I get the same "/usr/bin/ld: cannot find 
-largtable2" error.  But if I add a search path with -L it works...

   g++ -o conftest -g -O2   conftest.cpp -largtable2  -lcrypto 
-L/usr/local/lib   // This Succeeds!!!

So how can I get the configure script to use this search path? 

I've tried setting an LDFLAGS environment var....

$ echo $LDFLAGS
/usr/local/lib

I've tried setting the configure option --libdir=/usr/local/lib

But none of these courses seem to change the symptom even a little.  So 
any thoughts?

Thanx,

Chuck




Details:
==================

Check for ArgTables:
- - - - - - - - - - - - - -
$ pkg_info -L argtable-2.6p0
Information for argtable-2.6p0

Files:
/usr/local/lib/libargtable2.so.1.1
/usr/local/include/argtable2.h
/usr/local/lib/libargtable2.a
/usr/local/lib/libargtable2.la
/usr/local/man/man3/argtable2.3
....


Run Configure on Rsyncrypto
- - - - - - - - - - - - - - - - - - -
$ ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for gawk... no
...
...
checking for AES_encrypt in -lcrypto... yes
checking for arg_parse in -largtable2... no
configure: error: argtable2 not found
See `config.log' for more details.


Interesting Spot in config.log
- - - - - - - - - - - - - - - - - -
configure:3111: checking for arg_parse in -largtable2
configure:3146: g++ -o conftest -g -O2   conftest.cpp -largtable2  
-lcrypto  >&5
/usr/bin/ld: cannot find -largtable2
collect2: ld returned 1 exit status
configure:3152: $? = 1

| int
| main ()
| {
| return arg_parse ();
|   ;
|   return 0;
| }
configure:3170: result: no
configure:3180: error: argtable2 not found
See `config.log' for more details.

Re: file open

[Julian <jul...@gm...>]

>
> For one thing, rsync uses cygwin while rsyncrypto uses the Win32 API
> directly. You really need to find out what win32 APIs are necessary.


It seems the problem lies within FindFirstFile. From msdn:
"If the string ends with a wildcard, period (.), or directory name, the user
must have access to the root and all subdirectories on the path."

No idea what's being done in rsync/cygwin.

Re: file open

[Shachar S. <sh...@sh...>]

Julian wrote:

> Hi Shachar,
>
> I have noticed that in Windows sometimes not even the Administrator 
> has read access rights to user folders. (I think this is the case in 
> an active directory environment).
>
> As a result, when trying to encrypt C:\Documents and Settings\User1, 
> rsyncrypto fails with:
> opendir failed(C:\Documents and Settings\User1): Input/output error
>
> This has happened to me on 2 different machines with XP and 2003.
> The strange thing is that if I try and rsync these folders directly, 
> rsync completes without a hitch.
> Is there a difference in the way rsync and rsyncrypto try to read the 
> files and directories?
For one thing, rsync uses cygwin while rsyncrypto uses the Win32 API 
directly. You really need to find out what win32 APIs are necessary.

Shachar