rsyncrypto-devel — Development discussions, bug reports, and announcements for rsyncrypto

RE: performance of rsyncrypto

[hf h. <tim...@ho...>]

I'm running it on the redhat platform. and cpu is 4 cores. 
 


Date: Sun, 9 May 2010 12:52:19 +0300
From: sh...@sh...
To: tim...@ho...
CC: rsy...@li...
Subject: Re: performance of rsyncrypto

hhf wrote: 


hi:
    I just found the rsyncrypto project recently, and I use it to encrypt my own server's database, and then rsync to another place. it's a really good project.
 
    I think the encryption strength is enough for me, but the speed of the rsyncrypto is only 5MB/s in my case. my database is around 50GB, so it should cost few hours to encrypt the database file. 
    Is the speed 5MB/s normal? my PC's performance is quite good. If it's normal, the openssl AES cause it? or the gzip --rsyncable cause it?
What platform are you running rsyncrypto on? The Windows performance is not as good as the Linux performance, mainly due to the multi-process orientation of the program (data passes between the encryption and the compression on a pipe).

There are several causes to the performance issues in rsyncrypto. First, rsyncrypto only utilizes about 1.5 CPUs, so many multi cores today are under-utilized by rsyncrypto. We are also calling the AES encryption one block at a time, which might further be non-optimal.

Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com
 		 	   		  
_________________________________________________________________
Hotmail: Trusted email with Microsoft’s powerful SPAM protection.
https://signup.live.com/signup.aspx?id=60969

Re: performance of rsyncrypto

[Shachar S. <sh...@sh...>]

海峰 黄 wrote:
> hi:
> I just found the rsyncrypto project recently, and I use it to encrypt
> my own server's database, and thenrsyncto another place. it's a really
> good project.
>
> I think the encryption strength is enough for me, but the speed of the
> rsyncrypto is only 5MB/s in my case. my database is around 50GB, so it
> should cost few hours to encrypt the database file.
> Is the speed 5MB/s normal? my PC's performance is quite good. If it's
> normal, the openssl AES cause it? or the gzip --rsyncable cause it?
What platform are you running rsyncrypto on? The Windows performance is
not as good as the Linux performance, mainly due to the multi-process
orientation of the program (data passes between the encryption and the
compression on a pipe).

There are several causes to the performance issues in rsyncrypto. First,
rsyncrypto only utilizes about 1.5 CPUs, so many multi cores today are
under-utilized by rsyncrypto. We are also calling the AES encryption one
block at a time, which might further be non-optimal.

Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com

performance of rsyncrypto

[海峰 黄 <tim...@ho...>]

hi:
    I just found the rsyncrypto project recently, and I use it to encrypt my own server's database, and then rsync to another place. it's a really good project.
 
    I think the encryption strength is enough for me, but the speed of the rsyncrypto is only 5MB/s in my case. my database is around 50GB, so it should cost few hours to encrypt the database file. 
    Is the speed 5MB/s normal? my PC's performance is quite good. If it's normal, the openssl AES cause it? or the gzip --rsyncable cause it?
 

    thanks. 

 
                                  timen 		 	   		  
_________________________________________________________________
Hotmail: Powerful Free email with security by Microsoft.
https://signup.live.com/signup.aspx?id=60969

RE: Two different RsynCrypto scenario's

[Jan A. <jan...@do...>]

Hi Shachar,

 

I know both of the scenario's are supported, but I was just wondering if one
is better then the other.

 

Regards,

 

Jan

 

  _____  

Van: Shachar Shemesh [mailto:sh...@sh...] 
Verzonden: dinsdag 13 april 2010 11:21
Aan: Jan Alphenaar
CC: rsy...@li...
Onderwerp: Re: Two different RsynCrypto scenario's

 

Jan Alphenaar wrote: 

Hi,

 

I was just wondering if there was any difference in the level of encryption
for the two following scenario's.

 

In the first scenario I generate a certificate and a key. Encryption is done
with the certificate and decryption is done with the key. See these commands
directly below.

 

openssl.exe req -days 35600 -newkey rsa:1536 -keyout cert.key -nodes -x509
-out cert.crt -config openssl.conf"

rsyncrypto test_clear.txt test_enc.txt test.key cert.crt

rsyncrypto -d test_enc.txt test_clear.txt test.key cert.key"

 

In the second scenario I only generate an RSA key and both encrypt and
decrypt the files with this key. See the commands directly below.

 

openssl.exe req -days 35600 -newkey rsa: 1536 -keyout private.key -nodes
-config openssl.conf"

rsyncrypto test_clear.txt test_enc.txt test.key private.key"

rsyncrypto -d test_enc.txt test_clear test.key cert.key"

If you still have the symmetric key (test.key) around, you can also decrypt
using the certificate (cert.crt).

All of the above are supported. I'm not sure what the question is.

Shachar




-- 
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com

Re: Two different RsynCrypto scenario's

[Shachar S. <sh...@sh...>]

Jan Alphenaar wrote:
>
> Hi,
>
>  
>
> I was just wondering if there was any difference in the level of 
> encryption for the two following scenario's.
>
>  
>
> In the first scenario I generate a certificate and a key. Encryption 
> is done with the certificate and decryption is done with the key. See 
> these commands directly below.
>
>  
>
> openssl.exe req -days 35600 -newkey rsa:1536 -keyout cert.key -nodes 
> -x509 -out cert.crt -config openssl.conf"
>
> rsyncrypto test_clear.txt test_enc.txt test.key cert.crt
>
> rsyncrypto -d test_enc.txt test_clear.txt test.key cert.key"
>
>  
>
> In the second scenario I only generate an RSA key and both encrypt and 
> decrypt the files with this key. See the commands directly below.
>
>  
>
> openssl.exe req -days 35600 -newkey rsa: 1536 -keyout private.key 
> -nodes -config openssl.conf"
>
> rsyncrypto test_clear.txt test_enc.txt test.key private.key"
>
> rsyncrypto -d test_enc.txt test_clear test.key cert.key"
>
If you still have the symmetric key (test.key) around, you can also 
decrypt using the certificate (cert.crt).

All of the above are supported. I'm not sure what the question is.

Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com

Two different RsynCrypto scenario's

[Jan A. <jan...@do...>]

Hi,

 

I was just wondering if there was any difference in the level of encryption
for the two following scenario's.

 

In the first scenario I generate a certificate and a key. Encryption is done
with the certificate and decryption is done with the key. See these commands
directly below.

 

openssl.exe req -days 35600 -newkey rsa:1536 -keyout cert.key -nodes -x509
-out cert.crt -config openssl.conf"

rsyncrypto test_clear.txt test_enc.txt test.key cert.crt

rsyncrypto -d test_enc.txt test_clear.txt test.key cert.key"

 

In the second scenario I only generate an RSA key and both encrypt and
decrypt the files with this key. See the commands directly below.

 

openssl.exe req -days 35600 -newkey rsa: 1536 -keyout private.key -nodes
-config openssl.conf"

rsyncrypto test_clear.txt test_enc.txt test.key private.key"

rsyncrypto -d test_enc.txt test_clear test.key cert.key"

 

Any information is appreciated.

 

Warm regards,

 

Jan

rsyncrypto causes rsync rename errors

[Durham C. S. Inc. <ed...@du...>]

Durham Computer Solutions Inc. wrote: 
  I solved it myself

  permissions on the folders on the target machine
  were not set properly with the
  result all files were being set with read only attribute

  I changed the permissions and all is fine

Re: rsyncrypto causes rsync rename errors

[Pierre A. <ph...@ph...>]

On Thursday 04 February 2010 01:48:43 Shachar Shemesh wrote:
> Are you running a native Windows rsyncrypto, or the cygwin one? I'll
> just add that, while I'm aware that a cygwin port of rsyncrypto is,
> theoretically, possible with the current sources, I have never tested it.

I think what's going on is that the rsyncrypto he's using is unaware that 
cygwin uses a .lnk file to simulate a symbolic link. That would indicate a 
native Windows rsyncrypto.

Pierre
-- 
li fi'u vu'u fi'u fi'u du li pa

Re: rsyncrypto causes rsync rename errors

[Shachar S. <sh...@sh...>]

Hi Durham,

I usually don't make comments about such things, but the font you are 
using is extremely ill suited for deciphering technical information. 
Please help me try to help you.

Durham Computer Solutions Inc. wrote:
> I am running rsyncrypto under windows/cygwin, to encrypt
> and then using rsync/cygwin to rsync to another box runinng
> rsyncserver (under windows/cygwin)
>  
Are you running a native Windows rsyncrypto, or the cygwin one? I'll 
just add that, while I'm aware that a cygwin port of rsyncrypto is, 
theoretically, possible with the current sources, I have never tested it.
> rsync works fine if I don't encrypt with rsyncrypto
Does rsyncrypto work fine if you do not try to run rsync? In other 
words, do the files you encrypt decrypt correctly locally?
>  
> If I encrypt with rsyncrypto and then rsync the resulting
> files, I get rename errors:
>  
> examples of some of the errors:
>  
> rsync: rename "df/ed/c/Documents and Settings/ed/Application 
> Data/Microsoft/Office/Recent/.monitor.mdb.LNK.HHGv2l" (in os) -> 
> "df/ed/c/Documents and Settings/ed/Application 
> Data/Microsoft/Office/Recent/monitor.mdb.LNK": No such file or 
> directory (2)
I am unfamiliar with this error.

Please do an "ls -l" (from cygwin) of the directory, both with the 
original and with the encrypted files.

Also, please post the full rsyncrypto and rsync command line you are using.
> they all seem to windows .lnk (shortcut) files
Are they?
>  
> If I look at the properties of the encrypted .lnk files before sending,
> I get "This is not a valid shortcut"
Is that the link before or after rsyncrypto encrypts it?

Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com

rsyncrypto causes rsync rename errors

[Durham C. S. Inc. <ed...@du...>]

I am running rsyncrypto under windows/cygwin, to encrypt
and then using rsync/cygwin to rsync to another box runinng
rsyncserver (under windows/cygwin)

rsync works fine if I don't encrypt with rsyncrypto

If I encrypt with rsyncrypto and then rsync the resulting
files, I get rename errors:

examples of some of the errors:

rsync: rename "df/ed/c/Documents and Settings/ed/Application Data/Microsoft/Office/Recent/.monitor.mdb.LNK.HHGv2l" (in os) -> "df/ed/c/Documents and Settings/ed/Application Data/Microsoft/Office/Recent/monitor.mdb.LNK": No such file or directory (2)
rsync: rename "df/ed/c/Documents and Settings/ed/Recent/.email2db.lnk.2JotZb" (in os) -> "df/ed/c/Documents and Settings/ed/Recent/email2db.lnk": No such file or directory (2)

rsync: rename "df/ed/c/Documents and Settings/ed/Recent/.monitor.mdb.lnk.VyY2o9" (in os) -> "df/ed/c/Documents and Settings/ed/Recent/monitor.mdb.lnk": No such file or directory (2)

they all seem to windows .lnk (shortcut) files

If I look at the properties of the encrypted .lnk files before sending,
I get "This is not a valid shortcut"

any ideas?

Re: Option to exclude files?

[Shachar S. <sh...@sh...>]

Darren Share wrote:
> Hi,
>
> Thanks for this program, it works great.
>
> I've been hunting around for ages and there doesn't seem to be a way 
> to exclude certain files or directories in a path from being 
> encrypted. I know I can encrypt them and then exclude them from the 
> rysnc command but that still means having to store the encrypted files 
> locally for no good reason.
>
> The reason I need this is because I want to back up a directory, a 
> sub-directory of which contains my key files. If I encrypt the actual 
> key files I suspect I'm going to get some nasty recursion issues.
>
> If I'm wrong, please tell me how to do it.
>
> If I'm right, please consider this a feature request :)
I understand the feature request, but I really would rather not take 
rsyncrypto in that direction. The recommended way, right now, to do what 
you want is to pipe the output of "find" (that has more exclusions, 
inclusions and other filters than rsyncrypto will ever have) to the file 
list of things for rsyncrypto to encrypt. I believe this does exactly 
what you want.

Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com

Re: rsyncrypto does not back up either permission or ownership

[Shachar S. <sh...@sh...>]

Michael wrote:
> Hello
>
> there was a message from Paul  last answered at 2009-10-01. Paul
> discribes exactly the problem I have. Shachar I am very interested in
> fix you mention to preserve the permission and ownership. In case this
> is really may I ask if you could please implement the fix?
>
>   
At the moment, I have no time to work on rsyncrypto. I'm aware of the 
problem, and will try to get around to it soon. As always with open 
source, no promises  :-(
> Thanks a lot
> Michale
>   

-- 
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com

Option to exclude files?

[Darren S. <da...@mo...>]

Hi, 

Thanks for this program, it works great. 

I've been hunting around for ages and there doesn't seem to be a way to exclude certain files or directories in a path from being encrypted. I know I can encrypt them and then exclude them from the rysnc command but that still means having to store the encrypted files locally for no good reason. 

The reason I need this is because I want to back up a directory, a sub-directory of which contains my key files. If I encrypt the actual key files I suspect I'm going to get some nasty recursion issues. 

If I'm wrong, please tell me how to do it. 

If I'm right, please consider this a feature request :) 

Thanks. 

Darren. 

rsyncrypto does not back up either permission or ownership

[Michael <bus...@gm...>]

Hello

there was a message from Paul  last answered at 2009-10-01. Paul
discribes exactly the problem I have. Shachar I am very interested in
fix you mention to preserve the permission and ownership. In case this
is really may I ask if you could please implement the fix?

Thanks a lot
Michale

Paul van der Vlis wrote:
> Shachar Shemesh schreef:
>
>
>> At this point in time, rsyncrypto does not back up either permission or
>> ownership. It is planned for the future, but with no definitive time frame.
>>
>
> Maybe it is possible to save the ownership and permissions of all files
> into a file?
>
> Something like that is also used to store Linux backups on e.g. FAT.
>
> ( I make a local copy of the system before the encryption, so no risk on
> changings. )
>
>
One thing I can do quite easily is to have the backup files share the
same owners and permissions as the original. I can do that without
changing the file format (which is only planned for rsyncrypto 2, again
with no clear date in view).

Would that solve the problem for you?

Shachar

Re: Ownership and permissons?

[Paul v. d. V. <pa...@va...>]

Shachar Shemesh schreef:
> Paul van der Vlis wrote:
>> Shachar Shemesh schreef:
>>
>>   
>>> At this point in time, rsyncrypto does not back up either permission or
>>> ownership. It is planned for the future, but with no definitive time frame.
>>>     
>>
>> Maybe it is possible to save the ownership and permissions of all files
>> into a file?
>>
>> Something like that is also used to store Linux backups on e.g. FAT.
>>
>> ( I make a local copy of the system before the encryption, so no risk on
>> changings. )
>>
>>   
> One thing I can do quite easily is to have the backup files share the
> same owners and permissions as the original. I can do that without
> changing the file format (which is only planned for rsyncrypto 2, again
> with no clear date in view).
> 
> Would that solve the problem for you?

Maybe. Do you need to change the program to implement it?

A problem is that I would like to backup the files without root
privileges on the backup server. When all permissions and ownership
would be in a seperate file this would not be a problem.

Making a big tar.gz from the encrypted FS would be an option, but with
the disadvantage that I could not use hardlinks on the backupserver to
save diskspace.

With regards,
Paul van der Vlis.

Re: Ownership and permissons?

[Shachar S. <sh...@sh...>]

Paul van der Vlis wrote:
> Shachar Shemesh schreef:
>
>   
>> At this point in time, rsyncrypto does not back up either permission or
>> ownership. It is planned for the future, but with no definitive time frame.
>>     
>
> Maybe it is possible to save the ownership and permissions of all files
> into a file?
>
> Something like that is also used to store Linux backups on e.g. FAT.
>
> ( I make a local copy of the system before the encryption, so no risk on
> changings. )
>
>   
One thing I can do quite easily is to have the backup files share the 
same owners and permissions as the original. I can do that without 
changing the file format (which is only planned for rsyncrypto 2, again 
with no clear date in view).

Would that solve the problem for you?

Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com

Re: Ownership and permissons?

[Paul v. d. V. <pa...@va...>]

Shachar Shemesh schreef:

> At this point in time, rsyncrypto does not back up either permission or
> ownership. It is planned for the future, but with no definitive time frame.

Maybe it is possible to save the ownership and permissions of all files
into a file?

Something like that is also used to store Linux backups on e.g. FAT.

( I make a local copy of the system before the encryption, so no risk on
changings. )

With regards,
Paul van der Vlis.

Re: Ownership and permissons?

[Shachar S. <sh...@sh...>]

Paul van der Vlis wrote:
> Hello,
>
> When I encrypt (as root) a file what's owned by "paul", then the
> ownership becomes "root". After decryption (as root) the ownership is
> still root...
>
> Will rsyncrypto change the ownership of the files?
>
> And will it change the permissions?
>
> I would like to do a full system backup with rsyncrypto, but I need to
> keep the ownership and permissions.
>
> With regards,
> Paul van der Vlis.
>
>
>   
At this point in time, rsyncrypto does not back up either permission or 
ownership. It is planned for the future, but with no definitive time frame.

Sorry,
Shachar


-- 
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com

Ownership and permissons?

[Paul v. d. V. <pa...@va...>]

Hello,

When I encrypt (as root) a file what's owned by "paul", then the
ownership becomes "root". After decryption (as root) the ownership is
still root...

Will rsyncrypto change the ownership of the files?

And will it change the permissions?

I would like to do a full system backup with rsyncrypto, but I need to
keep the ownership and permissions.

With regards,
Paul van der Vlis.

Re: Rsyncrypto and openssl-0.9.8c+

[Milton C. <mi...@ca...>]

Shachar Shemesh wrote:
> :-( ). If you can send a patch, or at least send the line number in
> which this happens, that will be most helpful.

Patch attached.

Thanks.

-- 
Milton Calnek BSc, A/Slt(Ret.)
mi...@ca...
306-382-3657


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Re: Rsyncrypto and openssl-0.9.8c+

[Shachar S. <sh...@sh...>]

Milton Calnek wrote:
> Shachar Shemesh wrote:
>   
>> Milton Calnek wrote:
>>     
>>>   On the last call to aes_key::encrypt_block(), size is less than 16,
>>> for some reason
>>> AES_cbc_encrypt() has a seg fault when size is not the block size.
>>>
>>>   
>>>       
>> There are two options. The first is that rsyncrypto relies on openssl to
>> pad the size, or it does so itself. Off the top of my head, I find it
>>     
>
> I'll agree with you here... rsyncrpto passes a block of the proper size
> to AES_cbc_encrypt(). The problem I see is that we are reading the bytes read
> and passing that a the size of the block... when in fact the block is a fixed
> size.
>
>   
Ok, this is the kind of omission I'm actually likely to do. I'll have a 
look at it when I have some time (which, unfortunately, will not be soon 
:-( ). If you can send a patch, or at least send the line number in 
which this happens, that will be most helpful.
> I haven't actually read the buffer.get() yet, so I'm not entirely sure
> about what I just wrote.
>
>   
Shachar
>
> Yea: I have 0.9.8g repackaged.
>
>   


-- 
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com

Re: Rsyncrypto and openssl-0.9.8c+

[Milton C. <mi...@ca...>]

Shachar Shemesh wrote:
> Milton Calnek wrote:
>>
>>   On the last call to aes_key::encrypt_block(), size is less than 16,
>> for some reason
>> AES_cbc_encrypt() has a seg fault when size is not the block size.
>>
>>   
> There are two options. The first is that rsyncrypto relies on openssl to
> pad the size, or it does so itself. Off the top of my head, I find it

I'll agree with you here... rsyncrpto passes a block of the proper size
to AES_cbc_encrypt(). The problem I see is that we are reading the bytes read
and passing that a the size of the block... when in fact the block is a fixed
size.

I haven't actually read the buffer.get() yet, so I'm not entirely sure
about what I just wrote.



Yea: I have 0.9.8g repackaged.

-- 
Milton Calnek BSc, A/Slt(Ret.)
mi...@ca...
306-382-3657


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Re: Rsyncrypto and openssl-0.9.8c+

[Shachar S. <sh...@sh...>]

Milton Calnek wrote:
>
>   
> On the last call to aes_key::encrypt_block(), size is less than 16, for some reason
> AES_cbc_encrypt() has a seg fault when size is not the block size.
>
>   
There are two options. The first is that rsyncrypto relies on openssl to 
pad the size, or it does so itself. Off the top of my head, I find it 
extremely unlikely that it is the first option. I distinctly remember 
designing the padding function, and doing reviews to make sure no 
insecurities are introduced through it (or, at the very least, as little 
insecurities as possible). As such, if you see rsyncrypto sending a 
partial buffer to be encrypted, I think it must be a problem in 
rsyncrypto itself and not in openssl.

The main function is: why should that happen?

Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com

Re: Rsyncrypto and openssl-0.9.8c+

[Milton C. <mi...@ca...>]

Shachar Shemesh wrote:
> Milton Calnek wrote:
>> I'm going to try using a static block of 16 chars to see if that gets
>> me past the seg fault.
>>
>>   
> Debian Lenny currently carries openssl 0.9.8g, and rsyncrypto compiles
> and runs without a hitch there. Maybe it's a specific problem to your
> version?
> 

I'm still in the process of trying to compile 0.9.8g and 0.9.8k to see if that helps.

I see that F9 is using g... I'll try and use that src.rpm.

FWIW: here's what I see:
A file is encrypted... aes_key::encrypt_block() is called multiple times.
It's unlikely that a file would be exactly a multiple of 16 bytes, so there will
be some left over.

On the last call to aes_key::encrypt_block(), size is less than 16, for some reason
AES_cbc_encrypt() has a seg fault when size is not the block size.

When I change size (3rd arg to AES_cbc_encrypt()) to either 16 or sizeof(data),
I don't get a core dump.


-- 
Milton Calnek BSc, A/Slt(Ret.)
mi...@ca...
306-382-3657


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Re: Rsyncrypto and openssl-0.9.8c+

[Shachar S. <sh...@sh...>]

Milton Calnek wrote:
> Hi all,
>
> I was starting to use rsyncrypto when my distro updated from openssl-0.9.8b to openssl-0.9.8e.
> Once that happens, rsyncrypto dumps core.
>
> 1st question: are any of you using openssl-0.9.8c or newer?
>
> I've tracked the problem down to a call to AES_cbc_encrypt() in aes_crypt.cpp:99.
> rsyncrypto dumps core when there is part of a file left and we are trying to encrypt a
> block that is less than 16 chars.
>
> I'm going to try using a static block of 16 chars to see if that gets me past the seg fault.
>
>   
Debian Lenny currently carries openssl 0.9.8g, and rsyncrypto compiles 
and runs without a hitch there. Maybe it's a specific problem to your 
version?

Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com