rsyncrypto-devel — Development discussions, bug reports, and announcements for rsyncrypto

Re: metadata

[Shachar S. <rsy...@sh...>]

Tom Metro wrote:

> Your comments above, as well as this:
>
>   --fk, --fr

...

> from the man page, plus your comments about storing encrypted file
> names, suggests that there is a need for richer meta data storage in
> the destination file.

Yes and no. There is a need for richer metadata storage, and I'm working
on such a thing right now. I don't see how that stems fromt he sources
you point out, though.

> It also suggests that the symmetric key files really want to be more
> than just keys, but instead a collection of meta data needed to
> reproduce an identical encrypted file (if the source data doesn't
> change).

The symmetric key is all the information about the encryption procedure
(keys, parameters, etc.). The file itself is the data. I'm adding a
third location where file metadata is stored (file names, modification
dates, permissions etc.). I don't think it's necessary to break
backwards compatibility in order to store it inside the actual file.

> Have you looked at any existing schemes for storing file meta data,
> such as zip or gzip file headers? There may be value in co-opting one
> of those.

You obviously subscribed to the list after I put up the link to
http://cvs.sourceforge.net/viewcvs.py/rsyncrypto/rsyncrypto/docs/filelist.txt?view=markup.
That's the format I'll be storing the metadata.

> Another issue to consider is how much, if any, of the meta data should
> be encrypted (when part of the destination file)? Even though
> requiring the private key in order to access it may be inconvenient,
> probably makes sense to encrypt everything.

That's why I need to store the filenames in a seperate file. In 0.16 the
encrypted directory will be a series of files of equal length names
composed of a series of random characters (well, a base64 encoding of a
few random bytes, if you want the specifics). The translation will be
stored in a file called "filelist", which will be itself encrypted using
the usual rsyncrypto mechanisms. The specifics of said file are detailed
in the link above.

>  -Tom

          Shachar

Re: symmetric keys

[Shachar S. <rsy...@sh...>]

Tom Metro wrote:

> If we presume that change is infrequent among the file set

granted

> and that storage is probably a less limited resource than CPU in many
> cases,

ok

> I would expect keeping around a copy of the encrypted files to be a
> win, and thus a popular approach among rsyncrypto users.

Huh? If storage is cheap compared to CPU, wouldn't keeping around an
extra 60 bytes file be better than decrypting said file from within the
big file better?

> (In an ideal scenario, you'd have encryption hardware, and you could
> simply patch rsync to perform its differencing against an encrypted
> version of the source file in memory.

That would be extremely difficult, due to the way rsync works.

>> ...it allows us to not store the RSA private key, which is sensitive
>> information, on the machine that does the encryption.
>
>
> Hmmm...is keeping the private key "private" really an advantage, given
> the way rsyncrypto works?

Not mandating it is an advantage. Whether it really does help in each
and every scenario - I don't know.

Bear in mind that rsyncrypto was designed to be a tool in the Lingnu
online backup service (http://www.lingnu.com/backup.html). It is
therefor that scenario that takes outmost precedence in allocating my
personal time. If you can come up with a usage scenario that does not
break the interface consistency, I'll be glad to put it in.

>> ...the public key can be built from the private key, when encrypting
>> either the private or the public RSA key can be given as afourth
>> argument.
>
>
> Even better. So that means a user who is looking for a practical
> (rather than maximum) security environment may choose to ignore the
> symmetric keys (if there was an option to turn them off) and use a
> single private key for all operations. I'd start with this as the
> basic model for rsyncrypto, while continuing to support the more
> complex approaches for those who desire the greater security.

Well, it's too late to "start with this as the basic model". Rsyncrypto
is approaching it's 16th release. If you can come up with a way that
will not change the meaning of all existing invocations, I'll gladly
consider it.

>> I have now, and rejected the notion :-). Feel free to try and convince
>> me based on the newly gained knowledge.
>
>
> I gave it a shot. Did it work?

Frankly, I don't think that the this use scenario is a priority to me.
Like I said, if you come up with a command line syntax that is
compatible with the current one and does what you want, and then be
willing to implement it, I believe I'll put it in.

          Shachar

Re: Debian packages, argtable

[Shachar S. <rsy...@sh...>]

Tom Metro wrote:

> The stable version is of course a couple of versions behind, so I
> think for now, as I'm mostly using the source as a point of reference
> for getting the 0.14 version working under Windows, I'll stick with
> the source distribution.

I just confirmed a small suspicion I had. It seems I tagged 0.15 for
release, but never actually released it. It fixes a tiny problem in
Windows regarding filelist and line termination.

>> Argtable was only added (also by me) to Sid after Sarge was already
>> released, so it won't be in stable. Sorry.
>
> So is it compiled in to the stable package as a static library? I
> noticed it isn't on the dependency list of either stable or unstable
> packages.

That's because it was only added to rsyncrypto as part of the port to
Windows. It will be in the dependency list when 0.16 is uploaded to
debian. 0.13 (latest Sid version) still used getopt_long. As 0.14 only
added Windows support over what 0.13 could do, I didn't find the need to
hurry and upload a new version.

>> I'm the argtable Debian maintainer :-)
>
> Well that'll cut down on finger pointing when getting to the bottom of
> that bug that might be in rsyncrypto or argtable.

I'm only the Debian maintainer, not the upstream. I already sent an
email with a reduced sample to Steward, who is the argtable maintainer,
though. We'll see whether it gets fixed.

>>> Attempting to configure rsyncrypto 0.14 on Debian 3.1 (Sarge/stable) I
>>> see:
>>>
>>>  configure: error: argtable2 not found
>>>  See `config.log' for more details.
>>
>
> Is there a way to get configure to spit out more friendly messages
> when a dependency isn't found? (I guess with most people using binary
> packages these days, there isn't much motivation to improve build
> tools to that level.)

It just told you that argtable2 was not found. What else would you have
configure print?

>> If a mandatory argument...is not given, you get a segfault.
>> I'm not yet sure whether the bug is in argtable or rsyncrypto.
>
>
> Does argtable have a test suite?

No, but I've attached to this email a small program that demonstrates
that this is an argtable bug, not an rsyncrypto bug.

       Shachar

duplication of rsync functionality

[Tom M. <tme...@vl...>]

The man page says:

  -r     Recursively  encrypt/decrypt  an entire directory structure.

  --trim=num
         Determine how many directory levels to trim from the
         beginning of  the  srcdir path when creating directories under
         dstdir.

  --filelist
         The first argument is...a file that has a list of files to
         process.

  -c     Check the src and destination modified time...prior to
         performing the operation. If the modified time is the same,
         assume that the files are also the same, and skip the
         operation.

  --delete
         When in recursive mode, delete files and directories under
         dstdir that do not exist under srcdir.


These rsyncrypto options come close to duplicating functionality already 
in rsync. Practically speaking, that maybe a necessity.

But it also raises the question of whether rsyncrypto could perhaps be a 
far simpler program - much closer to a filter - if it was integrated 
into rsync.

Not having looked at the rsync source, that may be far easier said than 
done.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: https://www.linkedin.com/e/fps/3452158/

improving encryption performance

[Tom M. <tme...@vl...>]

Could rsyncrypto improve its encryption performance by incorporating the 
rsync algorithm?

Instead of re-encrypting the entire file each time a change occurs, it 
could use an rsync-like algorithm to determine which blocks of a file 
need to be re-encrypted.

If I remember the details of the rsync algorithm correctly, you probably 
couldn't literally use the rsync algorithm, as you can't re-encrypt some 
arbitrary size block in the middle of a file. You'd have to stick with 
fixed blocks.

For this to work, you'd need to have the prior version of a changed file 
on hand to compare against. Or to avoid keeping local copies of the 
prior version you could "index" the files, storing a series of block 
offsets and SHA or MD5 digests. Then only the indexes would need to be kept.

Either way I guess you wouldn't end up with a very robust algorithm, as 
something as simple as adding text to the top of a file would cause the 
entire file to appear changed. I'd be curious to know if your block 
encryption approach suffers from that same problem.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: https://www.linkedin.com/e/fps/3452158/

Re: metadata

[Tom M. <tme...@vl...>]

Shachar Shemesh wrote:
> If the third argument is a name of a non-existing file, then 
> rsyncrypto assumes that this is the first time such a file is 
> encrypted, and will create the symmetric key file. This is so that 
> future encryption will be able to use the same encryption parameters, 
> and thus create a file that is similar.
> [...]
> Now let's talk about decryption. One is "warm restore", where you 
> still have the symmetric key file for the file you are trying to 
> decrypt. If that's the case, you still need to provide the (RSA 
> assymetric) public key. This is mostly for the technical reason that 
> rsyncrypto needs to know how big the encrypted symmetric key is in the 
> file, and this depends on the size of the public key. We could save it 
> in the unencrypted key, but as you should never ever lose the private 
> key, much less the public key, this did not seem like a big enough 
> trouble to justify the extra work. Also, this gives rsyncrypto a 
> consistant command line across all invocations.

Your comments above, as well as this:

   --fk, --fr
     If command line, or a version with different  defaults,  dictate
     different  values  for  the  --roll-*  options or the -b option,
     these will only affect files for  which  keyfile  does  not  yet
     exist.  specifying  the --fk or --fr will recreate keyfile if it
     has values different than those in the previous key file.

from the man page, plus your comments about storing encrypted file 
names, suggests that there is a need for richer meta data storage in the 
destination file.

It also suggests that the symmetric key files really want to be more 
than just keys, but instead a collection of meta data needed to 
reproduce an identical encrypted file (if the source data doesn't change).

Have you looked at any existing schemes for storing file meta data, such 
as zip or gzip file headers? There may be value in co-opting one of those.

Another issue to consider is how much, if any, of the meta data should 
be encrypted (when part of the destination file)? Even though requiring 
the private key in order to access it may be inconvenient, probably 
makes sense to encrypt everything.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: https://www.linkedin.com/e/fps/3452158/

Re: symmetric keys

[Tom M. <tme...@vl...>]

Shachar Shemesh wrote:
> Not storing a symmetric key would mean that you have to provide both a
> private key (rather than a public key today) and a previously encrypted
> file (the whole size of the file, rather than just 60 or so bytes)
> during encryption. Right now, you have very little state to store
> between encryptions...

Ah, this is a critical bit of rsyncrypto "philosophy" that I was 
missing. I ran across mention of this concept - needing only a small 
amount of state information - (though I don't see it in the guide, the 
man page, or the email I was quoting from) but it didn't add up at the time.

So this explains why key files exist as separate files. The idea is that 
you encrypt a batch of files, rsync the encrypted files to some remote 
location, then you have the option of blowing away the encrypted files - 
treating them essentially as temporary files. Subsequent encryption runs 
  then make use of your key files to produce new encrypted files that 
are minimally changed from the previous encryption cycle.

The reason why this logic wasn't apparent to me initially is that in my 
mind I was planning on keeping the encrypted files locally, so the keys 
are simply redundant.

If we presume that change is infrequent among the file set (reasonably 
safe, otherwise rsync wouldn't be an advantage) and that storage is 
probably a less limited resource than CPU in many cases, I would expect 
keeping around a copy of the encrypted files to be a win, and thus a 
popular approach among rsyncrypto users.

You must have been thinking along these lines as well, as you have a -c 
switch that skips encrypting files that haven't changed. (Have you 
considered optionally using MD5/SHA hashes instead of relying on 
timestamps?)

(In an ideal scenario, you'd have encryption hardware, and you could 
simply patch rsync to perform its differencing against an encrypted 
version of the source file in memory. That plus a bit of meta data 
stored in the file system to make it easier to determine which files 
have changed, ought to do it. But back to reality...)


> The third argument...the symmetric AES key...[is used] so that repeated
> encryption of the same file will be possible with minimal changes to the
> file. It will do our rsync friendliness no good to re-encrypt a file
> with a different key.

Right, another important point. As I was playing around with rsyncrypto 
repeatedly encrypting the same file I noticed a different key was 
generated each time. That made me wonder how changes could be minimized 
in the crypt text, but before I followed through on that thought, I went 
on to other things. Of course the answer, as you point out, is that if 
the key already exists, it's used for all subsequent encryptions of that 
file, thus keeping the crypt text deterministic.

This might also be thought of as a justification for having separate key 
files, but the key files can be extracted from the encrypted files in 
cases where the user is keeping them on hand.

Although, that then means...

> In the case of a total loss of all files, it is assumed you will still
> have the private key. ... The fourth argument in such a case MUST be the
> private key, as we will need to open the RSA encryption of the symmetric
> key inside the encrypted file.

...you need to have the private key on hand, even when encrypting, just 
so you can extract the symmetric key. This leads to your point:

> ...it allows us to not store the RSA private key, which is sensitive
> information, on the machine that does the encryption.

Hmmm...is keeping the private key "private" really an advantage, given 
the way rsyncrypto works?

Conceptually, this sounds appealing. I could see, for example, not 
wanting to have my private key be sitting on a shared web server that I 
might be rsyncing encrypted files from.

Lets consider some scenarios:

Lets assume on the machine doing the encryption, I only have the public 
key, and that machine gets compromised. As an attacker, what are my 
options? I can 1. use the symmetric keys, which rsyncrypto currently 
encourages me to keep on hand, and decrypt the files, or 2. I can look 
at the plain text of the original files, because they exist on the 
machine doing the encryption.

If some intermediary machine is compromised, say the machine where the 
encrypted files are archived, having the private key on the encryption 
machine hasn't altered the situation.

Sure, you could come up with a scenario where the encryption machine is 
like an appliance and only momentarily stores the plain text and key 
files before discarding them, but that doesn't seem to describe typical 
usage.

And no doubt that the private key is more valuable, in that it acts like 
a master key to unlock all your files. But practically speaking I don't 
think keeping the private key any more secure than the original plain 
text is buying you added security, and the "master key" nature can be 
mitigated by using different private keys for each collection of files.

A typical PKI scenario - where one party might be less trusted and can 
only encrypt using the public key - doesn't really fit rsyncrypto, as 
the party doing the encryption only needs the symmetric keys to be able 
to extract the plain text, and rsyncrypto gives them the keys (which 
they are then free to distribute to third parties). Of course the 
encrypting party also has access to the plain text. PKI is more about 
authentication than data protection.


> ...the public key can be built from the private key, when encrypting
> either the private or the public RSA key can be given as afourth argument.

Even better. So that means a user who is looking for a practical (rather 
than maximum) security environment may choose to ignore the symmetric 
keys (if there was an option to turn them off) and use a single private 
key for all operations. I'd start with this as the basic model for 
rsyncrypto, while continuing to support the more complex approaches for 
those who desire the greater security.


> I'm afraid you slightly misunderstood the remaining two
> arguments, which is why you think only one of them is required at every
> given point.

I'm not sure it's a misunderstanding so much as a different perspective. 
  If what I've discussed above is correct, then it does seem possible to 
eliminate the key files.


>>Have you considered having generation of the key files being optional
>>and only enabled if a switch is specified?
> 
> I have now, and rejected the notion :-). Feel free to try and convince
> me based on the newly gained knowledge.

I gave it a shot. Did it work?


> I hope this sheds more light on the command line uses.

Yes, indeed. Thanks for the detailed reply.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: https://www.linkedin.com/e/fps/3452158/

Re: man page in Windows distribution

[Tom M. <tme...@vl...>]

Shachar Shemesh wrote:
>>>The rsyncrypto manual points you to the req(1) and x509(1) manual
>>>pages of openssl.
>>
>>Manual? Ah, I see it now in the source package. Apparently it didn't
>>make it into the Win32 package.
> 
> It's not rsyncrypto's job to provide the man pages for openssl. Both
> req(1) and x509(1) are openssl commands.

I was referring to the rsyncrypto man page, which isn't in the Windows 
distribution.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: https://www.linkedin.com/e/fps/3452158/

Re: Debian packages, argtable

[Tom M. <tme...@vl...>]

Shachar Shemesh wrote:
> Actually, I was very suprised to find that rsyncrypto made it into
> stable at all.

Silly me, I didn't even think to check whether there was a Debian 
package for rsyncrypto. I guess I just assumed that at its early stage 
of development it wouldn't have made it in. But of course it's there:

http://packages.debian.org/stable/utils/rsyncrypto

The stable version is of course a couple of versions behind, so I think 
for now, as I'm mostly using the source as a point of reference for 
getting the 0.14 version working under Windows, I'll stick with the 
source distribution.


> Argtable was only added (also by me) to Sid after Sarge was already
> released, so it won't be in stable. Sorry.

So is it compiled in to the stable package as a static library? I 
noticed it isn't on the dependency list of either stable or unstable 
packages.


> I'm the argtable Debian maintainer :-)

Well that'll cut down on finger pointing when getting to the bottom of 
that bug that might be in rsyncrypto or argtable.


>>Attempting to configure rsyncrypto 0.14 on Debian 3.1 (Sarge/stable) I
>>see:
>>
>>  configure: error: argtable2 not found
>>  See `config.log' for more details.

Is there a way to get configure to spit out more friendly messages when 
a dependency isn't found? (I guess with most people using binary 
packages these days, there isn't much motivation to improve build tools 
to that level.)


>>Would it be practical to fall back to using getopt_long() if
>>libargtable2 isn't available?
> 
> I'm afriad not. Argtable was chosen because it has getopt_long
> compatible command line syntax, and yet works on Windows.
[...]
> Unfortunately, it is not compatible to getopt_long in terms
> of function usage.

Ah, that's what I was wondering.


> If a mandatory argument...is not given, you get a segfault.
> I'm not yet sure whether the bug is in argtable or rsyncrypto.

Does argtable have a test suite?

Shouldn't be too hard to code up a small test program that utilizes 
argtable to see whether it chokes on missing required arguments.

  -Tom


-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: https://www.linkedin.com/e/fps/3452158/

Re: command line syntax

[Shachar S. <rsy...@sh...>]

Ok, Tom. You seem to have spent some time in trying to understand
rsyncrypto, but spread your conclusions among three emails :-). I'll try
to answer in a condensed manner.


Tom Metro wrote:

> The command line is specified as:
>
>        rsyncrypto [options] srcfile dstfile keyfile key
>
> which seems a little bit awkward, as the 3rd argument is sort of
> optional, but the 4rd is required.

All arguments are required. Src and dst are required for obvious
reasons. I'm afraid you slightly misunderstood the remaining two
arguments, which is why you think only one of them is required at every
given point.

Let's begin with the fourth argument - the RSA key (also known as
assymetric key). At some point called one of the keys "symetric RSA
key". There is no such thing. RSA is always assymetric, AES is always
symmetric. That's just how the algorithms are built.

Strike that. Let's first talk about encryption, and then decryption.

The fourth key has one function when encrypting. When encrypting, the
fourth key is used to embed a RSA encrypted version of the symmetric key
into the encrypted file. Since RSA encryption requires just the public
key, but the public key can be built from the private key, when
encrypting either the private or the public RSA key can be given as a
fourth argument.

The third argument, which is where the symmetric AES key is stored,
needs to be a file name when encrypting. If this file name exists, then
the key, encryption parameters and IV are taken from it. This is done so
that repeated encryption of the same file will be possible with minimal
changes to the file. It will do our rsync friendliness no good to
re-encrypt a file with a different key.

If the third argument is a name of a non-existing file, then rsyncrypto
assumes that this is the first time such a file is encrypted, and will
create the symmetric key file. This is so that future encryption will be
able to use the same encryption parameters, and thus create a file that
is similar.

Now let's talk about decryption. There are two possible decryption
modes. One is "warm restore", where you still have the symmetric key
file for the file you are trying to decrypt. If that's the case, you
still need to provide the (RSA assymetric) public key. This is mostly
for the technical reason that rsyncrypto needs to know how big the
encrypted symmetric key is in the file, and this depends on the size of
the public key. We could save it in the unencrypted key, but as you
should never ever lose the private key, much less the public key, this
did not seem like a big enough trouble to justify the extra work. Also,
this gives rsyncrypto a consistant command line across all invocations.
So, for warm restore, the public or private RSA key must be given as a
fourth argument, and an existing symmetric key file must be given for
third argument.

In the case of a total loss of all files, it is assumed you will still
have the private key. In the cold restore scenario, the third argument
will point to a non-existing file. This file will be created during the
restore process, so that future encryption and warm decryption can take
place. The fourth argument in such a case MUST be the private key, as we
will need to open the RSA encryption of the symmetric key inside the
encrypted file.

I hope this sheds more light on the command line uses.

> It also interleaves input and output files. I'd suggest:
>
>        rsyncrypto [options] pubkeyfile srcfile dstfile

Sorry, interleaving input and output file is pretty much unavoidable.
Sometimes the symmetric key file is input, sometimes it's output.
Putting the symmetric key between the source and the destination makes
no sense. Also, as rsyncrypto never accepts a file list (unless via file
or stdin), I allowed myself not to put the file as the last argument,
deviating from the standard unix argument order.

In any case, notice how the above example is wrong, as you need all four
arguments for all invocation modes.

> with encryption examples looking like:
>        rsyncrypto pubkeyfile srcfile dstfile
>        rsyncrypto --key=symkeyfile pubkeyfile srcfile dstfile
>        rsyncrypto -r pubkeyfile srcdir dstdir
>        rsyncrypto -r --key=symkeydir pubkeyfile srcdir dstdir
>        rsyncrypto --filelist pubkeyfile listfile dstdir

I don't think it's right to have an optional symmetric key. Not storing
a symmetric key would mean that you have to provide both a private key
(rather than a public key today) and a previously encrypted file (the
whole size of the file, rather than just 60 or so bytes) during
encryption. Right now, you have very little state to store between
encryptions, and you don't need the private key anywhere near the
encrypting machine unless a total disaster happens.

> Because only one of the two key files is needed, things get a bit
> messy. In the second case, it's easy enough to distinguish a file from
> a directory, but can you distinguish an rsyncrypto generated key from
> a public key? (Should be doable. One is binary.)

Like I said, both keys are needed always.

Moving on to your other email:

> After reading the "Quick use guide" here:
> http://sourceforge.net/docman/display_doc.php?docid=26727&group_id=129038
>
> I grabbed a certificate (cert) and private key (key) created with
> OpenSSL on my Debian box and attempted to run rsyncrypto 0.14 on a
> Windows NT system:
>
>   % rsyncrypto.exe src/file cryp/file key cert
>   Invalid version magic:

That's because you used an assymetric key where a symmetric key was needed.

> Shachar Shemesh writes:
> > AES keys (a.k.a. symmetric keys) are generated automatically by
> > rsyncrypto per encrypted files. This is what is stored in the "key"
> > file name you specify as the third parameter.
>
> Ah, so rsyncrypto creates the key file, rather than reading it. The
> guide, when it says stuff like:
>
>   If [the key file] already exists, the keys in it will be used for
>   this encryption.
>
> or
>   It can either exist, in which case the next parameter can be an x509
>   certificate with a public key, or not exist, in which case the next
>   parameter needs to be the private key corresponding to the public
>   key used during encryption. In the later case, the key file will be
>   created after decryption.
>
> makes it sound like supplying a key is required for encryption.

Which key? Supplying an RSA key is required for encryption. Supplying a
symmetric, AES key is not required. One will be created if you don't
supply it.

Maybe the guide needs to be more elaborate on the difference between the
two keys, but it is otherwise correct.

> After this was clarified, it made a lot more sense why the key
> parameter is a directory when you specify the -r switch.
>
>
> > The rsyncrypto manual points you to the req(1) and x509(1) manual
> > pages of openssl.
>
> Manual? Ah, I see it now in the source package. Apparently it didn't
> make it into the Win32 package.

It's not rsyncrypto's job to provide the man pages for openssl. Both
req(1) and x509(1) are openssl commands.

> The man page says:
>
>        rsyncrypto will encrypt files using a symmetric block
>        cipher (AES). Each file is encrypted using a unique key. The
>        file key is stored in two locations. One is the "key" file, and
>        the second is inside the encrypted file itself. The second copy
>        is encrypted using a RSA public key, which can be shared for
>        all encrypted files.
>
> which is a pretty clear description. However, saying "each file is
> encrypted using a unique key," is still vague.

Unique AES key, as mentioned in the previous sentence.

> Where does the key come from? Apparently it generated by rsyncrypto.
> That should be specified. Something like..."rsyncrypto generates a
> 128-bit (see -b switch) symmetric RSA

An RSA key cannot be symmetric, and in any case, rsyncrypto cannot
generate the RSA key for you. It's your job to create it. Rsyncrypto
only generates the AES keys.

> key and encrypts the file using symmetric block cipher (AES). It saves
> a copy of this key both in the encrypted file and optionally at the
> location specified by 'symkeyfile'. The former is encrypted using the
> 'pubkeyfile' (an X509 certificate), which is shared among all files,
> if processing multiple files."

Aside from the above comment, and the fact that the symmetric key is not
optional, feel free to send a patch against the documentation :-)

> Taking a step back for a moment, what's the benefit of having the
> individual keys stored in files? I'm sure there is some performance
> benefit when decrypting,

and when encrypting

> but for the typical user it seems like it would introduce an
> unnecessary complication and more files that need to be managed.

See begining of email for explanation. In short - it allows us to not
store the RSA private key, which is sensitive information, on the
machine that does the encryption.

> Have you considered having generation of the key files being optional
> and only enabled if a switch is specified?

I have now, and rejected the notion :-). Feel free to try and convince
me based on the newly gained knowledge.

> > Off the top of my head, the command line to generate would
> > probably be something like:
> >
> >  openssl req -new -nodes -x509 -out backup.crt -keyout backup.key
>
> I think that worked. It got rid of the 'Invalid version magic' error.
> Be good to add this to the man page. Actually, there should be a
> mini-HOWTO in the manual that explains how to generates the
> public/private keys, and how they relate to rsyncrypto. Much of the
> raw info for that is in the email I'm quoting.

It would be a greate service to the project if you wrote one, as I'm out
of time for such things right now.

> > The *.crt file is the certificate (public key) file. rsyncrypto
> > ignores just about all fields of the resulting certificate except
> > the actual key.
>
> Then I'm not sure why rsyncrypto didn't like my original certificate.
> It looked essentially the same, but had a bunch of headers before the
> 'BEGIN CERTIFICATE' block. (The original certificate was generated
> without the -nodes and -x509 options.)

there is a but in rsyncrypto, which means it cannot handle encrypted
certificates. The "nodes" option means that the private key is not
encrypted. This may be your problem (only applies to the key, not the
certificate).

Then again, the problem is more likely to do with you passing a private
RSA key where an AES key was expected.

> So after putting the new certificate (cert) and private key in place,
> I eventually got it to work in the recursive mode, after creating
> empty destination directories for the encrypted files (cbin) and the
> symmetric keys (keys):
>
>   % rsyncrypto.exe -r bin\ cbin\ keys\ cert
>
> and decrypted the files using:
>
>   % rsyncrypto.exe -d -r cbin\ ubin\ keys\ cert
>
>
> Along the way I erroneously used -c instead of -r and again got some
> less than useful error messages:
>
>   % rsyncrypto.exe -c bin cbin keys cert
>   file open failed: Input/output error
>
>   % rsyncrypto.exe -c bin\ cbin\ keys\ cert
>   file open failed: No such process

I'm not sure why that is.

> Also, after encrypting an individual file like so:
>
>   % rsyncrypto.exe bin/rsync.exe cbin/rsync.exe rsync.exe.key cert
>
> when attempting to decrypt it using:
>
>   % rsyncrypto.exe -d cbin/rsync.exe test.exe rsync.exe.key
> or
>   % rsyncrypto.exe -d cbin/rsync.exe test.exe cert
> or
>   % rsyncrypto.exe -d cbin/rsync.exe test.exe private
>
> rsyncrypto crashes.

The reason for the crash is the same reason I gave during your first
email. If you don't pass all mandatory arguments (the four file names)
libargtable segfaults.

> The second variation should have failed to decrypt, but the other two
> should have worked - no?

no. See above.

> If not, how do you decrypt a file when you've lost the symkey?
> Apparently like this:
>
>   % rsyncrypto.exe -d cbin/rsync.exe test.exe foo private
>
> where 'foo' is a non-existent file, which gets created. Technically
> the "Quick use guide" accurately describes how this works, but it's
> far from intuitive.

How else do you expect to recover the lost symmetric key?

> Regardless, none of them should have crashed.

See previous email. I'm not yet sure whether it's a bug in argtable or
rsyncrypto's use of argtable, but as it's non-crucial, I gave it a lower
priority.

> But:
>
>   % rsyncrypto.exe -d cbin/rsync.exe test.exe rsync.exe.key private
> or
>   % rsyncrypto.exe -d cbin/rsync.exe test.exe rsync.exe.key cert
>
> works, even though the second one doesn't make much sense.

If you have the symmetric key, you don't need the private key in order
to recover the data (warm restore). Try deleting rsync.exe.key, and then
try the second form again....

> I don't have a convenient build environment on Windows, but if I get
> the build problems worked out on my Debian box I'll see about adding
> verbosity to the error messages and look into the above crashes.

It seems you already have the build env for debian. As for the crash -
it somewhere in the argtable invocation when not enough mandatory
arguments were supplied.

Also, if you are going to be hacking the code, please try to work off
CVS. The "C" in "CVS" does, after all, stand for "concurrent", and I'm
doing rsyncrypto hacking too these days.

>  -Tom

Like I said - amendments and additions to the documentation will
probably be even more welcome than code hacks. If you do send patches,
please use "cvs diff -u" format.

          Shachar

Re: build fails on Debian stable

[Shachar S. <rsy...@sh...>]

Tom Metro wrote:

> Attempting to configure rsyncrypto 0.14 on Debian 3.1 (Sarge/stable) I
> see:
>
>   configure: error: argtable2 not found
>   See `config.log' for more details.
>
>
> I'm not sure about the first failure, but the second is obviously due
> to a missing libargtable2. I searched the Debian package archives for
> packages containing the substring 'argtable', but found nothing in the
> stable distribution. I did find it in unstable:
>
> http://packages.debian.org/unstable/libdevel/libargtable2-dev

Actually, I was very suprised to find that rsyncrypto made it into
stable at all. It was only added to Sid about a month before Sarge was
labeled "Stable", long after the official feature freeze. Argtable was
only added (also by me) to Sid after Sarge was already released, so it
won't be in stable. Sorry.

>   Argtable is an ANSI C library for parsing GNU style command line

I know. I'm the argtable Debian maintainer :-)

> Would it be practical to fall back to using getopt_long() if
> libargtable2 isn't available?

I'm afriad not. Argtable was chosen because it has getopt_long
compatible command line syntax, and yet works on Windows. It was
necessary to find some sane way of accepting arguments in a compatible
way when doing the Win32 port. Unfortunately, it is not compatible to
getopt_long in terms of function usage. As such, allowing a dual
interface would be maintaining two code paths, something which I don't
want to do.

As argtable is fairly straightforward to install, I don't think it's
such big a deal.

> I proceeded by installing libargtable2-dev from unstable (which
> fortunately didn't have any dependencies and didn't require a
> cascading upgrade), and then configure succeeded.

Well, Sarge is gcc-3 while Sid is gcc-4. What I usually do in such cases
is to add a line saying "deb-src .... sid". You can then do "apt-get
source", followed by "dpkg-buildpackage", and get a locally created
package that uses the correct build environment for your distribution.
Still, as argtable is C, rather than C++, this is ok.

> rsyncrypto built without error, but a simple test of:
>
>   % ./rsyncrypto --help
>   Usage: rsyncrypt <src> <dst> <keys> <publickey file>
>   Options:
>   -h                   Help - this page
>   -d                   Decrypt.
> [...]
>   Currently only AES encryption is supported
>   Segmentation fault
>
> produces a seg fault.

A known bug. I'm not yet sure whether the bug is in argtable or
rsyncrypto. If a mandatory argument (the four files) is not given, you
get a segfault. If all mandatory arguments are given rsyncrypto is
currently valgrind clean. As I'm currently busy handling critical path
development (meta-data encryption), I have not yet researched this one.

Feel free to file a bug using either the rsyncrypto bug tracking system
or debian's reportbug. It's going to be me that gets the bug either way
:-). If you are using the later, though, make sure that you do it for
"sid", not "sarge".

> I see libargtable2 is near the top of the stack, so perhaps it does
> have a dependency on libc that just isn't documented. Perhaps I'll try
> building it from source.

No, this particular bug is not a result of any build anomalities you
did. Regardless, see above for recommended way of installing Sid
packages in Sarge.

> \ -Tom

       Shachar

command line syntax

[Tom M. <tme...@vl...>]

The command line is specified as:

        rsyncrypto [options] srcfile dstfile keyfile key

which seems a little bit awkward, as the 3rd argument is sort of 
optional, but the 4rd is required. It also interleaves input and output 
files. I'd suggest:

        rsyncrypto [options] pubkeyfile srcfile dstfile

with encryption examples looking like:
        rsyncrypto pubkeyfile srcfile dstfile
        rsyncrypto --key=symkeyfile pubkeyfile srcfile dstfile
        rsyncrypto -r pubkeyfile srcdir dstdir
        rsyncrypto -r --key=symkeydir pubkeyfile srcdir dstdir
        rsyncrypto --filelist pubkeyfile listfile dstdir

and decryption examples looking like:
        rsyncrypto -d prvkeyfile|symkeyfile srcfile dstfile
        rsyncrypto -d -r prvkeyfile|symkeydir srcdir dstdir

Because only one of the two key files is needed, things get a bit messy. 
In the second case, it's easy enough to distinguish a file from a 
directory, but can you distinguish an rsyncrypto generated key from a 
public key? (Should be doable. One is binary.)

Another approach might be to use named parameters for everything:

        rsyncrypto --pk=pubkeyfile --if=srcfile --of=dstfile

which is more cumbersome to type, but makes parameters position 
independent and makes it cleaner to handle things like decryption and 
the --filelist parameter:

        rsyncrypto -d --pk=pubkeyfile --filelist=listfile --of=dstdir

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: https://www.linkedin.com/e/fps/3452158/

error messages, documentation need improvement

[Tom M. <tme...@vl...>]

After reading the "Quick use guide" here:
http://sourceforge.net/docman/display_doc.php?docid=26727&group_id=129038

I grabbed a certificate (cert) and private key (key) created with 
OpenSSL on my Debian box and attempted to run rsyncrypto 0.14 on a 
Windows NT system:

   % rsyncrypto.exe src/file cryp/file key cert
   Invalid version magic:

Not exactly a helpful error message. So I downloaded the source package, 
grepped the files for the error message, and found that it had something 
to do with not liking the format of the certificate.

I dug through the mailing list archives and came across this helpful 
message:

http://sourceforge.net/mailarchive/message.php?msg_id=11809991

Shachar Shemesh writes:
 > AES keys (a.k.a. symmetric keys) are generated automatically by
 > rsyncrypto per encrypted files. This is what is stored in the "key"
 > file name you specify as the third parameter.

Ah, so rsyncrypto creates the key file, rather than reading it. The 
guide, when it says stuff like:

   If [the key file] already exists, the keys in it will be used for
   this encryption.

or
   It can either exist, in which case the next parameter can be an x509
   certificate with a public key, or not exist, in which case the next
   parameter needs to be the private key corresponding to the public
   key used during encryption. In the later case, the key file will be
   created after decryption.

makes it sound like supplying a key is required for encryption.

After this was clarified, it made a lot more sense why the key parameter 
is a directory when you specify the -r switch.


 > The rsyncrypto manual points you to the req(1) and x509(1) manual
 > pages of openssl.

Manual? Ah, I see it now in the source package. Apparently it didn't 
make it into the Win32 package.

The man page says:

        rsyncrypto will encrypt files using a symmetric block
        cipher (AES). Each file is encrypted using a unique key. The
        file key is stored in two locations. One is the "key" file, and
        the second is inside the encrypted file itself. The second copy
        is encrypted using a RSA public key, which can be shared for
        all encrypted files.

which is a pretty clear description. However, saying "each file is 
encrypted using a unique key," is still vague. Where does the key come 
from? Apparently it generated by rsyncrypto. That should be specified. 
Something like..."rsyncrypto generates a 128-bit (see -b switch) 
symmetric RSA key and encrypts the file using symmetric block cipher 
(AES). It saves a copy of this key both in the encrypted file and 
optionally at the location specified by 'symkeyfile'. The former is 
encrypted using the 'pubkeyfile' (an X509 certificate), which is shared 
among all files, if processing multiple files."


Taking a step back for a moment, what's the benefit of having the 
individual keys stored in files? I'm sure there is some performance 
benefit when decrypting, but for the typical user it seems like it would 
introduce an unnecessary complication and more files that need to be 
managed.

Have you considered having generation of the key files being optional 
and only enabled if a switch is specified?


 > Off the top of my head, the command line to generate would
 > probably be something like:
 >
 >  openssl req -new -nodes -x509 -out backup.crt -keyout backup.key

I think that worked. It got rid of the 'Invalid version magic' error. Be 
good to add this to the man page. Actually, there should be a mini-HOWTO 
in the manual that explains how to generates the public/private keys, 
and how they relate to rsyncrypto. Much of the raw info for that is in 
the email I'm quoting.


 > The *.crt file is the certificate (public key) file. rsyncrypto
 > ignores just about all fields of the resulting certificate except
 > the actual key.

Then I'm not sure why rsyncrypto didn't like my original certificate. It 
looked essentially the same, but had a bunch of headers before the 
'BEGIN CERTIFICATE' block. (The original certificate was generated 
without the -nodes and -x509 options.)


So after putting the new certificate (cert) and private key in place, I 
eventually got it to work in the recursive mode, after creating empty 
destination directories for the encrypted files (cbin) and the symmetric 
keys (keys):

   % rsyncrypto.exe -r bin\ cbin\ keys\ cert

and decrypted the files using:

   % rsyncrypto.exe -d -r cbin\ ubin\ keys\ cert


Along the way I erroneously used -c instead of -r and again got some 
less than useful error messages:

   % rsyncrypto.exe -c bin cbin keys cert
   file open failed: Input/output error

   % rsyncrypto.exe -c bin\ cbin\ keys\ cert
   file open failed: No such process


Also, after encrypting an individual file like so:

   % rsyncrypto.exe bin/rsync.exe cbin/rsync.exe rsync.exe.key cert

when attempting to decrypt it using:

   % rsyncrypto.exe -d cbin/rsync.exe test.exe rsync.exe.key
or
   % rsyncrypto.exe -d cbin/rsync.exe test.exe cert
or
   % rsyncrypto.exe -d cbin/rsync.exe test.exe private

rsyncrypto crashes. The second variation should have failed to decrypt, 
but the other two should have worked - no? If not, how do you decrypt a 
file when you've lost the symkey? Apparently like this:

   % rsyncrypto.exe -d cbin/rsync.exe test.exe foo private

where 'foo' is a non-existent file, which gets created. Technically the 
"Quick use guide" accurately describes how this works, but it's far from 
intuitive.

Regardless, none of them should have crashed.

But:

   % rsyncrypto.exe -d cbin/rsync.exe test.exe rsync.exe.key private
or
   % rsyncrypto.exe -d cbin/rsync.exe test.exe rsync.exe.key cert

works, even though the second one doesn't make much sense.


I don't have a convenient build environment on Windows, but if I get the 
build problems worked out on my Debian box I'll see about adding 
verbosity to the error messages and look into the above crashes.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: https://www.linkedin.com/e/fps/3452158/

build fails on Debian stable

[Tom M. <tme...@vl...>]

Attempting to configure rsyncrypto 0.14 on Debian 3.1 (Sarge/stable) I see:

   % ./configure
   checking for a BSD-compatible install... /usr/bin/install -c
   checking whether build environment is sane... yes
   checking for gawk... no
   checking for mawk... mawk
   checking whether make sets $(MAKE)... yes
   checking for g++... g++
   checking for C++ compiler default output file name... a.out
   checking whether the C++ compiler works... yes
   checking whether we are cross compiling... no
   checking for suffix of executables...
   checking for suffix of object files... o
   checking whether we are using the GNU C++ compiler... yes
   checking whether g++ accepts -g... yes
   checking for style of include used by make... GNU
   checking dependency style of g++... gcc3
   checking for AES_encrypt in -lcrypto... yes
   checking for arg_parse in -largtable2... no
   configure: error: argtable2 not found
   See `config.log' for more details.

   Some relevant bits from the log:

   configure:2235: g++ -c -g -O2  conftest.cc >&5
   conftest.cc: In function `int main()':
   conftest.cc:15: error: `exit' undeclared (first use this function)
   conftest.cc:15: error: (Each undeclared identifier is reported only
   once for each function it appears in.)
   configure:2241: $? = 1
   configure: failed program was:
   | /* confdefs.h.  */
   |
   | #define PACKAGE_NAME "rsyncrypt"
   | #define PACKAGE_TARNAME "rsyncrypt"
   | #define PACKAGE_VERSION "0.14"
   | #define PACKAGE_STRING "rsyncrypt 0.14"
   | #define PACKAGE_BUGREPORT ""
   | #define PACKAGE "rsyncrypt"
   | #define VERSION "0.14"
   | /* end confdefs.h.  */
   |
   | int
   | main ()
   | {
   | exit (42);
   |   ;
   |   return 0;
   | }
   [...]
   configure:2587: checking for arg_parse in -largtable2
   configure:2617: g++ -o conftest -g -O2   conftest.cc -largtable2
   -lcrypto  >&5
   /usr/bin/ld: cannot find -largtable2
   collect2: ld returned 1 exit status
   configure:2623: $? = 1
   configure: failed program was:
   | /* confdefs.h.  */
   |
   | #define PACKAGE_NAME "rsyncrypt"
   | #define PACKAGE_TARNAME "rsyncrypt"
   | #define PACKAGE_VERSION "0.14"
   | #define PACKAGE_STRING "rsyncrypt 0.14"
   | #define PACKAGE_BUGREPORT ""
   | #define PACKAGE "rsyncrypt"
   | #define VERSION "0.14"
   | #ifdef __cplusplus
   | extern "C" void std::exit (int) throw (); using std::exit;
   | #endif
   | #define HAVE_LIBCRYPTO 1
   | /* end confdefs.h.  */
   |
   | /* Override any gcc2 internal prototype to avoid an error.  */
   | #ifdef __cplusplus
   | extern "C"
   | #endif
   | /* We use char because int might match the return type of a gcc2
   |    builtin and then its argument prototype would still apply.  */
   | char arg_parse ();
   | int
   | main ()
   | {
   | arg_parse ();
   |   ;
   |   return 0;
   | }
   configure:2648: result: no
   configure:2659: error: argtable2 not found
   [...]


I'm not sure about the first failure, but the second is obviously due to 
a missing libargtable2. I searched the Debian package archives for 
packages containing the substring 'argtable', but found nothing in the 
stable distribution. I did find it in unstable:

http://packages.debian.org/unstable/libdevel/libargtable2-dev

   Argtable is an ANSI C library for parsing GNU style command line
   arguments. ... argtable can function as a "getopt_long" replacement,
   without the user of the program noticing the difference. Unlike
   "getopt_long", however, argtable is cross platform, working on all
   Posix systems, as well as Windows and Mac.

Would it be practical to fall back to using getopt_long() if 
libargtable2 isn't available?


I proceeded by installing libargtable2-dev from unstable (which 
fortunately didn't have any dependencies and didn't require a cascading 
upgrade), and then configure succeeded.

rsyncrypto built without error, but a simple test of:

   % ./rsyncrypto --help
   Usage: rsyncrypt <src> <dst> <keys> <publickey file>
   Options:
   -h                   Help - this page
   -d                   Decrypt.
[...]
   Currently only AES encryption is supported
   Segmentation fault

produces a seg fault.

A gdb backtrace shows:

   (gdb) bt
   #0  0x4027b39b in mallopt () from /lib/libc.so.6
   #1  0x4027a15f in free () from /lib/libc.so.6
   #2  0x40020b84 in arg_freetable () from /usr/lib/libargtable2.so.0
   #3  0x0804a60b in __tcf_1 () at rsyncrypto.h:148
   #4  0x40234b62 in exit () from /lib/libc.so.6
   #5  0x4021ee3e in __libc_start_main () from /lib/libc.so.6
   #6  0x08049f21 in _start () at ../sysdeps/i386/elf/start.S:102

I see libargtable2 is near the top of the stack, so perhaps it does have 
a dependency on libc that just isn't documented. Perhaps I'll try 
building it from source.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: https://www.linkedin.com/e/fps/3452158/

Re: Selecting encrypted files list

[Tom M. <tm...@vl...>]

Gary Holzer wrote:
> I am running cygwin as there doesnt seem to be a windows version of
> rsync

There are actually a few of them, but this one:

cwRsync
http://www.itefix.no/phpws/index.php?module=pagemaster&PAGE_user_op=view_page&PAGE_id=6

appears to be the most popular and up to date. It bundles a minimal 
Cygwin environment and an ssh client or server, as well as a few other 
tools for setting up encrypted tunnels. It's packaged as either a client 
or server installation, and the server version runs as a service under 
Windows.

It'd be good to get rsyncrypto bundled with it. Or at minimum, the Win32 
binary of the patched gzip. (I originally found rsyncrypto when 
searching for a Win32 binary of the patched gzip.)

  -Tom

News and development

[Shachar S. <rsy...@sh...>]

Hi all,

I just wanted to draw everyone's attention to the latest development. I 
am currently working on adding file meta information encryption to 
rsyncrypto.

The encryption is done by a substitution code. Each file name is 
assigned a constant length random name that bears no relation to the 
original name. The translation between the original name and the new 
name is stored in a file called "filelist", which is, itself, encrypted 
using the normal rsyncrypto mechanism.

I've started to write the specs for filelist. You can grab the latest 
CVS version (as soon as SF's webcvs catches up) from 
http://cvs.sourceforge.net/viewcvs.py/rsyncrypto/rsyncrypto/docs/filelist.txt?view=markup

Enjoy,

          Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting ltd.
http://www.lingnu.com/

Re: Some questions about rsyncrypto

[Shachar S. <rsy...@sh...>]

Michal 'hramrach' Suchanek wrote:

>This plus the explanation below makes me beleive it could work. Testing
>with a small 26M file shows it works which is great :)
>  
>

>The fact that some parts of ciphertext are the same ( and reflect
>similarities in the plaintext) does not sound too dangerous to me.
>
>But revealing the names of the files and their lenght could be a
>problem.
>  
>
Actually, next version will solve that problem. We'll rename files when 
we encrypt them, and keep an (encrypted) file with the translation. This 
will also back up other meta data (permission, special files, links 
etc.). The only information not encrypted will be the file's last 
modified time, and the size. Of course, the size will not be the same as 
before the encryption, due to compression and overheads, but it is still 
a quantity that cannot be said to be through a proper encryption algorithm.

>So I would rather get a solution where the files are split in blocks and
>stuffed into some block pool on the remote side.
>
I'm not sure I trust myself to perform block splitting at this time. 
This method also doesn't work well with rsync.

> Which probably means I
>could not use standard rsync. I need to be able to get some blocks
>unconditionally (those which describe the pool layout) and rsync others
>(and they would have to be concatenated before syncing to get anything
>efficient).
>  
>
Which means that the remote side has to know which blocks go with which. 
That effectively eliminates all of the advantages you gained from the 
splitting.

>Also I was thinking I could use something like ext2 filesystem to store
>the blocks but there is a problem with the inodes. If I simply encrypted
>them it would enlarge them and the whole thing would not fit together.
>On the other hand, I can just append some garbage to the data blocks
>so that everything is the same size.
>  
>
You're going the wrong way, I think. Anything that requires the site 
storing the encrypted file to know how things are pieced together may 
just as well not be done at all. If the remote site needs to know which 
block comes before which, we might as well not split into blocks at all.

>Thanks
>
>Michal Suchanek
>  
>
       Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting ltd.
http://www.lingnu.com/

Re: Some questions about rsyncrypto

[Michal 'h. S. <hra...@ce...>]

On Thu, Jul 21, 2005 at 12:35:41PM +0300, Shachar Shemesh wrote:
> Michal 'hramrach' Suchanek wrote:
> 
> >Hello
> >
> >First I wonder if rsynrypto does really solve the problem of syncing
> >effectively a file to which a single byte was inserted in the middle.
> > 
> >
> If we said we do, and you don't believe us, then why would asking again 
> change anything? :-)
> 
> >I myself would not hope any software would solve that for encrypted
> >files.  But I do not understand cryptography and a person who studied it
> >already assured me that it is likely possible to solve the problem.
> > 
> >
> Rsyncrypto will reset the encryption stream back to what it looked like 
> eventually, and so only the area around the inserted byte would have to 
> be resynced. Don't take my word for it, however. Feel free to test it 
> out. We now offer a 50% discount over our usual free of charge price tag 
> for users of rsyncrypto who only want to test it out.

This plus the explanation below makes me beleive it could work. Testing
with a small 26M file shows it works which is great :)

> 
> >Second I do not understand why a new key is generated for each file.
> >
> Common industry practice.
> 
> >Is
> >the encryption so severely wakened that one cannot afford encrypting
> >larger amount of data?
> >
> No, but if two files started the same, this would show up if they were 
> encrypted using the same key and IV. Since generating a symmetric key is 
> very easy, there is really no reason not to use different keys for 
> different files.
> 
> >What about files that are already gigabytes
> >or tens of gigabytes long?
> > 
> >
> If standard CBC were used, then the IV would not repeat itself (unless 
> by chance), which means that the attacker cannot deduct anything from 
> cipher text repetitions.
> 
> In rsyncrypto things are a little less simple. Long enough (about 16KB 
> after compression, IIRC) repetitions inside the same file will result in 
> repetitions in the cipher text. There is really no way to implement what 
> rsyncrypto is trying to do without having this effect. If this is a 
> major problem for you, I can only suggest that you not use rsyncrypto. 
> If the principle is ok, but you need larger repetition before cipher 
> text patterns appear, use higher "--roll-min" values. Bear in mind that 
> this will reduce the efficiency of rsync later. Then again, if you are 
> using files gigabytes long, rsync will likely choose rather large blocks 
> to work by, which means that it will not matter much.

The fact that some parts of ciphertext are the same ( and reflect
similarities in the plaintext) does not sound too dangerous to me.

But revealing the names of the files and their lenght could be a
problem.

So I would rather get a solution where the files are split in blocks and
stuffed into some block pool on the remote side. Which probably means I
could not use standard rsync. I need to be able to get some blocks
unconditionally (those which describe the pool layout) and rsync others
(and they would have to be concatenated before syncing to get anything
efficient).

But splitting the rsyncrypto output would probably get a bit tricky. In
case I am syncing a large file I would not like to store it encrypted
somewhere while rsync works on it. But getting some random part without
calculating the cipher from the beginning is probably not easy.
Especially since the file is compressed.

Also I was thinking I could use something like ext2 filesystem to store
the blocks but there is a problem with the inodes. If I simply encrypted
them it would enlarge them and the whole thing would not fit together.
On the other hand, I can just append some garbage to the data blocks
so that everything is the same size.

Thanks

Michal Suchanek

Re: Windows Rsyncrypto

[Shachar S. <rsy...@sh...>]

Barry James wrote:

>Shachar
>
>Thanks for the quick and comprehensive response. Will look out for the new
>download.
>  
>
It took two months from the point of first reporting. I'm afraid I 
cannot boast "quick" on this one :-(.

>Am I right in thinking that, given the nature of compression (or gzip
>compression at least) that repeating blocks are somewhere between unlikley
>and impossible?
>
Unlikely - yes. Impossible - not at all. With "rsyncable", gzip only 
sees about 30KB back into the file. Any repetition that is more than 
that apart, and gzip will not spot it. Needless to say, this also means 
that it will likely (eventually) get compressed into the same compressed 
stream. That is, after all, what rsyncable compression and encryption is 
all about.

> (and if not, would it not be a useful and simple matter to
>design a 'top-up' compression algorithm that eliminates repeating blocks by
>replacing them with a pointer - as I believe gzip does. This could even be
>recursive?).
>  
>
No, it would not. Pointing out repeptitions is what the Lempel Ziv that 
gzip employs does. For both performance and compression optimization 
considerations, the size of the back window that gzip looks at is 
limited. If you design a revised lempel ziv that scans the whole the 
file, you will both have terrible performance AND hurt your compression 
ratio.

The reason for the apparent illogic is that storing the pointer takes 
space too. The bigger the potential pointer needs to be, the greater the 
space we need to reserve to store it. I can think of a compromize, where 
only blocks that come at the start of a gzip "block" are compared, and 
only if they are large enough (say, 10KB) will they be referenced. This 
is, however, somewhat in the far future, at the moment. It will, 
however, eliminate many of the attacks (but not all) currently available 
on the rsyncrypto model.

>>It works excellent, so long as let rsyncrypto run it.
>>    
>>
>
>Is this because the blocks used for encryption and comression need to be
>aligned to get maximum rsyncability?
>  
>
Not aligned with one another, no. It's because the blocks have to be 
aligned with the same blocks prior to the change. Gzip only does that if 
you tell it "rsyncable".

>>rsyncrypto runs gzip with the "rsyncable" flag. If you compress the
>>files prior to encryption using this flag, you will make the compression
>>    
>>
> >ration slightly lower (for all sane files). If, however, you run gzip
> >without "rsyncable", or compress in any other way, you will totally
> >destroy the rsync friendliness of the system. In such a case, you are
> >much better off just encrypting with gpg.
>
>So am I right in thinking that 'sane' files encryted using the --rsyncable
>flag will do well - but not as well as those where rsyncrypto controls the
>compression becuase it is able to align the block boundaries - or is it that
>the file will be double compressed? Or something else?
>  
>
It's the double compress. There is no difference between running "gzip 
--rsyncable" and what rsyncrypto does. In fact, you can use the dummy 
gzip available inside the sources to eliminate the compression 
altogether, and then you can run gzip --rsyncable yourself, and you will 
get the exact same behavior.

          Shachar

-- 
Shachar Shemesh
Lingnu Open Systems Consulting
http://www.lingnu.com/

Re: Windows Rsyncrypto

[Shachar S. <rsy...@sh...>]

Donald Orbin wrote:

>I ran it from the command line and received an error
>that : :dynamic link library LIBEAY32.dll could not be
>found in the specified path"
>  
>
So it took two months and two days, but I finally got around to doing it.

The site now carries a new Win32 rsyncrypto zip file, that has the 
missing DLL. I'm sorry for the inconvineince. It's just that I am no 
longer sure where this DLL came from. Whether it's a version of OpenSSL 
that I compiled myself, or something I downloaded as part of a seperate 
project.

Hope this solves everyone's problems.

          Shachar

-- 
Shachar Shemesh
Lingnu Open Systems Consulting
http://www.lingnu.com/

Windows Rsyncrypto

[Barry J. <bar...@ww...>]

Shachar

Thanks for the quick and comprehensive response. Will look out for the new
download.

I may have a suggestion to alleviate one of the known issues - relating to
encryption strength - of making encryption rsync friendly which has been
raised (here I think?), but I need to check some assumptions before I risk
making too big a fool of myself ;-)

Am I right in thinking that, given the nature of compression (or gzip
compression at least) that repeating blocks are somewhere between unlikley
and impossible? (and if not, would it not be a useful and simple matter to
design a 'top-up' compression algorithm that eliminates repeating blocks by
replacing them with a pointer - as I believe gzip does. This could even be
recursive?).

> >Also can I ask does anyone know how well the included gzip.exe works with
in
>>combination with rsyncrypto?
>>
>It works excellent, so long as let rsyncrypto run it.

Is this because the blocks used for encryption and comression need to be
aligned to get maximum rsyncability?

> rsyncrypto runs gzip with the "rsyncable" flag. If you compress the
> files prior to encryption using this flag, you will make the compression
 >ration slightly lower (for all sane files). If, however, you run gzip
 >without "rsyncable", or compress in any other way, you will totally
 >destroy the rsync friendliness of the system. In such a case, you are
 >much better off just encrypting with gpg.

So am I right in thinking that 'sane' files encryted using the --rsyncable
flag will do well - but not as well as those where rsyncrypto controls the
compression becuase it is able to align the block boundaries - or is it that
the file will be double compressed? Or something else?

 > Finally is an encrypted
 >file larger than the original?
 >
> The encryption adds some overhead, yes. Expect an increase in the order
> of magnitude of 100 bytes per file + ~8 bytes per 12KB over the
> compressed size of the original file. I don"t think this is anything you
> should be worried about.
Yes - pretty insignificant - thanks.

Thanks

Barry (uktekky)

Re: Windows Rsyncrypto

[Shachar S. <rsy...@sh...>]

Barry James wrote:

>Hi
>
>Have managed to download OK and like previous posters have downloaded
>LIBEAY32.dll separatley.
>
>Unfortunately I am now also getting the message:
>
> "The ordinal could not be located in the dynamic link library
>LIBEAY32.dll."
>  
>
I'll try to fix the download section of the site today. It's been real 
busy lately.

>Also can I ask does anyone know how well the included gzip.exe works with in
>combination with rsyncrypto?
>
It works excellent, so long as let rsyncrypto run it.

> I guess that best results would be by
>compressing first, the encrypting - is that right?
>
rsyncrypto runs the compression itself. If you compress yourself using 
the supplied gzip (or in any other way) you will likely only make things 
between slightly and horribly worse.

rsyncrypto runs gzip with the "rsyncable" flag. If you compress the 
files prior to encryption using this flag, you will make the compression 
ration slightly lower (for all sane files). If, however, you run gzip 
without "rsyncable", or compress in any other way, you will totally 
destroy the rsync friendliness of the system. In such a case, you are 
much better off just encrypting with gpg.

> Finally is an encrypted
>file larger than the original?
>  
>
The encryption adds some overhead, yes. Expect an increase in the order 
of magnitude of 100 bytes per file + ~8 bytes per 12KB over the 
compressed size of the original file. I don't think this is anything you 
should be worried about.

>TIA
>
>
>
>Barry (uktekky)
>
>  
>
Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting ltd.
http://www.lingnu.com/

Windows Rsyncrypto

[Barry J. <bar...@ww...>]

Hi

Have managed to download OK and like previous posters have downloaded
LIBEAY32.dll separatley.

Unfortunately I am now also getting the message:

 "The ordinal could not be located in the dynamic link library
LIBEAY32.dll."

when I try to run it.

The DLL seems to be version 0, 9, 4, 0 (593,920 bytes). I assume from later
posts that this problem has been overcome but after a long search I can't
see the solution so any help would be appreciated.

Also can I ask does anyone know how well the included gzip.exe works with in
combination with rsyncrypto? I guess that best results would be by
compressing first, the encrypting - is that right? Finally is an encrypted
file larger than the original?

TIA



Barry (uktekky)

Re: Some questions about rsyncrypto

[Shachar S. <rsy...@sh...>]

Michal 'hramrach' Suchanek wrote:

>Hello
>
>First I wonder if rsynrypto does really solve the problem of syncing
>effectively a file to which a single byte was inserted in the middle.
>  
>
If we said we do, and you don't believe us, then why would asking again 
change anything? :-)

>I myself would not hope any software would solve that for encrypted
>files.  But I do not understand cryptography and a person who studied it
>already assured me that it is likely possible to solve the problem.
>  
>
Rsyncrypto will reset the encryption stream back to what it looked like 
eventually, and so only the area around the inserted byte would have to 
be resynced. Don't take my word for it, however. Feel free to test it 
out. We now offer a 50% discount over our usual free of charge price tag 
for users of rsyncrypto who only want to test it out.

>Second I do not understand why a new key is generated for each file.
>
Common industry practice.

> Is
>the encryption so severely wakened that one cannot afford encrypting
>larger amount of data?
>
No, but if two files started the same, this would show up if they were 
encrypted using the same key and IV. Since generating a symmetric key is 
very easy, there is really no reason not to use different keys for 
different files.

> What about files that are already gigabytes
>or tens of gigabytes long?
>  
>
If standard CBC were used, then the IV would not repeat itself (unless 
by chance), which means that the attacker cannot deduct anything from 
cipher text repetitions.

In rsyncrypto things are a little less simple. Long enough (about 16KB 
after compression, IIRC) repetitions inside the same file will result in 
repetitions in the cipher text. There is really no way to implement what 
rsyncrypto is trying to do without having this effect. If this is a 
major problem for you, I can only suggest that you not use rsyncrypto. 
If the principle is ok, but you need larger repetition before cipher 
text patterns appear, use higher "--roll-min" values. Bear in mind that 
this will reduce the efficiency of rsync later. Then again, if you are 
using files gigabytes long, rsync will likely choose rather large blocks 
to work by, which means that it will not matter much.

>Thanks
>
>Michal Suchanek
>  
>
       Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting ltd.
http://www.lingnu.com/

Some questions about rsyncrypto

[Michal 'h. S. <hra...@ce...>]

Hello

Unfortunately I cannot think of a more specific subject, I would just
like to clarify a few points which the documentation does not cover (or
does not state them clearly enough for me to understand).

First I wonder if rsynrypto does really solve the problem of syncing
effectively a file to which a single byte was inserted in the middle.

I myself would not hope any software would solve that for encrypted
files.  But I do not understand cryptography and a person who studied it
already assured me that it is likely possible to solve the problem.

Second I do not understand why a new key is generated for each file. Is
the encryption so severely wakened that one cannot afford encrypting
larger amount of data? What about files that are already gigabytes
or tens of gigabytes long?


Thanks

Michal Suchanek