Re: command line syntax

[Shachar S. <rsy...@sh...>]

Ok, Tom. You seem to have spent some time in trying to understand
rsyncrypto, but spread your conclusions among three emails :-). I'll try
to answer in a condensed manner.


Tom Metro wrote:

> The command line is specified as:
>
>        rsyncrypto [options] srcfile dstfile keyfile key
>
> which seems a little bit awkward, as the 3rd argument is sort of
> optional, but the 4rd is required.

All arguments are required. Src and dst are required for obvious
reasons. I'm afraid you slightly misunderstood the remaining two
arguments, which is why you think only one of them is required at every
given point.

Let's begin with the fourth argument - the RSA key (also known as
assymetric key). At some point called one of the keys "symetric RSA
key". There is no such thing. RSA is always assymetric, AES is always
symmetric. That's just how the algorithms are built.

Strike that. Let's first talk about encryption, and then decryption.

The fourth key has one function when encrypting. When encrypting, the
fourth key is used to embed a RSA encrypted version of the symmetric key
into the encrypted file. Since RSA encryption requires just the public
key, but the public key can be built from the private key, when
encrypting either the private or the public RSA key can be given as a
fourth argument.

The third argument, which is where the symmetric AES key is stored,
needs to be a file name when encrypting. If this file name exists, then
the key, encryption parameters and IV are taken from it. This is done so
that repeated encryption of the same file will be possible with minimal
changes to the file. It will do our rsync friendliness no good to
re-encrypt a file with a different key.

If the third argument is a name of a non-existing file, then rsyncrypto
assumes that this is the first time such a file is encrypted, and will
create the symmetric key file. This is so that future encryption will be
able to use the same encryption parameters, and thus create a file that
is similar.

Now let's talk about decryption. There are two possible decryption
modes. One is "warm restore", where you still have the symmetric key
file for the file you are trying to decrypt. If that's the case, you
still need to provide the (RSA assymetric) public key. This is mostly
for the technical reason that rsyncrypto needs to know how big the
encrypted symmetric key is in the file, and this depends on the size of
the public key. We could save it in the unencrypted key, but as you
should never ever lose the private key, much less the public key, this
did not seem like a big enough trouble to justify the extra work. Also,
this gives rsyncrypto a consistant command line across all invocations.
So, for warm restore, the public or private RSA key must be given as a
fourth argument, and an existing symmetric key file must be given for
third argument.

In the case of a total loss of all files, it is assumed you will still
have the private key. In the cold restore scenario, the third argument
will point to a non-existing file. This file will be created during the
restore process, so that future encryption and warm decryption can take
place. The fourth argument in such a case MUST be the private key, as we
will need to open the RSA encryption of the symmetric key inside the
encrypted file.

I hope this sheds more light on the command line uses.

> It also interleaves input and output files. I'd suggest:
>
>        rsyncrypto [options] pubkeyfile srcfile dstfile

Sorry, interleaving input and output file is pretty much unavoidable.
Sometimes the symmetric key file is input, sometimes it's output.
Putting the symmetric key between the source and the destination makes
no sense. Also, as rsyncrypto never accepts a file list (unless via file
or stdin), I allowed myself not to put the file as the last argument,
deviating from the standard unix argument order.

In any case, notice how the above example is wrong, as you need all four
arguments for all invocation modes.

> with encryption examples looking like:
>        rsyncrypto pubkeyfile srcfile dstfile
>        rsyncrypto --key=symkeyfile pubkeyfile srcfile dstfile
>        rsyncrypto -r pubkeyfile srcdir dstdir
>        rsyncrypto -r --key=symkeydir pubkeyfile srcdir dstdir
>        rsyncrypto --filelist pubkeyfile listfile dstdir

I don't think it's right to have an optional symmetric key. Not storing
a symmetric key would mean that you have to provide both a private key
(rather than a public key today) and a previously encrypted file (the
whole size of the file, rather than just 60 or so bytes) during
encryption. Right now, you have very little state to store between
encryptions, and you don't need the private key anywhere near the
encrypting machine unless a total disaster happens.

> Because only one of the two key files is needed, things get a bit
> messy. In the second case, it's easy enough to distinguish a file from
> a directory, but can you distinguish an rsyncrypto generated key from
> a public key? (Should be doable. One is binary.)

Like I said, both keys are needed always.

Moving on to your other email:

> After reading the "Quick use guide" here:
> http://sourceforge.net/docman/display_doc.php?docid=26727&group_id=129038
>
> I grabbed a certificate (cert) and private key (key) created with
> OpenSSL on my Debian box and attempted to run rsyncrypto 0.14 on a
> Windows NT system:
>
>   % rsyncrypto.exe src/file cryp/file key cert
>   Invalid version magic:

That's because you used an assymetric key where a symmetric key was needed.

> Shachar Shemesh writes:
> > AES keys (a.k.a. symmetric keys) are generated automatically by
> > rsyncrypto per encrypted files. This is what is stored in the "key"
> > file name you specify as the third parameter.
>
> Ah, so rsyncrypto creates the key file, rather than reading it. The
> guide, when it says stuff like:
>
>   If [the key file] already exists, the keys in it will be used for
>   this encryption.
>
> or
>   It can either exist, in which case the next parameter can be an x509
>   certificate with a public key, or not exist, in which case the next
>   parameter needs to be the private key corresponding to the public
>   key used during encryption. In the later case, the key file will be
>   created after decryption.
>
> makes it sound like supplying a key is required for encryption.

Which key? Supplying an RSA key is required for encryption. Supplying a
symmetric, AES key is not required. One will be created if you don't
supply it.

Maybe the guide needs to be more elaborate on the difference between the
two keys, but it is otherwise correct.

> After this was clarified, it made a lot more sense why the key
> parameter is a directory when you specify the -r switch.
>
>
> > The rsyncrypto manual points you to the req(1) and x509(1) manual
> > pages of openssl.
>
> Manual? Ah, I see it now in the source package. Apparently it didn't
> make it into the Win32 package.

It's not rsyncrypto's job to provide the man pages for openssl. Both
req(1) and x509(1) are openssl commands.

> The man page says:
>
>        rsyncrypto will encrypt files using a symmetric block
>        cipher (AES). Each file is encrypted using a unique key. The
>        file key is stored in two locations. One is the "key" file, and
>        the second is inside the encrypted file itself. The second copy
>        is encrypted using a RSA public key, which can be shared for
>        all encrypted files.
>
> which is a pretty clear description. However, saying "each file is
> encrypted using a unique key," is still vague.

Unique AES key, as mentioned in the previous sentence.

> Where does the key come from? Apparently it generated by rsyncrypto.
> That should be specified. Something like..."rsyncrypto generates a
> 128-bit (see -b switch) symmetric RSA

An RSA key cannot be symmetric, and in any case, rsyncrypto cannot
generate the RSA key for you. It's your job to create it. Rsyncrypto
only generates the AES keys.

> key and encrypts the file using symmetric block cipher (AES). It saves
> a copy of this key both in the encrypted file and optionally at the
> location specified by 'symkeyfile'. The former is encrypted using the
> 'pubkeyfile' (an X509 certificate), which is shared among all files,
> if processing multiple files."

Aside from the above comment, and the fact that the symmetric key is not
optional, feel free to send a patch against the documentation :-)

> Taking a step back for a moment, what's the benefit of having the
> individual keys stored in files? I'm sure there is some performance
> benefit when decrypting,

and when encrypting

> but for the typical user it seems like it would introduce an
> unnecessary complication and more files that need to be managed.

See begining of email for explanation. In short - it allows us to not
store the RSA private key, which is sensitive information, on the
machine that does the encryption.

> Have you considered having generation of the key files being optional
> and only enabled if a switch is specified?

I have now, and rejected the notion :-). Feel free to try and convince
me based on the newly gained knowledge.

> > Off the top of my head, the command line to generate would
> > probably be something like:
> >
> >  openssl req -new -nodes -x509 -out backup.crt -keyout backup.key
>
> I think that worked. It got rid of the 'Invalid version magic' error.
> Be good to add this to the man page. Actually, there should be a
> mini-HOWTO in the manual that explains how to generates the
> public/private keys, and how they relate to rsyncrypto. Much of the
> raw info for that is in the email I'm quoting.

It would be a greate service to the project if you wrote one, as I'm out
of time for such things right now.

> > The *.crt file is the certificate (public key) file. rsyncrypto
> > ignores just about all fields of the resulting certificate except
> > the actual key.
>
> Then I'm not sure why rsyncrypto didn't like my original certificate.
> It looked essentially the same, but had a bunch of headers before the
> 'BEGIN CERTIFICATE' block. (The original certificate was generated
> without the -nodes and -x509 options.)

there is a but in rsyncrypto, which means it cannot handle encrypted
certificates. The "nodes" option means that the private key is not
encrypted. This may be your problem (only applies to the key, not the
certificate).

Then again, the problem is more likely to do with you passing a private
RSA key where an AES key was expected.

> So after putting the new certificate (cert) and private key in place,
> I eventually got it to work in the recursive mode, after creating
> empty destination directories for the encrypted files (cbin) and the
> symmetric keys (keys):
>
>   % rsyncrypto.exe -r bin\ cbin\ keys\ cert
>
> and decrypted the files using:
>
>   % rsyncrypto.exe -d -r cbin\ ubin\ keys\ cert
>
>
> Along the way I erroneously used -c instead of -r and again got some
> less than useful error messages:
>
>   % rsyncrypto.exe -c bin cbin keys cert
>   file open failed: Input/output error
>
>   % rsyncrypto.exe -c bin\ cbin\ keys\ cert
>   file open failed: No such process

I'm not sure why that is.

> Also, after encrypting an individual file like so:
>
>   % rsyncrypto.exe bin/rsync.exe cbin/rsync.exe rsync.exe.key cert
>
> when attempting to decrypt it using:
>
>   % rsyncrypto.exe -d cbin/rsync.exe test.exe rsync.exe.key
> or
>   % rsyncrypto.exe -d cbin/rsync.exe test.exe cert
> or
>   % rsyncrypto.exe -d cbin/rsync.exe test.exe private
>
> rsyncrypto crashes.

The reason for the crash is the same reason I gave during your first
email. If you don't pass all mandatory arguments (the four file names)
libargtable segfaults.

> The second variation should have failed to decrypt, but the other two
> should have worked - no?

no. See above.

> If not, how do you decrypt a file when you've lost the symkey?
> Apparently like this:
>
>   % rsyncrypto.exe -d cbin/rsync.exe test.exe foo private
>
> where 'foo' is a non-existent file, which gets created. Technically
> the "Quick use guide" accurately describes how this works, but it's
> far from intuitive.

How else do you expect to recover the lost symmetric key?

> Regardless, none of them should have crashed.

See previous email. I'm not yet sure whether it's a bug in argtable or
rsyncrypto's use of argtable, but as it's non-crucial, I gave it a lower
priority.

> But:
>
>   % rsyncrypto.exe -d cbin/rsync.exe test.exe rsync.exe.key private
> or
>   % rsyncrypto.exe -d cbin/rsync.exe test.exe rsync.exe.key cert
>
> works, even though the second one doesn't make much sense.

If you have the symmetric key, you don't need the private key in order
to recover the data (warm restore). Try deleting rsync.exe.key, and then
try the second form again....

> I don't have a convenient build environment on Windows, but if I get
> the build problems worked out on my Debian box I'll see about adding
> verbosity to the error messages and look into the above crashes.

It seems you already have the build env for debian. As for the crash -
it somewhere in the argtable invocation when not enough mandatory
arguments were supplied.

Also, if you are going to be hacking the code, please try to work off
CVS. The "C" in "CVS" does, after all, stand for "concurrent", and I'm
doing rsyncrypto hacking too these days.

>  -Tom

Like I said - amendments and additions to the documentation will
probably be even more welcome than code hacks. If you do send patches,
please use "cvs diff -u" format.

          Shachar