rsyncrypto-devel — Development discussions, bug reports, and announcements for rsyncrypto

Congratulations!!! 0.16 has been released

[Shachar S. <rsy...@sh...>]

Hi list,


0.16 of rsyncrypto has just been released. I trust I don't have to tell
anyone on this list what's new in this release :-).


In many respects, this is the most important release of rsyncrypto ever.
This is not saying much, I guess, as the next two released are also
expected to bear the same title. To recap to anyone who was not listening:

0.16 - added support for file name encryption

This includes full support for both -r (recursive) and --filelist,
including support for --delete and --delete-keys on both previous
options. Program was tested on Windows and Linux - feedback on other
platforms welcome.


0.17 - planned change is the addition of signing of the encrypted files.
This will allow and external party to verify that the file was not
tampered with, without having the key to open the file.

0.18 - Meta information encryption. All relevant meta information will
be encrypted into the file, including modification time, owner,
permissions, and plaintext file name.

0.19 - integrate zlib into rsyncrypto, eliminating the need to rely on
an external "gzip" executeable.

Future feature and release numbers are, obviously, subject to change. I
have not, yet, decided when will be the right time to switch
rsyncrypto's status from "beta" to "release". It will either be 0.18 or
0.19, feature wise.

Waiting to hear your feedback,
             Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html

Re: Version 0.16-beta2 released

[Shachar S. <rsy...@sh...>]

Tom Metro wrote:

>> 0.17 will have signing of the encrypted file...
>
> What do you mean by that? Given that a public key is involved, aren't
> they essentially signed now?

No, the public key is only used to encrypt the symmetric key right now.

Signing means that an automatic process can verify, without being able
to decrypt the package, that it was signed by a specific key. This is
important if you happen to create more than one backup copy, and want to
make sure that you are not backing up compromised data.

>> The only thing missing from this beta release before it can be called
>> "0.16" is deleting encrypted versions of deleted plain text files when
>> the file names are encrypted. This is not going to be very easy to
>> implement efficiently, so it will take a little time.
>
>
> Are you referring to a scenario in which you have a source directory
> with files a, b, and c. You process them through rsyncrypto and get 3
> encrypted files with encrypted file names. Then file b is deleted from
> the source directory, and rsyncrypto is called upon to process the
> directory again?

Yes.

> What is the problem? That it is inefficient to go through the list of
> files in the destination directory, map their file names back to the
> unencrypted names, and check to see if the source files still exist?

It's not difficult for the "recursive" case. It is, however, difficult
when you use the "filelist" option, and some of the files you mention in
the file list are directories. In such a case, if file names are
encrypted, all files from all directories find their way to the same
directory. This means that we need to keep track which file came from
which directory. Not a huge task, but also not a straight forward one.

>  -Tom

Shachar


-- 
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html

Re: Version 0.16-beta2 released

[Tom M. <tme...@vl...>]

Shachar Shemesh wrote:
> Due to some feedback from Tom Metro
> (as well as some internal dialog), I've gone off the idea of having the
> external file hold all meta-information. Instead, the meta information
> will be encrypted inside the file itself...

I'm glad to hear my suggestions proved to be useful.


> 0.17 will have signing of the encrypted file...

What do you mean by that? Given that a public key is involved, aren't 
they essentially signed now?


> The only thing missing from this beta release before it can be called
> "0.16" is deleting encrypted versions of deleted plain text files when
> the file names are encrypted. This is not going to be very easy to
> implement efficiently, so it will take a little time.

Are you referring to a scenario in which you have a source directory 
with files a, b, and c. You process them through rsyncrypto and get 3 
encrypted files with encrypted file names. Then file b is deleted from 
the source directory, and rsyncrypto is called upon to process the 
directory again?

What is the problem? That it is inefficient to go through the list of 
files in the destination directory, map their file names back to the 
unencrypted names, and check to see if the source files still exist?

  -Tom


-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: http://tmetro.venturelogic.com/

Version 0.16-beta2 released

[Shachar S. <rsy...@sh...>]

Hi all,


First of all, does anyone even read these mails? I know there are quite
a few people subscribed to this list (about 11), and I just wanted to
make sure that I'm not just blowing air here.


Back to the subject at hand - beta 2 is out.

Main changes from Beta 1 - almost everything. Really.

The file format for storing the mapping between the encrypted and plain
text file names changed to a minimalistic format. In fact, if you run it
through "tr '\0' '\n'" you will get a plain text file (assuming no file
name contains a newline, of course). Due to some feedback from Tom Metro
(as well as some internal dialog), I've gone off the idea of having the
external file hold all meta-information. Instead, the meta information
will be encrypted inside the file itself (in the future - 0.16 still
doesn't have that). Each file will also know its own unencrypted file
name, so that we can regenerate the filemap file from the files present
in the directory (assuming we have the proper keys, of course). This
change is planned for 0.18 (0.17 will have signing of the encrypted
file, which is more important for Lingnu's backup service).

Due to this change, and to avoid confusion with the "filelist" option,
the whole terminology changed. The translation file is now called
"filemap". The command line options also changed accordingly.

The only thing missing from this beta release before it can be called
"0.16" is deleting encrypted versions of deleted plain text files when
the file names are encrypted. This is not going to be very easy to
implement efficiently, so it will take a little time.

Go get it, and tell me what you think (please, do tell me what you
think. Over 50 people downloaded beta1, and not a single feedback was
given).

             Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html

Change of format

[Shachar S. <rsy...@sh...>]

Hi all,

After some consideration, and following some of the discussion on this
list, I've decided to change the purpose of the "filelist" file. It will
no longer hold the whole meta-data. The filelist file will now only hold
the file name information. I.e. - encrypted filename, unencrypted file
name and directory seperator character.

A file format similar to the one in docs/filelist.txt WILL be used, but
as part of the encrypted file itself. For that reason, I'm not removing
the documentation, but rather just marked it as "not used".

          Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html

Change of format

[Shachar S. <rsy...@sh...>]

Hi all,

After some consideration, and following some of the discussion on this
list, I've decided to change the purpose of the "filelist" file. It will
no longer hold the whole meta-data. The filelist file will now only hold
the file name information. I.e. - encrypted filename, unencrypted file
name and directory seperator character.

A file format similar to the one in docs/filelist.txt WILL be used, but
as part of the encrypted file itself. For that reason, I'm not removing
the documentation, but rather just marked it as "not used".

          Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html

Beta 1 of version 0.16 released

[Shachar S. <rsy...@sh...>]

Hi all,

Rsyncrypto 0.16-beta1 is now available for download from the sourceforge
site. This version sports filename encryption, after long long last.

At the moment, this support is fairly incomplete (hence the "beta"
status). It will encrypt file names when doing recursive
encryption/decryption, but not when doing filelist
encryption/decryption. It will also not encrypt file names when doing
single file encryption/decryption, but this is not considerd a bug at
this stage.

Consideration as to what should be the right thing to do when using
"--meta-encrypt" and neither "-r" nor "--filelist" is still done. I
think that, for decryption, I'll assume that "src" is a name of a
directory to which encryption was done using one of "-r" or
"--filelist", diging out a single file from the archive. The exact
semantics for such a move are not clear, however. Your ideas welcome. In
any case, 0.16 will come out without support for that command line
combination.

Also note that compiling rsyncrypto on Windows does not currently work.
I've added a compatibility function to the compatibility layer, and have
not yet ported that new function to Windows. I'll only get around to
doing that after I finish getting filelist to work, so if anyone else is
interested in picking that task up, you're welcome to it. It should be
fairly easy. Just try to compile on Windows, see what function the
linker complains that is missing, and look at the Unix implementation of
a function with the same name to figure out what the function should do.

Most of all, however, I'm shipping this beta version to get your
feedback. Please let me know how it works out for you.

          Thanks,

                   Shachar
-- 
Shachar Shemesh
Lingnu Open Source Consulting ltd.
Have you backed up today's work? http://www.lingnu.com/backup.html

new version of argtable in Debian SID

[Shachar S. <rsy...@sh...>]

HI all,


I received a question a little while back about a segfault in latest
rsyncrypto (0.14 or CVS) when not passing enough parameters on the
command line. This was a problem in argtable2.


Debian SID should now (probably starting tomorrow) carry
libargtable2-0_2.4-2, which fixes this problem. If you are compiling
your own rsyncrypto, be sure to get the latest version.


                Shachar

Re: mmap failed: No such device

[Julien Le N. <jul...@en...>]

> Thanks for your help, I warn you I'm gonna ask for some more help in the 
> future, because I am trying to make a script that would allow me to 
> encrypt the files I want to backup, rsync them to a remote server, than 
> delete the working files on my computer. I have the feeling that I would 
> need some checking over this script, I don't want to discover it's wrong 
> when I need the back-up files ;-)

I am just trying to start with this script. I want to be able to perform 
a backup from any terminal by typing a command. I have already done that 
for a simple backup using rsync, but not having encrypted the files before.

I have one problem right at the beginning. My script looks like this 
(unwrapped):

#!/bin/sh

rsyncrypto -r home/jul/websites/ home/jul/working_backups/backweb/ 
home/jul/working_backups/keyweb/ home/jul/backup.key

and the result:

jul@ubuntu:~$ buex
/home/jul/.local/bin/buex: line 3:  8530 Segmentation fault 
rsyncrypto -r home/jul/websites/ home/jul/working_backups/backweb/ 
home/jul/working_backups/keyweb/ home/jul/backup.key

I am really sorry to annoy you with that, but I am willing to learn and 
read whatever documentation youcould point me at.

Cheers :-)
Julien

Re: mmap failed: No such device

[Julien Le N. <jul...@en...>]

> Hi Julien,
> 
> I believe you will find our ability to help you is greatly increased if
> you gave us a few details on your setup. Such details can include the
> version you are using, as well as the command line you used to perform
> the encryption and the decryption.

Many sorry, I wrongly assumed this was kind of a generic error.

And many more sorry as it seems to work today. I installed rsyncrypto 
yesterday and spent from the middle of the afternoon until the evening 
to install it and try to make it working (yes, I know, I am slow, but I 
am just learning :-) ). I kept having this error. I tried with the same 
commands than yesterday (I copy-pasted them) and it works. There must be 
a need for a reboot at some time and I missed it, although I rebooted 
several times.

Thanks for your help, I warn you I'm gonna ask for some more help in the 
future, because I am trying to make a script that would allow me to 
encrypt the files I want to backup, rsync them to a remote server, than 
delete the working files on my computer. I have the feeling that I would 
need some checking over this script, I don't want to discover it's wrong 
when I need the back-up files ;-)

Julien

> 
> 
>>Cheers,
>>Julien
> 
> 
>           Shachar
> 
> 
> -------------------------------------------------------
> SF.Net email is Sponsored by the Better Software Conference & EXPO
> September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
> Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
> Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
> _______________________________________________
> Rsyncrypto-devel mailing list
> Rsy...@li...
> https://lists.sourceforge.net/lists/listinfo/rsyncrypto-devel

Re: mmap failed: No such device

[Shachar S. <rsy...@sh...>]

Julien Le Nestour wrote:

> Hi everyone,
>
> First, thanks a lot for this idea and piece of software, I've long
> searched for one of this kind. I am not an advanced user of Linux, and
> I'm running the latest kubuntu, having installed rsyncrypto by adding
> the unstable to my sources. I have installed it, and resolve the
> dependencies issues by an apt-get -f install.
>
> I can run rsyncrypto to encrypt a directory, it seems to run smoothly
> and correctly. However, when I try to run it to decrypt this
> directory, or even to encrypt a single file, I have the following error.
>
> If you could help me running rsyncrypto, it would be very grateful.
> Any pointers or brief summary of the actions needed to be taken very
> welcome. :-)

Hi Julien,

I believe you will find our ability to help you is greatly increased if
you gave us a few details on your setup. Such details can include the
version you are using, as well as the command line you used to perform
the encryption and the decryption.

> Cheers,
> Julien

          Shachar

mmap failed: No such device

[Julien Le N. <jul...@en...>]

Hi everyone,

First, thanks a lot for this idea and piece of software, I've long 
searched for one of this kind. I am not an advanced user of Linux, and 
I'm running the latest kubuntu, having installed rsyncrypto by adding 
the unstable to my sources. I have installed it, and resolve the 
dependencies issues by an apt-get -f install.

I can run rsyncrypto to encrypt a directory, it seems to run smoothly 
and correctly. However, when I try to run it to decrypt this directory, 
or even to encrypt a single file, I have the following error.

If you could help me running rsyncrypto, it would be very grateful. Any 
pointers or brief summary of the actions needed to be taken very 
welcome. :-)

Cheers,
Julien

Re: man page in Windows distribution

[Shachar S. <rsy...@sh...>]

Tom Metro wrote:

>> I don't know of a tool that will easilly let me package it in a form
>> easilly readable on Windows. Suggestions welcome.
>
>
> Simply:
>
> % nroff -man rsyncrypto.man > rsyncrypto.txt
>
> plus appropriate line ending conversion, will do it.

No, it won't, but thanks anyways. I think I found a good way of
producing the right files.


          Shachar

Re: metadata

[Tom M. <tme...@vl...>]

Shachar Shemesh wrote:
>Tom Metro wrote:
>>Aren't you concerned that loss or corruption of 'filelist' could
>>render an entire collection of files as near useless?
> 
> No, not really. There is an unencrypted version, as well as an encrypted
> version.

There are two copies of 'filelist'? I guess I missed that in your write 
up. Though it doesn't change the situation much.

So one copy goes into the root of the destination directory, and gets 
encrypted, and the other copy goes into the keys directory and is left 
as plain text?


>>Why choose a single file model for this data, when you choose multi
>>file model for the symmetric keys?
> 
> Because, unlike the symmetric keys, this file has to be at a known
> location, and cannot be done without.
[...]
> You need the information in filelist
> in order to find which is the file you refer to. This means that if you
> wanted to store this information in seperate files, you will need to
> read each and every one of them anyways.
[...]
> If a file called "/etc/passwd" is stored in the encrypted archive as
> "As9sm23irmsk", and the only way to correlate the later name to the
> former is through a correlation data... 

Sounds like a pretty good point, though consider the usage scenarios.

I started with the assumption that the "meta files" would retain the 
original file name.

So to extract "/etc/passwd" the program would simply read in 
"keys/etc/passwd", and get the translation to 
"dest/a97a66d03c4a/As9sm23irmsk." (I presume you're scrambling directory 
names rather than mapping to a flat hierarchy.)

Are there any scenarios in which the program would be given the 
encrypted file name, and then need to locate the meta file? If you're 
doing a batch operation (decrypting all files), you could avoid the 
issue by iterating over the meta files instead of the encrypted files. 
Or if such an operation is rare, you simply accept the overhead and 
extract the meta data from the encrypted file's header (if it gets 
stored there).

Another trick that would make navigation to meta data stored in 
individual files faster would be to create a parallel hierarchy using 
hard links (which are supported on both UNIX and NTFS). Then 
"keys/a97a66d03c4a/As9sm23irmsk" resolves to the same file as 
"keys/etc/passwd." Though I'm not convinced that this is at all necessary.


As for the idea of storing the encrypted file name translation in the 
encrypted file's header...

> ...how on earth do you propose to
> store this correlation data inside "As9sm23irmsk", and encrypted at
> that. It means that if you ask to decrypt "/etc/passwd", rsyncrypto has
> to go over all the files in the archive, decrypting the private key
> header of each, and trying to locate the right one.

Yes, if you don't have the external "meta files" on hand.

I consider the meta files to be like a cache. They're nice to have to 
speed things up, but if you don't have them, and it necessitates 
decrypting all files (or their headers) in a set to find a specific 
file, that seems like a reasonable price.

Again, I'd consider usage scenarios. If the intended purpose of 
rsyncrypto is the storage of backup files, extraction will be a rare 
operation, and it is acceptable for it to be slow. Any user who recovers 
their files after a loss of the originals is going to be a happy user, 
and isn't going to mind that the process might be an order of magnitude 
slower than a simple copy of unencrypted files.


>>Consider that you can then use the same chunk of code to process the
>>meta data, regardless of where it was stored.
> 
> I think you are suggesting unimplementable solutions here...

Does the above clarification resolve that concern?


> Putting the data currently in the symmetric keys into the single file is
> an option to consider...

Interesting thought, but not an approach I'm voting for.


>>You said above that the symmetric key files really contain more than
>>the actual key, so why not extend it to include this additional meta
>>data?
> 
> Because the key file contains data about the encryption, while filelist
> contains data about the unencrypted file. It's just not the same thing.

Yet both are needed in order to recover the original (with the exception 
that if the 'filelist' files are lost, you're hosed).

Consider it from an operational standpoint: on initial encryption, 
you're writing stuff to both the symmetric key file and filelist 
describing how the file was packaged, and on decryption you are 
consulting both of those files to determine how to recover the original.

Practically speaking, there is little differentiating the two sources of 
information, except that a lost symmetric key is recoverable.


> This is even before I start talking about limited RSA block sizes and
> other technical problems with encrypting arbitrary length data using an
> assymetric cypher.

That's a good point. But given the choice between storing the complete 
meta data only in external files, or taking on the extra overhead of 
storing the meta data in an additional AES encrypted chunk as part of 
the encrypted file's structure, I'd take the latter.


>>  All strings are NULL terminated.
>>
>>Seems redundant if you're storing sizes,
> 
> I wasn't aware that I was storing sizes. Not of strings, in any case.

If a block is defined like:

   == Block 0002 - Encoded File Name ==

   2 bytes : block length
   2 bytes : block type, always 0002
   string : The name of the file (ASCII)

unless you add additional variable length elements to that block, you've 
effectively defined the length of the string.


>>unless you plan to pack multiple strings into a single block.
> 
> Could be necessary in the future, yes.

Right, so best to leave it as you have it.


>>What about a block and/or chunk checksum?
> 
> What good is a checksum if there is nothing you can do
> in case it's wrong?

You don't blindly create scrambled output when trying to restore a file. 
You can notify the user that there is a problem. You can write a 
recovery tool that cleans a 'filelist' by throwing out corrupted chunks, 
allowing at least partial recovery of the file set.


>>I'm not sure [and end block] serves a purpose. If the file is not corrupted,
>>then the chunk header tells you when you are done...
> 
> I would be delighted to hear in what way the chunk header provides this
> information.

You define the chunk as:

   == Chunk Format ==

   Each chunk is composed of a series of specific data. The first two
   bytes are the number of data blocks in this chunk.

If I know there are N blocks in a chunk, I know I'm done processing a 
chunk after I've seen N blocks. No need for an end block marker. Same 
deal as knowing the size of a sting vs. null terminated.


>> ...and if the file is corrupted, FFFF probably isn't adequately unique
>> to assist in reconstruction.
> 
>>If you stick with the idea of a single 'filelist' file, you might also
>>want to use a magic number to mark the start of each chunk.
> 
> Why?

If the start of a block has a unique identifier, you can write a 
recovery tool that can re-synch after scanning past one or more corrupt 
chunks that don't have the expect block count or block size.

  -Tom


-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: https://www.linkedin.com/e/fps/3452158/

Re: documentation patches

[Tom M. <tme...@vl...>]

Shachar Shemesh wrote:
> Tom Metro wrote:
>>Would you mind if we came up with a new name for this file in the
>>documentation?
> 
> Tom, with all due respect, I think it is time someone put their money
> where their mouth is. If you want to discuss documentation changes, do
> so with a patch against CVS.

I find it more respectful to discuss before patching, rather than 
patching first and asking for forgiveness later. :-) It also give other 
people an opportunity to voice an opinion.

And from a selfish perspective, it doesn't make sense to spend a lot of 
time developing a new document if it is going to be rejected.

The purpose of these discussions are both to improve my understanding of 
the project enough such that I can contribute with code or documentation 
patches, as well as to get a feel for what is acceptable, so I'm not 
pointlessly spinning my wheels.

What I'd like to do as a next step is post some proposed changes to the 
man page. If they sound reasonable, I convert them into a proper patch 
against the CVS trunk.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: https://www.linkedin.com/e/fps/3452158/

Re: symmetric keys

[Tom M. <tme...@vl...>]

Shachar Shemesh wrote:
> Tom Metro wrote:
>>Although if you compared two scenarios:
>>
>>1. keeping they keys, but throwing away the encrypted files after each
>>run;
>>
>>and
>>
>>2. keeping the encrypted files, but opting not to store the keys in
>>external files;
>>
>>I think #2 would be a big win. In most cases, the prior version of the
>>encrypted file will be left untouched. Occasionally, when a file has
>>changed, you'll need to decrypt the meta data in order to produce a
>>new encrypted file, but decrypting a few kilobytes of meta data from a
>>known location should be reasonably quick, as CPU time for
>>decompression/compression is proportional to the data quantity.
> 
> The keys directory will typically be a fraction the size of the
> encrypted files. How can you possibly compare the two?

This was a continuation of a thread in which one of the opening 
assumptions was that CPU was a more limited resource than disk storage. 
The "big win" I refer to above is with respect to CPU usage.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: https://www.linkedin.com/e/fps/3452158/

Re: metadata

[Shachar S. <rsy...@sh...>]

Tom Metro wrote:

> Shachar Shemesh wrote:
>
>> The symmetric key is all the information about the encryption procedure
>> (keys, parameters, etc.).
>
>
> Ah, so it already is a block of meta data.
>
> Would you mind if we came up with a new name for this file in the
> documentation?

Tom, with all due respect, I think it is time someone put their money
where their mouth is. If you want to discuss documentation changes, do
so with a patch against CVS.

> Aren't you concerned that loss or corruption of 'filelist' could
> render an entire collection of files as near useless?

No, not really. There is an unencrypted version, as well as an encrypted
version. The encrypted version is accessible through the same private
key that unlocks the rest of the encrypted files.

> Why choose a single file model for this data, when you choose multi
> file model for the symmetric keys?

Because, unlike the symmetric keys, this file has to be at a known
location, and cannot be done without. Putting the data currently in the
symmetric keys into the single file is an option to consider, but I'm
not sure I have managed to wrap my mind around all implications of doing
so yet.

Another reason just came to mind. You need the information in filelist
in order to find which is the file you refer to. This means that if you
wanted to store this information in seperate files, you will need to
read each and every one of them anyways.

> You said above that the symmetric key files really contain more than
> the actual key, so why not extend it to include this additional meta
> data?

Because the key file contains data about the encryption, while filelist
contains data about the unencrypted file. It's just not the same thing.

> I would think it would be worth breaking backwards compatibility for
> the vast benefits of having the block of meta data stored inside the
> file be identical to the block stored externally (with the exception
> that one is encrypted, of course).

See, that's the whole point, though, isn't it? If information crucial
for finding which file is which is of a different level of importance
than information about a specific file.

> Consider that you can then use the same chunk of code to process the
> meta data, regardless of where it was stored. And that you can ditch
> all the special case code you'll have to add for dealing with
> 'filelist'. And 'filelist', being a "sequence of 'chunks'," is
> essentially a database, which is bound to require even more code to
> manage, as well as introduce potential memory issues when dealing with
> huge file sets.

I guess I'll give you the same answer I gave you in my previous email. I
think you are suggesting unimplementable solutions here, but feel free
to prove me wrong by sending in patches against CVS. Grabbing the latest
CVS according to the instructions in the site will get you the most up
to date version I have, almost always.

> I don't follow why that requires either an external file or a separate
> file. Yes, an external file is necessary to avoid needing the private
> key on decryption, but you've already got an external meta data file.
> (And if the user doesn't have the external meta data file on hand,
> then they need the private key anyway.)

If a file called "/etc/passwd" is stored in the encrypted archive as
"As9sm23irmsk", and the only way to correlate the later name to the
former is through a correlation data, how on earth do you propose to
store this correlation data inside "As9sm23irmsk", and encrypted at
that. It means that if you ask to decrypt "/etc/passwd", rsyncrypto has
to go over all the files in the archive, decrypting the private key
header of each, and trying to locate the right one.

This is even before I start talking about limited RSA block sizes and
other technical problems with encrypting arbitrary length data using an
assymetric cypher.

> It would add to the prerequisites, but might have been less work to
> link in an XML parser. (One of the ideas behind XML is to write a
> decent parser once, and not have to reinvent one for every project.)

Could be. Too late for 1.0.0.16, but maybe in the future. See, the
parser is already writter :-).

> Otherwise the data structure seems decent. A magic number, which would
> permit locating the file or meta data chunk in the event of
> corruption. Variable number of blocks, and variable size blocks. And
> the concept that unknown block types should be ignored, helping to
> maintain backwards compatibility.

Actually, the magic number only serves to identify the file in case we
need to change the basic structure in the future (say, moving the file
over to XML format), while maintaining backwards compatibility. In any
case, thanks for giving me marks.

>   A writer must always issue all mandatory blocks for the file version
>   generated by it (as determined by the magic number at the start of the
>   file).
>
> You might want to make the magic number fixed and have the version be
> a separate attribute. Other programs/tools might want to be able to
> recognize the magic number, but only your program needs to be able to
> interpret the contents.

It's easier to just switch magics if something fundemental needs to be
changed. This also saves the trouble of trying to figure out how to
handle version 5 with magic 2 etc.

>   All strings are NULL terminated.
>
> Seems redundant if you're storing sizes,

I wasn't aware that I was storing sizes. Not of strings, in any case.

> unless you plan to pack multiple strings into a single block.

Could be necessary in the future, yes.

>   All blocks start on a file offset that is 4 bytes aligned. If a
>   natural block size is not a multiple of 4, writers must pad the block
>   with zero (null) bytes. The block length must include the padding, and
>   must divide by 4.
>
> What's the benefit of this? A bit of a performance boost once the
> structure is put into word-aligned memory?

Exactly.

> What about a block and/or chunk checksum?

What about them? What good is a checksum if there is nothing you can do
in case it's wrong?

>   == Block FFFF - End of Chunk ==
>
>   Writers must place this block at the end of each chunk. Readers should
>   assume that any data after this chunk is the begining of the next
>   chunk.
>
> I'm not sure that serves a purpose. If the file is not corrupted, then
> the chunk header tells you when you are done, and if the file is
> corrupted, FFFF probably isn't adequately unique to assist in
> reconstruction.

I would be delighted to hear in what way the chunk header provides this
information.

> If you stick with the idea of a single 'filelist' file, you might also
> want to use a magic number to mark the start of each chunk.

Why?

>   == Block 0000 - Platform ==
>   == Block 0001 - Original File Name ==
>   == Block 0002 - Encoded File Name ==
>   == Block 0003 - Posix File Permission ==
>
> What about an MD5 or SHA digest of the file, or is that stored
> elsewhere? What about the original file size, which could be utilized
> by -c?

Good ideas. I'll be expecting the patch by end of next week, which is
when 0.16 must, come rain or high water, be released.

> In your document you might also want to address that you aren't
> scrambling the files' time stamps, which theoretically is a leak of
> information, but a necessity in order for rsync to operate.

The document documents the filelist file format.

             Shachar

Re: symmetric keys

[Shachar S. <rsy...@sh...>]

Tom Metro wrote:

> Although if you compared two scenarios:
>
> 1. keeping they keys, but throwing away the encrypted files after each
> run;
>
> and
>
> 2. keeping the encrypted files, but opting not to store the keys in
> external files;
>
> I think #2 would be a big win. In most cases, the prior version of the
> encrypted file will be left untouched. Occasionally, when a file has
> changed, you'll need to decrypt the meta data in order to produce a
> new encrypted file, but decrypting a few kilobytes of meta data from a
> known location should be reasonably quick, as CPU time for
> decompression/compression is proportional to the data quantity.

The keys directory will typically be a fraction the size of the
encrypted files. How can you possibly compare the two?

> I don't think I can, as what I'm proposing is to make an existing
> required parameter, which is interleaved among other parameters,
> optional.

That was my thought too.

> If you feel the "API" is fixed, the best I can hope for is to get your
> approval for an option that would make storing symmetric keys
> optional. Which I could then combine with a wrapper script to present
> users with the most basic interface.

Sure. I see no problem with that. Send a patch and I'm sure we'll find a
way to put it in.

          Shachar

Re: improving encryption performance

[Shachar S. <rsy...@sh...>]

Tom Metro wrote:

> Shachar Shemesh wrote:
>
>>> ...something as simple as adding text to the top of a file would cause
>>> the entire file to appear changed.
>>
>>
>> Neither rsync...
>
>
> Obviously.
>
>
>> ...nor rsyncrypto currently suffer from this problem.
>
>
> Could you explain how rsyncrypto avoids it or refer me to a document
> that explains it?

It's in the archives.

>
>  -Tom
>

Re: metadata

[Tom M. <tme...@vl...>]

Shachar Shemesh wrote:
> The symmetric key is all the information about the encryption procedure
> (keys, parameters, etc.).

Ah, so it already is a block of meta data.

Would you mind if we came up with a new name for this file in the 
documentation?


> I'm adding a
> third location where file metadata is stored (file names, modification
> dates, permissions etc.). I don't think it's necessary to break
> backwards compatibility in order to store it inside the actual file.
> 
> http://cvs.sourceforge.net/viewcvs.py/rsyncrypto/rsyncrypto/docs/filelist.txt?view=markup.

   If the -m (--meta-encrypt) option is given, the file names, as well
   as other meta-data about the files, is garbled. ... The real
   information about all of tha above is stored inside a special file,
   called "filelist".

Aren't you concerned that loss or corruption of 'filelist' could render 
an entire collection of files as near useless?

Why choose a single file model for this data, when you choose multi file 
model for the symmetric keys?

You said above that the symmetric key files really contain more than the 
actual key, so why not extend it to include this additional meta data?

I would think it would be worth breaking backwards compatibility for the 
vast benefits of having the block of meta data stored inside the file be 
identical to the block stored externally (with the exception that one is 
encrypted, of course).

Consider that you can then use the same chunk of code to process the 
meta data, regardless of where it was stored. And that you can ditch all 
the special case code you'll have to add for dealing with 'filelist'. 
And 'filelist', being a "sequence of 'chunks'," is essentially a 
database, which is bound to require even more code to manage, as well as 
introduce potential memory issues when dealing with huge file sets.


>>Another issue to consider is how much, if any, of the meta data should
>>be encrypted (when part of the destination file)? Even though
>>requiring the private key in order to access it may be inconvenient,
>>probably makes sense to encrypt everything.
> 
> That's why I need to store the filenames in a seperate file.

I don't follow why that requires either an external file or a separate 
file. Yes, an external file is necessary to avoid needing the private 
key on decryption, but you've already got an external meta data file. 
(And if the user doesn't have the external meta data file on hand, then 
they need the private key anyway.)


quoting more from:
http://cvs.sourceforge.net/viewcvs.py/rsyncrypto/rsyncrypto/docs/filelist.txt?view=markup

   Also, in order to keep parsing of filelist simple, it is in binary
   format.

It would add to the prerequisites, but might have been less work to link 
in an XML parser. (One of the ideas behind XML is to write a decent 
parser once, and not have to reinvent one for every project.)

Otherwise the data structure seems decent. A magic number, which would 
permit locating the file or meta data chunk in the event of corruption. 
Variable number of blocks, and variable size blocks. And the concept 
that unknown block types should be ignored, helping to maintain 
backwards compatibility.


   A writer must always issue all mandatory blocks for the file version
   generated by it (as determined by the magic number at the start of the
   file).

You might want to make the magic number fixed and have the version be a 
separate attribute. Other programs/tools might want to be able to 
recognize the magic number, but only your program needs to be able to 
interpret the contents.


   All strings are NULL terminated.

Seems redundant if you're storing sizes, unless you plan to pack 
multiple strings into a single block.


   All blocks start on a file offset that is 4 bytes aligned. If a
   natural block size is not a multiple of 4, writers must pad the block
   with zero (null) bytes. The block length must include the padding, and
   must divide by 4.

What's the benefit of this? A bit of a performance boost once the 
structure is put into word-aligned memory?

What about a block and/or chunk checksum?


   == Block FFFF - End of Chunk ==

   Writers must place this block at the end of each chunk. Readers should
   assume that any data after this chunk is the begining of the next
   chunk.

I'm not sure that serves a purpose. If the file is not corrupted, then 
the chunk header tells you when you are done, and if the file is 
corrupted, FFFF probably isn't adequately unique to assist in 
reconstruction. If you stick with the idea of a single 'filelist' file, 
you might also want to use a magic number to mark the start of each chunk.


   == Block 0000 - Platform ==
   == Block 0001 - Original File Name ==
   == Block 0002 - Encoded File Name ==
   == Block 0003 - Posix File Permission ==

What about an MD5 or SHA digest of the file, or is that stored 
elsewhere? What about the original file size, which could be utilized by -c?

As I've implied above, I think this information, the original file size, 
time stamps, a digest of the file, and the AES key, should just be 
elements of a larger meta data chunk, which is stored both in the 
encrypted file (a separate chunk encrypted with the RSA key) and 
optionally also stored in an external file.

In your document you might also want to address that you aren't 
scrambling the files' time stamps, which theoretically is a leak of 
information, but a necessity in order for rsync to operate.


>> Have you looked at any existing schemes for storing file meta data,
>> such as zip or gzip file headers? There may be value in co-opting one
>> of those.
> 
> You obviously subscribed to the list after I put up the link to
> [document above]...

Correct, I hadn't seen it before.

But that doesn't answer the question (unless you are mimicking one of 
those - it's been a long time since I looked at Zip headers).

Originally I was thinking one of those projects would be a good 
reference just for the mechanics of storing meta data headers (rather 
than reinventing the wheel), but they could also be a valuable source of 
multi-platform code for getting/setting file attributes, and a reference 
for what is considered important to preserve.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: https://www.linkedin.com/e/fps/3452158/

Re: symmetric keys

[Tom M. <tme...@vl...>]

Shachar Shemesh wrote:
> Tom Metro wrote:
>>I would expect keeping around a copy of the encrypted files to be a
>>win, and thus a popular approach among rsyncrypto users.
> 
> Huh? If storage is cheap compared to CPU, wouldn't keeping around an
> extra 60 bytes file be better than decrypting said file from within the
> big file better?

True.

Although if you compared two scenarios:

1. keeping they keys, but throwing away the encrypted files after each run;

and

2. keeping the encrypted files, but opting not to store the keys in 
external files;

I think #2 would be a big win. In most cases, the prior version of the 
encrypted file will be left untouched. Occasionally, when a file has 
changed, you'll need to decrypt the meta data in order to produce a new 
encrypted file, but decrypting a few kilobytes of meta data from a known 
location should be reasonably quick, as CPU time for 
decompression/compression is proportional to the data quantity.


>>...a user who is looking for a practical
>>(rather than maximum) security environment may choose to ignore the
>>symmetric keys (if there was an option to turn them off) and use a
>>single private key for all operations. I'd start with this as the
>>basic model for rsyncrypto, while continuing to support the more
>>complex approaches for those who desire the greater security.
> 
> Well, it's too late to "start with this as the basic model". Rsyncrypto
> is approaching it's 16th release. If you can come up with a way that
> will not change the meaning of all existing invocations, I'll gladly
> consider it.

I don't think I can, as what I'm proposing is to make an existing 
required parameter, which is interleaved among other parameters, optional.

If you feel the "API" is fixed, the best I can hope for is to get your 
approval for an option that would make storing symmetric keys optional. 
Which I could then combine with a wrapper script to present users with 
the most basic interface.

I think presenting the simplest UI is important, so the solution I'd 
rather see is making the built-in argument syntax follow the simplest 
model, and if necessary, use wrapper scripts to maintain backwards 
compatibility with existing lingnu.com backup users. But obviously it's 
your call.


> Bear in mind that rsyncrypto was designed to be a tool in the Lingnu
> online backup service (http://www.lingnu.com/backup.html). It is
> therefor that scenario that takes outmost precedence in allocating my
> personal time.

Understood.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: https://www.linkedin.com/e/fps/3452158/

Re: improving encryption performance

[Tom M. <tme...@vl...>]

Shachar Shemesh wrote:
>>...something as simple as adding text to the top of a file would cause
>>the entire file to appear changed.
> 
> Neither rsync...

Obviously.


> ...nor rsyncrypto currently suffer from this problem.

Could you explain how rsyncrypto avoids it or refer me to a document 
that explains it?

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: https://www.linkedin.com/e/fps/3452158/

Re: man page in Windows distribution

[Tom M. <tme...@vl...>]

> I don't know of a tool that will easilly let me package it in a form
> easilly readable on Windows. Suggestions welcome.

Simply:

% nroff -man rsyncrypto.man > rsyncrypto.txt

plus appropriate line ending conversion, will do it.

Or wrap minimal HTML headers and PRE tags around it and give it an html 
extension.

  -Tom

-- 
Tom Metro
Venture Logic, Newton, MA, USA
"Enterprise solutions through open source."
Professional Profile: https://www.linkedin.com/e/fps/3452158/

Re: improving encryption performance

[Shachar S. <sh...@li...>]

Tom Metro wrote:

> Either way I guess you wouldn't end up with a very robust algorithm,
> as something as simple as adding text to the top of a file would cause
> the entire file to appear changed. I'd be curious to know if your
> block encryption approach suffers from that same problem.

Neither rsync nor rsyncrypto currently suffer from this problem. Any
change you suggest that does suffer is of no use.

>  -Tom

    Shachar

Re: man page in Windows distribution

[Shachar S. <rsy...@sh...>]

Tom Metro wrote:

> I was referring to the rsyncrypto man page, which isn't in the Windows
> distribution.
>
>  -Tom

I don't know of a tool that will easilly let me package it in a form
easilly readable on Windows. Suggestions welcome.


          Shachar