#17 orphaned files after interrupt with --name-encrypt

[Alex Frase]

If rsyncrypto is interrupted before completion while using the --name-encrypt option, then any files which were encrypted before the interruption can be found on disk but are missing from the filemap, causing them to be forever orphaned. Even --delete and --delete-keys fail to clean them up.

I've attached a shell script that demonstrates this issue.

The problem seems to stem from the fact that rsyncrypto does not write the filemap until the very end, after all files have been processed. If it is interrupted, some encrypted files and their file-keys will be on disk, but there will be no filemap. Without --name-encrypt this isn't so bad since the already-encrypted files will be overwritten or skipped depending on --changed. But when using --name-encrypt, the second run of rsyncrypto will choose a new, different random name for each input file which will not match what was used the first time; thus the already-encrypted files (under different names) will not be overwritten or skipped. Furthermore, --delete-keys seems to assume that the filemap is complete and authoritative, so it will also not delete the orphaned encrypted files.

I can think of two possible solutions:

1) Ideally, just flush the filemap to disk after choosing the random encrypted name for each plaintext file, but before writing the key file or the encrypted data file. This would ensure that a filename mapping is always available for every encrypted file on disk, allowing an interrupted run to be resumed (with --changed) or restarted without leaving any orphaned files behind.

Of course this would add some IO overhead, especially for the encrypted copy of the filemap. The plaintext copy can just be appended, but the encrypted copy would have to be re-encrypted with every flush. Luckily, since is is rsyncrypto, only the tailing chunk of the file since the last IV reset would have to be updated. It might also be sufficient to only flush the plaintext file (which a re-run after interruption ought to have access to), and still wait to write the complete encrypted filemap until the very end.

2) Alternatively, make --delete and/or --delete-keys check for any files on disk which are not named in the filemap, and delete them just like any other file for which no plaintext counterpart still exists. This would not allow interrupted runs to be resumed, but it would at least clean up the orphaned files after re-running (to completion) with the --delete option.

Thanks for your excellent software! Please let me know if you need any more information.

[Alex Frase]

bash shell script to demonstrate the problem

[Shachar Shemesh]

Sorry about the long time it took to look at your report.

Option 2 seems the more likely, at this point. Another option is an external tool. Will have to think about it.