Re: Having rsync issues when modified part of file isnt at end

[Guillaume F. <gui...@fr...>]

Ok, did the tests without --inplace, it saves the BW, but completly
nullify the benefits of ZFS snapshots.

This is disturbing.

Le 2015/02/03 21:54, Guillaume Friloux a écrit :
> Hello, thanks for answering.
> 
> I have to use --inplace to limit writes on the ZFS dataset, otherwise
> each snapshot
> will use the total file size instead of only the diff.
> In my env, i dont do local copies, i do send over SSH on a BSD host.
> 
> I will redo all the tests without --inplace to see if it does better 
> for
> rsync
> (but wont be a real solution for my ZFS vol).
> 
> I am using /dev/urandom only because i wanted a simple test case, but
> the problem
> occurs with real files, like outlook PST files, PPT files and so on.
> 
> I intentionally not used gzip because gzip itself also produce some
> problems
> here with the files, and i do use --rsyncable, or tell rsyncrypto to 
> use
> gzip (encounter the problem with both methods).
> 
> Youre saying rsyncrypto uses gzip, but i did give --gzip=nullgzip that
> is
> a bash script calling cat, and so, not compression should be done, or 
> is
> rsyncrypto adding compression over what gzip did ?
> 
> 
> Le 2015/02/03 20:54, Shachar Shemesh a écrit :
>> rsyncrypto compresses as part of the encryption. You obviously did not
>> notice this, as you were using /dev/random as your source, and hence
>> producing uncompressible files. This is also the reason (at least part
>> of it) that the encrypted files were not the same size.
>> 
>>  On 03/02/15 10:34, Guillaume Friloux wrote:
>> 
>>> The only thing i can see is that between file1.iso.enc and
>>> file2.iso.enc,
>>> the filesize dropped a little, and between file2.iso.enc and
>>> file3.iso.enc it is higher,
>>> but i have no idea if this can be related...
>>  You are using rsync with --inplace. In that mode, rsync cannot reuse
>> blocks that were already overwritten by the destination file. When the
>> new file is bigger than the old one, you are overwriting the data you
>> would reuse while transferring, severely limiting rsync's ability to
>> optimize your transfer. If you remove --inplace, you will see that
>> rsync has no problem optimizing your encrypted files, no matter the
>> size changes.
>> 
>>  You have not asked your gzip question, but I am guessing it is either
>> the same issue there, or you forgot to pass it the --rsyncable flag.
>> 
>>  Shachar