Re: [BackupPC-users] very slow compression of big file

[Tony S. <sch...@bc...>]

On Jul 8, 2008, at 2:55 PM, Evren Yurtesen wrote:

> Tony Schreiner wrote:
>> On Jul 3, 2008, at 4:07 PM, Evren Yurtesen wrote:
>>> Renke Brausse wrote:
>>>> Hello Tony,
>>>>> I've written before about backups involving very big files  
>>>>> that  seem  to execute slowly.
>>>>>
>>>>> What can be slowing things down so much? Except for this   
>>>>> operation,  everything else runs about as I would expect.
>>>> I have no clue what the reason is but I experienced that backups of
>>>> large files are much faster with tar over ssh instead of rsync   
>>>> over ssh.
>>>> Not an explanation but maybe this can solve your problem.
>>> I believe the reason for this is how rsync works. It normally  
>>> tries  to transfer only the changed parts of the file. This is to  
>>> save  bandwidth, to do this, it has to scan the whole file on  
>>> both sides (I guess). This is unnecessary unless you are over  
>>> slow links. You  might want to try the whole-file option with rsync:
>>>
>>>         -W, --whole-file            copy files whole (w/o delta-  
>>> xfer algorithm)
>>>
>>> Please let us know the results, as a side-note if you still want  
>>> to  shrink the transferred file size you can use the ssh  
>>> compression  with -C option of ssh.
>>>
>>> Thanks,
>>> Evren
>> Reporting back on this.  Using the -W option did not make much   
>> difference. The dumps in question continue to run for a long time  
>> and  eventually fail with an ALARM.
>> For the time being I am excluding the really large files in question.
>> I may try tar instead of rsync at some point as was suggested.
>> Tony Schreiner
>
> Are you sure that -W option was active for sure? Did you check with  
> ps axwww to be sure? or just took it granted that it was active?
>
> Just asking because I have made several times similar mistake  
> myself :)
>
> By the way, if you are using compression, also enabling --checksum- 
> seed=32761 will make a dramatic speed difference. (although manual  
> suggests that this can be visible only on 3rd full backup).
>
> Thanks,
> Evren

Yes, unfortunately, I looked with ps and the  --whole-file option was  
set. I also have --checksum--seed=32761 set.
I'm willing to try again, in case I was hallucinating.

Tony