Re: [Dar-discussions] Binary delta in 2.6.0

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 25/12/2018 21:26, gulikoza via Dar-discussions wrote:
> Hi Denis,
> 

Hi Gulikoza,

> 
> 
> 
> 
> Congratulations on dar 2.6 release :-)
> 

Thanks!

> 
> 
> Thank you for all your hard work all these years!
> 
> I really enjoy working with dar, especially because of your
> attention to detail which makes dar very versatile tool.
> 
> 
> 
> Unfortunately I didn’t really have the time yet to test the binary
> delta feature before the final 2.6.0 release, but it’s just in time
> to make my yearly full backups with the signatures now and start
> testing binary deltas on production, haha ;-)

No problem. By the way, I've noted an important memory consumption
when using delta signatures, more than expected. I'm currently trying
to figure out whether reducing that memory footprint is possible and how
.

So to workaround that possible problem in the meanwhile, you should as
much precisely as possible specify the files to consider for binary
delta by mean of --include-delta-sig / --exclude-delta-sig options.

> 
> 
> 
> Just to let you know, there is an issue compiling dar 2.6 in RHEL-7
> due to gcc 4.8.5:
> 
> 
> 
> In file included from i_archive.hpp:35:0,
> 
> from archive.cpp:31:
> 
> erreurs.hpp:110:15: error: function 'libdar::Egeneric::niveau& 
> libdar::Egeneric::niveau::operator=(libdar::Egeneric::niveau&&)' 
> defaulted on its first declaration with an exception-specification
> that differs from the implicit declaration
> 'libdar::Egeneric::niveau& 
> libdar::Egeneric::niveau::operator=(libdar::Egeneric::niveau&&)'
> 
> niveau & operator = (niveau && ref) noexcept = default;
> 
> …
> 
> (^ and so on, basically all the prototypes with noexcept =
> default)

OK, I will look into this, thanks for feedback!

> 
> 
> 
> I understand that gcc 4.8.5 has a very limited (buggy) support for 
> c++11, but unfortunately I guess this means dar 2.6 will not be 
> available in EPEL-7.

Correct, it will be difficult, as dar strongly rely on some features
brought by C++11 and C++14, like move operator, smart pointers and
several other things that reduce memory footprint and/or CPU cycle.

> 
> It should probably also be stated in the documentation that gcc 4.8
> is no longer supported (if this was the intention).

The oldest gcc I've tested dar/libdar with is gcc 4.9 and it compiles
fine without even a warning :-) so I will upgrade the documentation
accordingly, thanks for this feedback.

> 
> 
> 
> I was able to compile it with devtoolset-7 (gcc 7.3.1), create a
> RPM (along with statically linked libthreadar) and the resulting
> RPM seems to work fine for now on vanilla Centos 7 (without the
> newer gcc and libs> installed).

OK, good!

> 
> 
> 
> Looking forward, I see that dar is using RS_DEFAULT_BLOCK_LEN which
> is just 2048 bytes. If the signatures will be switched to Blake2,
> this means the signature will be 36 bytes (256-bit blake2 + 4 bytes
> crc32) for each 2k block. This might be a bit too much for larger
> files and I was thinking that instead of having (yet another)
> command line option for that, dar might try to optimize the
> block_len based on file size. I don’t see the need for 2k
> signatures on multi-gigabyte files, where 8k or 16k signature might
> be just ok and a lot smaller. I have a proof of concept patch for
> that already, for instance with the 2GB file the blake2 catalog
> size is reduced from 36MB to 9MB with 8k blocks (default dar
> catalog with md4 signature would be around 18MB for the same
> file).

Well, reducing the size of signature more than it is already would
increase the risk of collision (unnoticed difference, ...), for that
reason I would rather let the user decide (yes, by adding a new option
...)

> I can send the patch to github if needed ;-)>
> 
> 
> And to nitpick a bit, the man page is not mentioning [Delta] in
> the [data] displayed fields. :-)

Oops, must fix that! Thanks!

> 
> 
> 
> 
> 
> Thanks again and Happy Holidays!

Thanks, Happy new year and I hope the Holidays were good to you too :-)

> 
> 
> 
> Regards,
> 
> gulikoza
> 

Regards,
Denis

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEOzEprx3d76WjfYGPCDGwvQPYsYIFAlwzfQgACgkQCDGwvQPY
sYLyxw/+JwP7IT5FJUnLyIkWnuv8xcBiPe5qg6cQLTFr7s2SK6M66lhxOBuHLU8h
aK7pVTaQDqf9b04sPw5CuuZ4Sb6PfzXNE3lX4+tG7daErzquoZoX73R7VN4xGeRW
n6YV0DtBZqbmWIyt8KhUwilQOkNCNmGdZAkhCY5E2YjPGJthQFJQv+DAGX7Fy2Cw
spvFu4K9EKcYAhMwcaQP4x/EH0MEEWF2d8sfAXeOmkKYgbonImDwJtn/03VOsGt2
HIrIVIoERAEd7CaJZHKltCJMu+QFEUkkhQbhGGVT1lX2MJqITM//ZW65f1net0H7
MUAVAvLcjlR1bz1zPBZadoz0baCOtcT9ObF4n/ndXfxruOUy4+jNeeY3eahG76wQ
0ZwMBZ1uDwnrtEg/3A4TF5adknQ7PFKAfg6SPlgrmppgaSSEtNVpp/KgiFhGKrG8
kNfPj868T8azitMIK7SgreVi7Z9CXF6cCbthTtsZFEPBhjErGP3IuWu8Ve4Fgy8Y
EFw/BZt0Rv07s6fYf1o+FiNypHnujThfDP7bQbsI1qOA2VQz5+8K5YMdXJBBnhqI
u8X/LId1EEEtEbo6llsj+wfdzsH3c2yX9YneFMwniE718nLmGvv3sVQgqFMwVMkb
3iUY8QQ0k6JZvppduSlwTbBP5gfGzDa/tIYTHpvuO4QbLWf2rSw=
=kR1R
-----END PGP SIGNATURE-----