dar-libdar_api — discussion area for developpers that want to use the libdar API

Re: [Dar-libdar_api] libdar::list_entry date

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/03/2014 21:20, Denis Corbin wrote:
> On 28/02/2014 21:53, Tobias wrote:
>> Hi Denis,
[...]
> 
> Yes, it will be possible in seconds starting release 2.4.13 and
> second and microsecond starting 2.5.0.

All this is now available from GIT respectively on branch
"branch_2.4.x" and on branch "master"

> 
> 
> 
>> Regards, Tobias
> 
> Regards, Denis.
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIVAwUBUxzMpQgxsL0D2LGCAQJK7A/8DM56lE/tPM2hkCKsxcPiOzshI3F61kHi
TbclPTp8uO/JdR2F3mhJzUR74zUCqeEO9VSodTpHyjS9/ljalDgndc8E7bNR7ols
dNbu+6MYn//8k+u4IYU+J5Fs3Tsedf/aLtPfT2i9CTIze1Xm7luNJsN0a5PIfGfK
8W/LztWDIdDAf55otYwcDaCiidCADkcUHYxW2Ttt12mtOxDRop5fxKMzxJ7ZKXD2
QDNCoZbxkMp1DD4fxQUYnL0sVRHB67OYrdaCDpH+YUBVG6itlm9aqZPbO/lDIe2A
SG0b+9LYQ+1S4VCpYxcp+sodAm3q4sCW2GVTJfo0eu1tEujoUfyWEWlHed3S0uPf
uv1xs/UYZHNn03wYCnGyDe18OVAcKOY0Ah2IbpYh0eq5M1+mIGxC0G4nEF5Hgmf3
Mt0eBp4IKbxgnHJop2G1uvea/KOJqIlZivjMEOUY6HGlQumOApmpcVOJfmjrLCE8
6eWfjKuZ85rWaX7OrZO3kzHJjIRd5viHt3Dkb3VXc/4NTuwRyOtpFS6BMulKzdnN
8dWp/E9M/ivFGmxHGtccTOC+ZiEXj1f6AJGn3rkMPBEUfeId5viVRYoUcUQuQBKo
RlCFPHyfNXMt4BjiZEnVGPvMV7YsjAZZM2PEH0J3jVfPRtyxUcNmpxYbiZ/T0OSy
3n9Ej152UrI=
=R/0N
-----END PGP SIGNATURE-----

Re: [Dar-libdar_api] libdar::list_entry date

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 28/02/2014 21:53, Tobias wrote:
> Hi Denis,

Hello Tobias,

Sorry for the delay answering you.

> 
> thank you for the new 2.4.12 release with the list_entry class.
> 
> Is it possible to get the last_access/modif/change dates as
> seconds since The Epoch?

Yes, it will be possible in seconds starting release 2.4.13 and second
and microsecond starting 2.5.0.


> 
> Regards, Tobias

Regards,
Denis.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIVAwUBUxY1gggxsL0D2LGCAQIU/Q//cE2EZ8GTVQXRouIqdH2IBKqWdZKtGUQp
q8w6ohod6m7YB0WjF/oDi0m4c82vYc9tolRiEfkVj4T9uBex64Ff6Vt27MZChzhy
aWYADcUUwg2DJlURCeffQRWbGcaaO8MnJRKmV6w1fnW9rM1naRA+RfDR/oeIVWMB
Ny5+pvurjuanXlw/SG2EBkH6tUhLWJdwyuFmWCYaau6kzDVyfWHlY8yVKomvvEeP
2R5VcpxVEN3+vWG9IlZdCGAObqf8MgR8fTEuTeNh2wMs4mXGNXsFG/A1WQ6kP+Un
/vQLuGRAIBFM6ZuPd1l/OKLcgGrf7om84kEpNxfMG71OyouDUWHT9zs1NYLNqgfl
t/uYKLI1I2Ojqe0SkuWFYIMhDhCKZlrYGe8p2iR/OczufpOvJxfADwssD7av1SDZ
IMvM8xnXzF0gUIRq+9zFXbyVk3VIy8fE7rJiiXMoAZUF+hO5FdSvN/UUgvx1OFBb
wGAXvUbQ0bGqqlbrzQUgRp9/FdppEtNk9PVxurF8V/lvH85FY1i/bVQbdrY+Dj88
CT8gZj422b7qQKjazfVOBI3UkGXmXoEgKFQ4sdUnWjhwDHHzaC9+8568+NvLkoQP
9xm1JBSPv/yJojpDo6KRALdH32h6rsqMdWXXTDH4GSrs7vuDjsYAas8fQi+/Jea5
Sho5GuwWer8=
=NT6p
-----END PGP SIGNATURE-----

[Dar-libdar_api] libdar::list_entry date

[Tobias <spe...@gm...>]

Hi Denis,

thank you for the new 2.4.12 release with the list_entry class.

Is it possible to get the last_access/modif/change dates as seconds
since The Epoch?

Regards,
Tobias

Re: [Dar-libdar_api] libdar::user_interaction::listing splitting the flag argument

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



On 31/10/2013 23:37, Tobias wrote:
> Hi Denis,

Hello Tobias,

> 
> I hope that helps you.

Sorry for the delay... but thanks for your help.

Well, I've check your files and the problem should not have subsisted. I
can't explain how you got it... "make install" should have fixed the
problem (or maybe you've installed in the default /usr/local keeping an
older copy in /usr which included files were used to compile your
program???)

Anyway, if you keep having this problem, don't hesitate to report it
here, we will find out why.

> 
> 

Kind Regards,
Denis.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIVAwUBUoUSHwgxsL0D2LGCAQK24g/+MBrE4TG5R+xGCsaeCoKMY7ZOg9ylNIWp
H/iS6uRfTBMgmUVQFTaKsJdhmXCMjJ7NVOoZ55mA59wC3JT+fE2csxKD8lLloWuU
PRwg5FYxzh2I7cgYlq8BbhPeMKVS6s3UyQ4SGirnRzS7wJeR8ytXa9PK5Soz2Ni2
RE+m2OZC0QEX1Qdq/CuBBzCn9V7CE/j941PA5ejT91a4XBxEc+ChARtySRTlqkn2
kJcOvfDrvbuPenSpUmK6ImMHJebf8ezo5cegl3kKSLd2eLucTyghpCk4CIesoYdd
zFu5WvU+r0LwPwn7WtWQqbyk6MCiiUm2ojfbwzqLe0vEQPtkSFTU9EQaftB6qakq
KRgmdMgICdy1EH2sPLWmJ88nj2rydJjs1BqcCgrSur4a67+COR4sjXvUqbn9bjrh
r3tEP8bQuckhmf98yMcuh48VgrPP2SoCPaRBGB5GRXqvLw04WAhwdj2MvvEfzgaG
Q2bYTqAxEwYlv6n9Fmf6zyuh7RxPTU0c+jqTZtpLg/LJMg6XGZijpfrkamIL3Zcq
SzoI09ndhQQlvJGRyVfiEDIxznmBaaPw/PRp+5TAkfVVCaISaz2IoOZnKlNKPUK6
J0zqSOpwKZkLHiRH60qUZX7FOGkhkTgDwD6ESDI6TRy0u30z8gQQrgtMVgVRyuPE
6WCtGg36EZ4=
=cyCj
-----END PGP SIGNATURE-----

Re: [Dar-libdar_api] libdar::user_interaction::listing splitting the flag argument

[Tobias <spe...@gm...>]

Hi Denis,

I hope that helps you.


Am Mittwoch, den 30.10.2013, 19:30 +0100 schrieb Denis Corbin:
> On 29/10/2013 21:00, Tobias wrote:
> > Hi Denis,
> 
> Hello Tobias,
> 
> > 
> > The libdar_config.h problem seems not to be solved.
> 
> :-/ Strange. Can you send me the libdar_config.h (as it is installed)
> and the config.h file generated at the root of the source tree?
> 
> > But get_children_in_table is working now, also with other dar archives.
> > Thank you very much.
> 
> Great :)
> 
> > 
> > Regards,
> > Tobias
> > 
> ------------------------------------------------------------------------------
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> _______________________________________________
> Dar-libdar_api mailing list
> Dar...@li...
> https://lists.sourceforge.net/lists/listinfo/dar-libdar_api

Re: [Dar-libdar_api] libdar::user_interaction::listing splitting the flag argument

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 29/10/2013 21:00, Tobias wrote:
> Hi Denis,

Hello Tobias,

> 
> The libdar_config.h problem seems not to be solved.

:-/ Strange. Can you send me the libdar_config.h (as it is installed)
and the config.h file generated at the root of the source tree?

> But get_children_in_table is working now, also with other dar archives.
> Thank you very much.

Great :)

> 
> Regards,
> Tobias
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIVAwUBUnFQRggxsL0D2LGCAQIBQRAAxbNtEPj0fbCq6ualMWBlS9C3xtChVWMw
6BqgK5kSYBDPZsK8y7fHmOb9HA1dfVNMpwz1+bG1LrTp/DX9jbEUNHYT8reM1wTm
p+PxoCQ4FrzhOL13uHSdnthpKELnSjp0aWdYyUJlrj6M/c4aGEUhmUsu4WPo6lPn
DE3JL7kgLTSmSicyqeM3jwAIeNYhRieTqrc7jEQmr6j+Z+IeHQI5+7cM0n2TQ2/A
BNLdKd0t1owOYjnipTeipa69qWwfrh9YzgmYX/ss84ZhPCvwcMIsWcY3Eko7fmRg
iavgjPZerViKlOugMqOXOC04NMph3jqVDmFoRqxRAzxa+yxxuphvWov8bD597KYx
hR76Okp2sT1P08f3FsWDmM6p1R8V6b5vfbyeufzj8MANWQ53urrbrFfFfvY0/EPV
aqTI5WoMu/QlTWOZU0BiJ3jLq3X308dLSRNM697PVe9YRN6Y7FuKVHtI7FCr1BNd
X076MNq/xLDxVzQYLmxdvvJyqO57yhv8XI3tPlH1n5QT1upXZKCHb+TieLafbgD2
sW4VWY9yUt7eVRQLbBbR9M3LXYfZ+RHQFIv3WEIBnQHnarnxowi+uM1ZttMa7z4W
UbFbG5WALTBSUlDj0S/O+vFmnhY5j4mO+0SN5dMBq0dWr0vtoUjPCA+ixHHElLd7
87ejtqjbM9o=
=D4Qi
-----END PGP SIGNATURE-----

Re: [Dar-libdar_api] libdar::user_interaction::listing splitting the flag argument

[Tobias <spe...@gm...>]

Hi Denis,

The libdar_config.h problem seems not to be solved.
But get_children_in_table is working now, also with other dar archives.
Thank you very much.

Regards,
Tobias

Am Montag, den 28.10.2013, 20:10 +0100 schrieb Denis Corbin:
> Le 28/10/2013 17:12, Tobias wrote:
> > Hi Denis,
> 
> Hello Tobias,
> 
> > 
> > Thank you very much for the back-port!
> > That's the function I was looking for.
> > 
> > I have compiled libdar from git.
> > By the way, it seems like a #endif in the libdar_config.h include file
> > is missing:
> >> #ifndef _DARWIN_USE_64_BIT_INODE
> >> # define _DARWIN_USE_64_BIT_INODE 1
> > + #endif
> 
> Could not reproduce this from the installed data I get here, but I think
> I have found the cause. Normally only DAR related macro are present in
> this file, I've changed the filter for "DARWIN" to not match the
> DAR/LIBDAR macro filtering. Thanks to tell me if you have further
> problem or not.
> 
> > 
> > Unluckily I can list only the root directory ("") of the archive. 
> > When requesting a subdirectory I get an Exception:
> > "subdri/ entry does not exist"
> > Calling get_children_of within the same archive is working as usual.
> > Can you reproduce this behavior?
> 
> yes I could, I had just back-ported the initial patch about this feature
> and forgot that subsequent patches were fixing several bugs about that
> new feature. Now, it works better for me, and I hope for you too.
> 
> > 
> > Can I only use the get_children_in_table method when working with
> > archives that had been created with the current git release?
> 
> No, it should work for any archive even for archives generated by dar
> release 1.0.0 more than 11 years ago :)  Please pay attention to call
> archive::init_catalogue() once after the archive has been built and
> before calling get_children_in_table() the first time on that archive.
> 
> init_catalogue() loads the catalogue in memory when it is not already
> done. The catalogue is not loaded only when reading an archive in
> sequential mode before any action has been asked on that archive, as the
> contents in learned along the archive being read ... sequentially
> 
> But you have not to call init_catalogue() if you want to
> extract/diff/test the archive before exploring the archive contents: In
> sequential read mode, the access to the archive cannot be done twice so
> either you do an operation on the archive (like extracting some files)
> which finally builds up the catalogue in memory once the operation is
> completed, or you call init_catalogue() which read the whole archive in
> order to build the catalogue in memory without doing anything else on
> about file's data or EA (just skipping over it).
> 
> if not in sequential read mode, init_catalogue() does nothing and it
> does not hurt to call it anyway before calling get_children_in_table()
> to be sure the catalogue has been built in memory.
> 
> > When opening other archives and calling get_children_in_table I ran into
> > an Exception: 
> > "it seems to be a bug here"
> > 
> > 
> > Regards,
> > Tobias
> > 
> 
> Regards,
> Denis.
> 
> ------------------------------------------------------------------------------
> Android is increasing in popularity, but the open development platform that
> developers love is also attractive to malware creators. Download this white
> paper to learn more about secure code signing practices that can help keep
> Android apps secure.
> http://pubads.g.doubleclick.net/gampad/clk?id=65839951&iu=/4140/ostg.clktrk
> _______________________________________________
> Dar-libdar_api mailing list
> Dar...@li...
> https://lists.sourceforge.net/lists/listinfo/dar-libdar_api

Re: [Dar-libdar_api] libdar::user_interaction::listing splitting the flag argument

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Le 28/10/2013 17:12, Tobias wrote:
> Hi Denis,

Hello Tobias,

> 
> Thank you very much for the back-port!
> That's the function I was looking for.
> 
> I have compiled libdar from git.
> By the way, it seems like a #endif in the libdar_config.h include file
> is missing:
>> #ifndef _DARWIN_USE_64_BIT_INODE
>> # define _DARWIN_USE_64_BIT_INODE 1
> + #endif

Could not reproduce this from the installed data I get here, but I think
I have found the cause. Normally only DAR related macro are present in
this file, I've changed the filter for "DARWIN" to not match the
DAR/LIBDAR macro filtering. Thanks to tell me if you have further
problem or not.

> 
> Unluckily I can list only the root directory ("") of the archive. 
> When requesting a subdirectory I get an Exception:
> "subdri/ entry does not exist"
> Calling get_children_of within the same archive is working as usual.
> Can you reproduce this behavior?

yes I could, I had just back-ported the initial patch about this feature
and forgot that subsequent patches were fixing several bugs about that
new feature. Now, it works better for me, and I hope for you too.

> 
> Can I only use the get_children_in_table method when working with
> archives that had been created with the current git release?

No, it should work for any archive even for archives generated by dar
release 1.0.0 more than 11 years ago :)  Please pay attention to call
archive::init_catalogue() once after the archive has been built and
before calling get_children_in_table() the first time on that archive.

init_catalogue() loads the catalogue in memory when it is not already
done. The catalogue is not loaded only when reading an archive in
sequential mode before any action has been asked on that archive, as the
contents in learned along the archive being read ... sequentially

But you have not to call init_catalogue() if you want to
extract/diff/test the archive before exploring the archive contents: In
sequential read mode, the access to the archive cannot be done twice so
either you do an operation on the archive (like extracting some files)
which finally builds up the catalogue in memory once the operation is
completed, or you call init_catalogue() which read the whole archive in
order to build the catalogue in memory without doing anything else on
about file's data or EA (just skipping over it).

if not in sequential read mode, init_catalogue() does nothing and it
does not hurt to call it anyway before calling get_children_in_table()
to be sure the catalogue has been built in memory.

> When opening other archives and calling get_children_in_table I ran into
> an Exception: 
> "it seems to be a bug here"
> 
> 
> Regards,
> Tobias
> 

Regards,
Denis.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIVAwUBUm62pAgxsL0D2LGCAQIcwBAAtMtqCj+/MN54QRMq4mVbCP67Il/s8Wjf
8egptIb/a4qGiXUyAd+VAfafkIHGtXThfgIeJwcjIoTRtfMtFs8BiN7TfCpsothC
MsyKfXxL689+bhTVgTNf+04D5Tl6TK7rjWFWfuRRcyaeOlVQ8KYd0km9CGAA8i0K
hf4zwfpZS4jGI6TAir3yeyQ1lzjgnP9qSQd4Dom0Rjav7Lo03LOBzkUTXhyqowfi
mLNA8/q+sWccTV3FcPlOsQIc7vL09mvR9ZgH4oZ+7nTPGLhloJ3gNH8JMjuwDxvB
EJEl6OOg40LVmGbxwm82ZjnLN+U+paZDgJC/D2zNyuQATsgpeKuBwGKqrrG8fQ0y
MCrJE1NrwhK2uM/2N6V4pnTfBkN8nmxVepnwwpTBRRajXhBLzDw3bISnUpDpfy42
CslQRjSNJ/Rs/whFS/sfXDzmhqPKqP7xnY9R6tXuzN3ZJ/1pYomGeY7P4pqBojSc
kyo/VyG7Vgaa/XEma7uaKqlQKpEB05WAsiFBpnWPsf/bHl1StbTJT7hz6bbTYtR6
auxGmdgoyde+GL9Ubrv/8bk/PE3aLuJ4RP2yrMaiI1iwUleqxM+ks8HsT4kMtvUt
anHst6+TyS4rIgHhIa3mlrsv9XdHwBAH9Zww1DPzl22RglPIpN6kf9lGidHAFHYh
h3WYWQJh3sM=
=snl+
-----END PGP SIGNATURE-----

Re: [Dar-libdar_api] libdar::user_interaction::listing splitting the flag argument

[Tobias <spe...@gm...>]

Hi Denis,

Thank you very much for the back-port!
That's the function I was looking for.

I have compiled libdar from git.
By the way, it seems like a #endif in the libdar_config.h include file
is missing:
> #ifndef _DARWIN_USE_64_BIT_INODE
> # define _DARWIN_USE_64_BIT_INODE 1
+ #endif

Unluckily I can list only the root directory ("") of the archive. 
When requesting a subdirectory I get an Exception:
"subdri/ entry does not exist"
Calling get_children_of within the same archive is working as usual.
Can you reproduce this behavior?

Can I only use the get_children_in_table method when working with
archives that had been created with the current git release?
When opening other archives and calling get_children_in_table I ran into
an Exception: 
"it seems to be a bug here"


Regards,
Tobias

Am Freitag, den 25.10.2013, 21:16 +0200 schrieb Denis Corbin:
> On 24/10/2013 20:55, Tobias wrote:
> > Hi Denis,
> 
> Hello Tobias,
> 
> 
> >[...]
> >  
> > 
> > I have another question.
> > 
> > I'm overwrite the listing method to get the output of get_children_of.
> > 
> > Is it possible to break up the "flag" argument into its components?
> 
> Well, I see that there is still room for improvement in this area since
> the libdar/dar split at release 2.0.0.
> 
> In the current development libdar (GIT master) I've added the method
> 
>    archive::get_children_in_table(const std::string & dir) const
> 
> which returns a std::vector of objects of type libdar::list_entry.
> The class list_entry provides all the necessary methods to know about
> the entry of a given directory. This an much more simple alternative to
> get_children_of() method that relies on user_interaction classes.
> 
> references:
> 	src/libdar/archive.hpp 	
> 	src/libdar/list_entry.hpp
> 
> As I cannot tell you when will take place the future major release of
> dar/libdar (release 2.5.0) because there is still important feature to
> add, I have back-ported this feature to the current stable code (GIT
> branch_2.4.x) which will be released with next minor version of
> dar/libdar (release 2.4.12) probably next January unless important bug
> are found in dar/libdar implying a more sooner fix/release.
> 
> You can use this code from GIT branch_2.4.x. If you need to a tarball
> tell me I will send it to you, no problem. But if you prefer you can
> also read the USING_SOURCE_FROM_GIT at the root of the GIT tree on
> branch_2.4.x.
> 
> > 
> > Particularly I'm interested in the "saved" attribute, it is a localised
> > string
> 
> the
> 	list_entry::has_data_present_in_the_archive()
> 	list_entry::is_dirty()
> 
> methods should answer your need.
> 
> > 
> > and thus problematic to work with.
> > 
> > Is there another method implementing this feature?
> > 
> 
> Now, Yes! :)
> 
> >  
> > 
> > Regards,
> > 
> > Tobias
> > 
> >  
> 
> Regards,
> Denis.
> 
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk
> _______________________________________________
> Dar-libdar_api mailing list
> Dar...@li...
> https://lists.sourceforge.net/lists/listinfo/dar-libdar_api

Re: [Dar-libdar_api] libdar::user_interaction::listing splitting the flag argument

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 24/10/2013 20:55, Tobias wrote:
> Hi Denis,

Hello Tobias,


>[...]
>  
> 
> I have another question.
> 
> I'm overwrite the listing method to get the output of get_children_of.
> 
> Is it possible to break up the "flag" argument into its components?

Well, I see that there is still room for improvement in this area since
the libdar/dar split at release 2.0.0.

In the current development libdar (GIT master) I've added the method

   archive::get_children_in_table(const std::string & dir) const

which returns a std::vector of objects of type libdar::list_entry.
The class list_entry provides all the necessary methods to know about
the entry of a given directory. This an much more simple alternative to
get_children_of() method that relies on user_interaction classes.

references:
	src/libdar/archive.hpp 	
	src/libdar/list_entry.hpp

As I cannot tell you when will take place the future major release of
dar/libdar (release 2.5.0) because there is still important feature to
add, I have back-ported this feature to the current stable code (GIT
branch_2.4.x) which will be released with next minor version of
dar/libdar (release 2.4.12) probably next January unless important bug
are found in dar/libdar implying a more sooner fix/release.

You can use this code from GIT branch_2.4.x. If you need to a tarball
tell me I will send it to you, no problem. But if you prefer you can
also read the USING_SOURCE_FROM_GIT at the root of the GIT tree on
branch_2.4.x.

> 
> Particularly I'm interested in the "saved" attribute, it is a localised
> string

the
	list_entry::has_data_present_in_the_archive()
	list_entry::is_dirty()

methods should answer your need.

> 
> and thus problematic to work with.
> 
> Is there another method implementing this feature?
> 

Now, Yes! :)

>  
> 
> Regards,
> 
> Tobias
> 
>  

Regards,
Denis.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIVAwUBUmrDeQgxsL0D2LGCAQK2xg/9EqVJS5X6INWybeJLuStFjIPhReeo3xRw
TU+Rb1gtqDdR94xEiYahBeHLPIUBTdcHxO2I4EiZ5AHK3KhQglTVQrHnSgkJA+qQ
2HRBIp29O3EIw8zWKKAFkjGP/hbG+kXiz6Vr2vXTSNNHWz8W9e3QVBk2PyY3dzg7
cv1VO35Pl/ivcSYGTUcMmoYrBZqxlQF2N7FcMvTmoI6o2ymxUVri+BvQsN6pqb3k
6PlN0Pm1p0ZChu8ClUWLQdXadS4FynM+CSbGcl4Z2GEN4xweP3ZsedAndMeJ4i+n
tg/I6OboYF+e3WWsml+1KWfvKCGrbP2pvI4TEceYJaNyd/ba9ulUmEDhVqiNTAv7
BbKfIn0zYcoN4afgvsTQYoZXThbipOXEMrB32arfDx02IpHY4WCp46YzC/UUQ6P/
jNQqM4crtzEvKv/HomNnD1PwScM/KbzxpUEHko89XVCpJXcpW2EJaR0UzYXyeDbj
DkAy1+xloK9fzXzLNkB0aDQLawWVVJ/THQ2bmGz58f3qPR53r6P1mqwdIyyavqDA
cbYx8gEN3m0sbN2Na/9bcSEAXJIyh5Ogt0W6FobqJu9AkMwWV3RQM7mi/vIwTVDy
OWgnCeS2zmsXrlRxQm1zNADiu98/NO9Co8OfvyBzNwxL3OgWDV7MoGkqQEiZcafQ
w6spvv3xNjA=
=ldub
-----END PGP SIGNATURE-----

[Dar-libdar_api] libdar::user_interaction::listing splitting the flag argument

[Tobias <spe...@gm...>]

Hi Denis,

 

thank you really much for your time regarding "open archive per level"!

I have now a better knowledge of the libdar internals, that helps a lot.

 

I have another question.

I'm overwrite the listing method to get the output of get_children_of.

Is it possible to break up the "flag" argument into its components?

Particularly I'm interested in the "saved" attribute, it is a localised
string

and thus problematic to work with.

Is there another method implementing this feature?

 

Regards,

Tobias

 

Re: [Dar-libdar_api] open archive per level

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 22/10/2013 21:26, Tobias wrote:
> Hi,

Hello Tobias,

> 
>> Last point is implementation constraint: The way to drop catalogue
>> contents to archive or read it from archive is done by a recursive
>> method (dump() for dropping, and a specific constructor for reading)
>> common to all catalogue object types (directories, files, etc.). The
>> directory constructor should act differently having the knowledge of a
>> target path to read, whether:
>> a) - we are a directory containing the path to the target (only retain
>> sub entry of the target, temporarily create others and destroy them
>> afterward
>> b) - we are the last directory of the target path (act as today, read
>> all entries)
>> c) - we are the subdirectory of the target path (create temporary
>> objects up to its EOD)
>> d) - we are a directory "after" the last component of the target path
>> has been completely read (in which case we can stop the catalogue reading)
> 
> That's exactly the point.
> What I thought about was:
> When we are at case a) or c) we need only to know:
> 1) is it a file -> skip

yes, but in order to skip over you must known how far to skip forward
... this is only possible by reading an analyzing the following entries
in the catalogue dump... that's to say quite the same CPU processing as
building the corresponding object, but without memory requirement, I agree.

> 2) is it a directory -> increase "depth counter" -> parse directory
> And this is the point where we could save CPU cycles and memory, because
> we don't need to extract and store all information about the inode.

You don't have to allocate memory, right, but you have to parse the
catalogue dump up to the time you read the balanced End Of Directory
mark along the dump, meaning you've reach the end of that directory.

> But I don't know exactly how the "read" constructor reads the catalogue
> and what the structure built up in memory looks like.

What is important here is to understand the way the catalogue object
dumps its data into the archive. Each inode object it contains
(directory, file, pipe, symlink, hardlink, etc.) dumps in turn one after
the other the data that will be necessary to rebuilt the catalogue from
the adhoc constructor in the future. Unfortunately, there is no
Type-Length-Value information, so you cannot simply skip over a given
entry (knowing its length) you have to "reading" it in depth: filename
length vary, date, UID, GID are integers stored in a specific format
named "infinint" that can be arbitrarily long and does not suffer any
limitations, some other fields are present only under certain
circumstances, etc.

A TLV-like structure was not necessary. Moreover it would have brought
some sever limitations because is would have implies fixed length for
the Type and Length fields, which might be a huge constraint when saving
an arbitrarily large directory, for example.

> 
> What is the most time consuming part when open the archive?
> a) Is it just the reading and decompressing of the catalogue stored in the archive?

yes, it is.

> b) The interpreting of the data (what kind of inode, name, permissions, ...)?

no, this is negligible

> c) Or to build up the catalogue in memory?

this much less important than decompressing, but comes in second point
after it.

> We could only influence c) and maybe b).

Right. For c) there is already optimization layer that asks to the
system larger blocks of memory and split them for internal memory
allocation/reallocation. You can try disabling it passing
- --disable-special-alloc to ./configure and recompiling dar/libdar (make
clean / make) to see the performance impact.

There is also a caching layer in dar to avoid asking many small portion
of data from the filesystem, and ask a larger file portion each time.
This reduces the context switch and allow dar to work efficiently over a
the network where latency is important (using ssh for example).

> This depends on the format of the inode.
> Can we only read a part of the attributes?

Only if it is the last thing we expect to read from the catalogue dump
in the archive.

> 
> 
>> It will make the implementation more complicated and more subject to
>> bugs, this is a certainty. Will it boost the process? Probably but at
>> different level of gain depending on the "target path" location and
>> extension inside the archive, and assuming this is one shot operation on
>> that particular archive (seems not realistic for a GUI, more the case of
>> a script or CLI software).
> 
> Yes, this is complicated and maybe the performance boost is so small, 
> that we will loos it by opening a subdirectory.
>

I agree,

By the way, if you have any questions or need clarifications using
libdar don't hesitate to ask on this mailing-list.

> 
> 
> Regards,
> Tobias
> 

Regards,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIVAwUBUmlLiggxsL0D2LGCAQLBDhAAxX5DMtBD+hDKIkGgKfhDcvgCqEnzn6W1
xYgLrll/vG5y4ZkUDFnilFV4Xc93GWgdMmbf6V2zeX+lmDfPQhNA6bJdXmfpNp54
6nKK3pTWifD2rK+h6fsqjk9HIFGKOEjTLMNeohiYBWb7d5xRd6pgxkLlZgPMpAxh
VLyyWoWsCnCVsPFtLpjqbOcOZ36YrzNxL0W4nULIqQs9E8SgjNiiYw8AdqTNh51j
cOqeLcYjW6xgRiKcZrIFaomiKDyPQOTdgcj5KKdXQhG/tflMWsQ2P7ccNGXNR+to
yEtdhPB4vBSD78yYlwQlF8LOY4UKQWGC3TJoOPNuk5KKJe41AcfgMV+bfBsTCuba
QymzCqqynxMJaZqbiEl5QhIemKuYIFjfBAF43PWg19VTSuizZZDs9PHhOZTGvHhw
HXhb1kJC1BE4p4rw7wrp4gNcUuabuLtTPvitxivrDyebpTt8uPrc/qbAscKRyhHr
Nvxh2D60XomPbv6hnNmiwjOZJMy2L1IHHRR857WS+eQ2vDeqwmDtV40tG3xSSIL+
k76KXsIHXHNccpOX1x3Yxbv+t/iw6f47kmUUb47xDCn0Job1kYALQ28WGRuNLySE
8tqapU5UypctWSkIBFBOlOwjPb1vbOXrka1q7nHNIqp4HfcUzgOEi3z/ljnfFLlQ
DgJ7JozohMM=
=YXkK
-----END PGP SIGNATURE-----

Re: [Dar-libdar_api] open archive per level

[Tobias <spe...@gm...>]

Hi,

> Last point is implementation constraint: The way to drop catalogue
> contents to archive or read it from archive is done by a recursive
> method (dump() for dropping, and a specific constructor for reading)
> common to all catalogue object types (directories, files, etc.). The
> directory constructor should act differently having the knowledge of a
> target path to read, whether:
> a) - we are a directory containing the path to the target (only retain
> sub entry of the target, temporarily create others and destroy them
> afterward
> b) - we are the last directory of the target path (act as today, read
> all entries)
> c) - we are the subdirectory of the target path (create temporary
> objects up to its EOD)
> d) - we are a directory "after" the last component of the target path
> has been completely read (in which case we can stop the catalogue reading)

That's exactly the point.
What I thought about was:
When we are at case a) or c) we need only to know:
1) is it a file -> skip
2) is it a directory -> increase "depth counter" -> parse directory
And this is the point where we could save CPU cycles and memory, because
we don't need to extract and store all information about the inode.
But I don't know exactly how the "read" constructor reads the catalogue
and what the structure built up in memory looks like.

What is the most time consuming part when open the archive?
a) Is it just the reading and decompressing of the catalogue stored in the archive?
b) The interpreting of the data (what kind of inode, name, permissions, ...)?
c) Or to build up the catalogue in memory?
We could only influence c) and maybe b).
This depends on the format of the inode.
Can we only read a part of the attributes?


> It will make the implementation more complicated and more subject to
> bugs, this is a certainty. Will it boost the process? Probably but at
> different level of gain depending on the "target path" location and
> extension inside the archive, and assuming this is one shot operation on
> that particular archive (seems not realistic for a GUI, more the case of
> a script or CLI software).

Yes, this is complicated and maybe the performance boost is so small, 
that we will loos it by opening a subdirectory.


Regards,
Tobias

Am Montag, den 21.10.2013, 09:30 +0200 schrieb Denis Corbin:
> Le 20/10/2013 23:24, Tobias wrote:
> > I understand the problem.
> > 
> > 1: I think memory shouldn't be the problem, so we could store the
> > uncompressed catalogue in memory.
> 
> Actually, the decompression is done on-fly while reading the catalogue
> and creating corresponding objects on-fly (directory objects, files
> objects, etc.), so no memory is used to store "uncompressed" copy of the
> catalogue dump.
> 
> here it would require copying the portion of the archive where resides
> the catalogue dump into memory and then only build some of the
> catalogue's objects from it.
> 
> > 
> > 2: This example is from your documentation of the Dar 5 archive format:
> > - toto
> >    | titi
> >    | tutu
> >    | tata
> >    |  | blup
> >    |  +---
> >    | boum
> >    | coucou
> >    +---
> > 
> > +-------+------+------+------+------+-----+------+--------+-----+
> > | toto  | titi | tutu | tata | blup | EOD | boum | coucou | EOD |
> > |       |      |      |      |      |     |      |        |     |
> > +-------+------+------+------+------+-----+------+--------+-----+
> > 
> > When parsing the filesystem tree, would it be possible to skip the
> > subdirectories when building up the catalogue in memory?
> > Based on the example. You want to list the content of toto. 
> > So you go toto, titi, tutu, tata. Now you are leaving the desired level
> > and your going to search for the next EOD to come back to the toto
> > level. So your only reading blup but don't add it to the struct in
> > memory.
> 
> In the catalogue there is no "marks" or TLV-like structure that can be
> used to skip over a given entry. So to know what is the next entry after
> toto you have read tata entry completely, you need to understand all the
> next entries up to the directory balanced EOD entry that means you have
> reached the end of tata contents and are back into toto contents.
> 
> Worse, the length of a particular object is strongly dependent on its
> contents. The objective was to provide flexibility and allow new fields
> and new objects types (classes) to answer future needs. This is one of
> the strength of the current design which allowed 8 new evolutions of
> archive format without breaking backward compatibility.
> 
> The consequence is that to know when you are back to "toto" level you
> need to build temporary objects up to the EOD meaning the end of
> directory "tata". This work of object creation costs the same as today,
> having to destroy them just afterward brings extra work, thus more time
> to reach that directory (toto) contents than today.
> 
> This is only when the EOD corresponding to toto would be met that you
> could stop parsing the rest of the catalogue. Here in the example this
> means the end of the catalogue! OK, this example is particular, assuming
> we would like to read the content of tata we would could stop reading
> the archive before reading "boum". You see that even in that opposite
> case we already have read more than the half of the catalogue...
> 
> Last point is implementation constraint: The way to drop catalogue
> contents to archive or read it from archive is done by a recursive
> method (dump() for dropping, and a specific constructor for reading)
> common to all catalogue object types (directories, files, etc.). The
> directory constructor should act differently having the knowledge of a
> target path to read, whether:
> a) - we are a directory containing the path to the target (only retain
> sub entry of the target, temporarily create others and destroy them
> afterward
> b) - we are the last directory of the target path (act as today, read
> all entries)
> c) - we are the subdirectory of the target path (create temporary
> objects up to its EOD)
> d) - we are a directory "after" the last component of the target path
> has been completely read (in which case we can stop the catalogue reading)
> 
> Assuming all this worse the effort, from the GUI you will get (let's
> hope faster than today) the content of a given directory. What if the
> user asks to expand another subdirectory? Woudn't we have to redo all
> the process with a more complicated situation where some objects (root
> directory for example) already exist and must not be duplicated?
> What about the compressed archive in memory, would it have to stay in
> memory up to the time the whole archive would be read (and how to know
> it would be completely read?), or have it to be uncompressed each time?
> You'll probably prefer the first solution because you are focused on CPU
> time. But one of the drawback of dar is that it may be fond of memory
> when the archive contains a lot of small files, here we'll probably not
> doubled the memory requirement but will not help that many users that
> find it annoying on embedded systems for example...
> 
> 
> 
> > Of course this is a really simple example. In reale world you have some
> > more subdirs and determining the point where your back on the desired
> > level will not that easy. 
> > But do you understand what I mean?
> 
> I guess so, yes.
> 
> > Shouldn't this boost up the process, because you don't have to add
> > hundreds of (in this moment uninteresting) inodes to the struct in
> > memory?
> 
> It will make the implementation more complicated and more subject to
> bugs, this is a certainty. Will it boost the process? Probably but at
> different level of gain depending on the "target path" location and
> extension inside the archive, and assuming this is one shot operation on
> that particular archive (seems not realistic for a GUI, more the case of
> a script or CLI software).
> 
> > 
> > Ok, I admit when you want to build up the whole tree, this technique
> > costs more CPU cycles. But then I only want to extract one directory
> > from the archive I don't wont to wate over half a minute until the
> > catalogue is read.
> 
> I understand the need. I hope you understand the implications and
> complexity doing so, waiting for your objections :-)
> 
> > 
> > Tobias
> > 
> 
> Regards,
> Denis.
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
> _______________________________________________
> Dar-libdar_api mailing list
> Dar...@li...
> https://lists.sourceforge.net/lists/listinfo/dar-libdar_api

Re: [Dar-libdar_api] open archive per level

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Le 20/10/2013 23:24, Tobias wrote:
> I understand the problem.
> 
> 1: I think memory shouldn't be the problem, so we could store the
> uncompressed catalogue in memory.

Actually, the decompression is done on-fly while reading the catalogue
and creating corresponding objects on-fly (directory objects, files
objects, etc.), so no memory is used to store "uncompressed" copy of the
catalogue dump.

here it would require copying the portion of the archive where resides
the catalogue dump into memory and then only build some of the
catalogue's objects from it.

> 
> 2: This example is from your documentation of the Dar 5 archive format:
> - toto
>    | titi
>    | tutu
>    | tata
>    |  | blup
>    |  +---
>    | boum
>    | coucou
>    +---
> 
> +-------+------+------+------+------+-----+------+--------+-----+
> | toto  | titi | tutu | tata | blup | EOD | boum | coucou | EOD |
> |       |      |      |      |      |     |      |        |     |
> +-------+------+------+------+------+-----+------+--------+-----+
> 
> When parsing the filesystem tree, would it be possible to skip the
> subdirectories when building up the catalogue in memory?
> Based on the example. You want to list the content of toto. 
> So you go toto, titi, tutu, tata. Now you are leaving the desired level
> and your going to search for the next EOD to come back to the toto
> level. So your only reading blup but don't add it to the struct in
> memory.

In the catalogue there is no "marks" or TLV-like structure that can be
used to skip over a given entry. So to know what is the next entry after
toto you have read tata entry completely, you need to understand all the
next entries up to the directory balanced EOD entry that means you have
reached the end of tata contents and are back into toto contents.

Worse, the length of a particular object is strongly dependent on its
contents. The objective was to provide flexibility and allow new fields
and new objects types (classes) to answer future needs. This is one of
the strength of the current design which allowed 8 new evolutions of
archive format without breaking backward compatibility.

The consequence is that to know when you are back to "toto" level you
need to build temporary objects up to the EOD meaning the end of
directory "tata". This work of object creation costs the same as today,
having to destroy them just afterward brings extra work, thus more time
to reach that directory (toto) contents than today.

This is only when the EOD corresponding to toto would be met that you
could stop parsing the rest of the catalogue. Here in the example this
means the end of the catalogue! OK, this example is particular, assuming
we would like to read the content of tata we would could stop reading
the archive before reading "boum". You see that even in that opposite
case we already have read more than the half of the catalogue...

Last point is implementation constraint: The way to drop catalogue
contents to archive or read it from archive is done by a recursive
method (dump() for dropping, and a specific constructor for reading)
common to all catalogue object types (directories, files, etc.). The
directory constructor should act differently having the knowledge of a
target path to read, whether:
a) - we are a directory containing the path to the target (only retain
sub entry of the target, temporarily create others and destroy them
afterward
b) - we are the last directory of the target path (act as today, read
all entries)
c) - we are the subdirectory of the target path (create temporary
objects up to its EOD)
d) - we are a directory "after" the last component of the target path
has been completely read (in which case we can stop the catalogue reading)

Assuming all this worse the effort, from the GUI you will get (let's
hope faster than today) the content of a given directory. What if the
user asks to expand another subdirectory? Woudn't we have to redo all
the process with a more complicated situation where some objects (root
directory for example) already exist and must not be duplicated?
What about the compressed archive in memory, would it have to stay in
memory up to the time the whole archive would be read (and how to know
it would be completely read?), or have it to be uncompressed each time?
You'll probably prefer the first solution because you are focused on CPU
time. But one of the drawback of dar is that it may be fond of memory
when the archive contains a lot of small files, here we'll probably not
doubled the memory requirement but will not help that many users that
find it annoying on embedded systems for example...



> Of course this is a really simple example. In reale world you have some
> more subdirs and determining the point where your back on the desired
> level will not that easy. 
> But do you understand what I mean?

I guess so, yes.

> Shouldn't this boost up the process, because you don't have to add
> hundreds of (in this moment uninteresting) inodes to the struct in
> memory?

It will make the implementation more complicated and more subject to
bugs, this is a certainty. Will it boost the process? Probably but at
different level of gain depending on the "target path" location and
extension inside the archive, and assuming this is one shot operation on
that particular archive (seems not realistic for a GUI, more the case of
a script or CLI software).

> 
> Ok, I admit when you want to build up the whole tree, this technique
> costs more CPU cycles. But then I only want to extract one directory
> from the archive I don't wont to wate over half a minute until the
> catalogue is read.

I understand the need. I hope you understand the implications and
complexity doing so, waiting for your objections :-)

> 
> Tobias
> 

Regards,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIVAwUBUmTYAAgxsL0D2LGCAQI4jQ/+Kjb2JEwrNk3kcWbHV7O7M9HmJ3S5XTEz
JJs364PZzudY97xXA+WeSF8J8sRpbQiF5wfbRAguynggjJEyOwMks8p93HfHrCX5
Pz4edoD0xxKG+40/un1kYqUtnGL5ShVRg+6Ao+ilQwXpQN8HlNkYlMdC20uzNoUL
uByLQjfl14h7KTvhmh+RsbRg4xn0npSnBY8X6H1ow+ZoycWwFKy0DDuN3DUmnsI3
hWkBXx3tTfhfFhcgS1c/90JrMBBubZ3n/QsTvJgSmxLvmuE7nd2LnT2KQz3XeroH
cccCFHOVY9FOn3bNbshZPOJDPmLvxD9HyiHBiOl+a4DfpmR/cNZRsH8od1dZjhie
U/YPy5HRk24lJLTGj3iB2VUMgi4+oEUDLYhQGGUnkKq4kKrpg1taZND2MabZi78n
H5604asOKRtsroEDAzMXPFRqeJ+zio0T7tFORGtu7J5ix0hEmtcffl27OUGWccVH
BgcP8ZUky/Uq+AMmU+Cxa0NuBb6R8L07L0nKarMMf8Cqe0Zp5F9QTwP00gX4CrjC
U64XGPR+nVRiHBYW8boH+84ZXFnedJ9yNkbB1Yx3nCGcggjOl1b+nhb33SW+UZd+
tYwa8IQxVS/IwK0exOYsQe+AaLHtiDlEBhh/+K/NBybvl1vjI/yuDCBBcKh3Y4NQ
NaPK3ixsYxs=
=SpXC
-----END PGP SIGNATURE-----

Re: [Dar-libdar_api] open archive per level

[Tobias <spe...@gm...>]

I understand the problem.

1: I think memory shouldn't be the problem, so we could store the
uncompressed catalogue in memory.

2: This example is from your documentation of the Dar 5 archive format:
- toto
   | titi
   | tutu
   | tata
   |  | blup
   |  +---
   | boum
   | coucou
   +---

+-------+------+------+------+------+-----+------+--------+-----+
| toto  | titi | tutu | tata | blup | EOD | boum | coucou | EOD |
|       |      |      |      |      |     |      |        |     |
+-------+------+------+------+------+-----+------+--------+-----+

When parsing the filesystem tree, would it be possible to skip the
subdirectories when building up the catalogue in memory?
Based on the example. You want to list the content of toto. 
So you go toto, titi, tutu, tata. Now you are leaving the desired level
and your going to search for the next EOD to come back to the toto
level. So your only reading blup but don't add it to the struct in
memory.
Of course this is a really simple example. In reale world you have some
more subdirs and determining the point where your back on the desired
level will not that easy. 
But do you understand what I mean?
Shouldn't this boost up the process, because you don't have to add
hundreds of (in this moment uninteresting) inodes to the struct in
memory?

Ok, I admit when you want to build up the whole tree, this technique
costs more CPU cycles. But then I only want to extract one directory
from the archive I don't wont to wate over half a minute until the
catalogue is read.

Tobias

Am Sonntag, den 20.10.2013, 19:15 +0200 schrieb Denis Corbin:
> On 20/10/2013 15:06, Tobias wrote:
> > Hi,
> 
> Hello,
> 
> > 
> > i'm programming a graphical user interface for restoring Dar backups.
> 
> nice to hear about that :)
> 
> > 
> > To list the archive content I'm using the dynamic archive content
> > listing option like described in chapter 7 of your API tutorial.
> > Thank you for that, it's working really well. 
> > But it seams like the "read" constructor reads the whole catalogue.
> 
> Yes that's correct
> 
> > Is it possible to change that behavior in a way
> > that the catalogue gets read per level?
> 
> it would be difficult to be implemented that way, because:
> 1 - the catalogue is compressed as a whole (thus this per level read of
> a catalogue would imply uncompressing the same data several time to read
> further catalogue entries
> 2 - even without compression, to find the contents of a given directory
> would require a lot of work: the directory tree is stored using a
> "in-depth" tree order. In other words, at creation time when comes the
> time to write down the catalogue, when a subdirectory is found its whole
> contents is written down to the archive before proceeding to the next
> file or next subdirectory found at that level.
> 
> So its more efficient to build in memory the whole catalogue from the
> archive and then to extract by level the requested information from it.
> 
> > That would really boost the performance.
> 
> I don't think so. Even, if you just want to read the contents of one
> directory it would takes almost the same amount of time and will reduce
> memory requirement. But for the next directory reading you would have to
> do the same operation another time ... so globally I think this is
> faster the way it is actually implemented (with all the CPU cycle being
> done before being able to read any archive content).
> 
> > 
> > Kind regards
> > Tobias
> > 
> > 
> 
> Kind Regards,
> Denis.
> ------------------------------------------------------------------------------
> October Webinars: Code for Performance
> Free Intel webinars can help you accelerate application performance.
> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from 
> the latest Intel processors and coprocessors. See abstracts and register >
> http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
> _______________________________________________
> Dar-libdar_api mailing list
> Dar...@li...
> https://lists.sourceforge.net/lists/listinfo/dar-libdar_api

Re: [Dar-libdar_api] open archive per level

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 20/10/2013 15:06, Tobias wrote:
> Hi,

Hello,

> 
> i'm programming a graphical user interface for restoring Dar backups.

nice to hear about that :)

> 
> To list the archive content I'm using the dynamic archive content
> listing option like described in chapter 7 of your API tutorial.
> Thank you for that, it's working really well. 
> But it seams like the "read" constructor reads the whole catalogue.

Yes that's correct

> Is it possible to change that behavior in a way
> that the catalogue gets read per level?

it would be difficult to be implemented that way, because:
1 - the catalogue is compressed as a whole (thus this per level read of
a catalogue would imply uncompressing the same data several time to read
further catalogue entries
2 - even without compression, to find the contents of a given directory
would require a lot of work: the directory tree is stored using a
"in-depth" tree order. In other words, at creation time when comes the
time to write down the catalogue, when a subdirectory is found its whole
contents is written down to the archive before proceeding to the next
file or next subdirectory found at that level.

So its more efficient to build in memory the whole catalogue from the
archive and then to extract by level the requested information from it.

> That would really boost the performance.

I don't think so. Even, if you just want to read the contents of one
directory it would takes almost the same amount of time and will reduce
memory requirement. But for the next directory reading you would have to
do the same operation another time ... so globally I think this is
faster the way it is actually implemented (with all the CPU cycle being
done before being able to read any archive content).

> 
> Kind regards
> Tobias
> 
> 

Kind Regards,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIVAwUBUmQPrggxsL0D2LGCAQKXig//ZpOtlcvX6Dh76Z+xRYHf/QYeH6SpP1Cq
viWSh3pC+5ONYtGUWJRT16ze6NWj7P8kmmRX3dSj68C/BYWJ0YdX9jYqgneLSzIn
JUb8/H6kuwZ1htE/BDIFVIBXl4Ye7VjJP6T8J2Xf/qx4Owkug3Aby43QI2/fr7DL
Mxog5PgpeJpNAK84frPvSsAMciEhf+dDnBcMLmCgDkqUDGqsSsay4E/E/51cteM6
2NbN5FHep4gAOY60uEpVqYLLkIbJN9pPgUnyptKDJ5XC1ApZo51NB1UBJe5lt5K5
U58u43vzy/ZF12IwqXOfea0fAl5k7XZ7Pu3WnIg+OOamtQA995rHLx3CKUvd2qhk
NcJdjiChnu90HpouYqgmhcsCSSVFeiTokO1I3mdm+qk2J2sW7kUxGZrCTPpT3AMR
YtU4p2I0KXIL4v8ljdvggC+97pt3NdmF7gQTpmPA19TMZfzx6iHJWz1UXaym/cQw
XzFo63x+xxwBzE95+vdzAoIRtCajIVCFzA3vCoaii9JCqPLS2CsOlJYzWaJ2D+Cd
u9Z56oU7eQ7Ny4MjlQoDeMWYSKUpwrBuvbpLWGQU/nZwG5UpocijGizGvNMGxerI
zXtRDqP/PM5NflPF6hk7G/WrZQ3xWkqjKUM5RXt2X+JldeTQ9nVKt4t+7ZtGxPC4
MsSNNvc/q38=
=PWgR
-----END PGP SIGNATURE-----

[Dar-libdar_api] open archive per level

[Tobias <spe...@gm...>]

Hi,

i'm programming a graphical user interface for restoring Dar backups.

To list the archive content I'm using the dynamic archive content
listing option like described in chapter 7 of your API tutorial.
Thank you for that, it's working really well. 
But it seams like the "read" constructor reads the whole catalogue.
Is it possible to change that behavior in a way
that the catalogue gets read per level?
That would really boost the performance.

Kind regards
Tobias

[Dar-libdar_api] frozen API

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi All,

Just to let you know that current development code has been frozen for
its API specification. Still remains some documentation to review before
starting the pre-release phase, but all API related documentation are
updated (I still have to review them to fix probable bad spelling and
bad English phrases...)

Some words about the new API.

First, yes, it is a new API. But the old API (4.4.x version) is still
available through a compatibility wrapper included in libdar. So if you
do not want to take advantage of libdar's new features in your program
but want to be able to compile and link against the new API with a
minimum effort, you will have to include "libdar_4_4.hpp" in place of
"libdar.hpp" and use the namespace "libdar_4_4" in place of "libdar".
That's all. Compilation and linking stay the same.

Why yet a new API?

Mainly because for each new feature a new argument to a method somewhere
had to be added, which broke backward compatibility (see API version
4.4.x compared to major release version 2.4.x).

What are the changes between 4.4.x and 5.0.0 API?

Well, first, there is no more any method with a ton of arguments. This
should let code more readable and avoid argument mix. So now, how this
info get passed to libdar? Thanks to a new set of "option" classes.

Each of these "option" classes has a constructor with no argument, which
set the different options to their default values. Beside the
constructor, for each former argument (aka for each feature) can be
found two methods the first is "get_...()" the second is "set_...()".
API user will mainly use the set_...() methods while libdar will read
them thanks to the get_...() ones.

Then what's the advantage of theses classes?

Each new feature will bring a new couple of method for the concerned
option class, which will be set to its default value by the constructor.
This way, adding new feature should no more break backward compatibility
and neither oblige any user program to add argument somewhere even if
the corresponding feature is not used by it.

Where get more detailed information?

Checkout the CVS trunk, and point your browser toward the doc/index.html
page. In addition, if you have Doxygen, you can build the API
documentation or else read the same information directly from libdar
header files (src/libdar/archive_options.hpp; src/libdar/archive.hpp)
Last, you can read src/dar_suite/dar.cpp as a full illustration on how
to use libdar's new API.

Any question/suggestions/remarks about the API and its documentation are
of course welcome in this mailing-list.

Kind Regards,
Denis.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFMrhmXpC5CI8gYGlIRAhagAKCuBP514uI2YTx7RjDqz6t+YXtOMgCeKoXu
bwlCzwXI7onWJfC4UGLevN0=
=BS8/
-----END PGP SIGNATURE-----

Re: [Dar-libdar_api] low-level reading of file inside an archive?

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

Guido De Rosa a écrit :
> Hi all,
> 
> is there a way to do something like this:
> 
> read n bytes at offset t of file path/to/my/file.ext inside a DAR
> archive and put the result into a std::string or char* or something?
> 
> I would like to implement a read-only FUSE file system to look inside
> DAR archives, and I need to perform this kind of basic I/O operations
> without extracting real files to temporary  directories (and then make
> the I/O on the file) because this would be highly inefficient.

Yep, I understand the need.

In current development code (which should start its pre-release phase
this year I hope), you have such a possibility:

When you open an archive using the libdar API, you create an "archive"
object, which provide a get_catalogue() method.

The object returned by this archive::get_catalogue() method has several
methods to navigate in the directory tree:
  catalogue::reset_read()
  catalogue::read()
  catalogue::skip_read_to_parent_dir()

and also
  catalogue::direct_read() but which is much less efficient.

the catalogue::read() method will provide you an object of class
"entree" which is the root parent of several classes, among which the
class "file" which corresponds to plain files, you will have to use
dynamic_cast to know whether the returned object is a plain file or not
and to be able to use its specific methods.

The methods you will be able to use are in particular:
file::get_saved_status() which will return the state of the file. If
that state is "saved" then the data of that plain file is present in the
archive and you can get it using the method:

file::get_data()

This method will return you a newly allocated generic_file object (well
a pointer to that object) you will have to destroy using the delete
operator.

This generic_file object has the generic_file::skip() method and the
generic_file::read(char *, size_t) which will provide the data you
requested.



> 
> Thank you very much!
> 


Well, I cannot imagine something more simple, because accessing to one
file's data assumes it is a plain file and it has been saved in the
archive (in other words, its data changed since the archive of reference).

In the hope this will help,

Regards,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFLV0NwpC5CI8gYGlIRAuiDAJ4qEOjhLZ1sFLz7II5jOq1z8tracACeM6SU
MujxmErEJgCYbxEF2ot8GjY=
=QY9c
-----END PGP SIGNATURE-----

[Dar-libdar_api] low-level reading of file inside an archive?

[Guido De R. <gui...@gm...>]

Hi all,

is there a way to do something like this:

read n bytes at offset t of file path/to/my/file.ext inside a DAR
archive and put the result into a std::string or char* or something?

I would like to implement a read-only FUSE file system to look inside
DAR archives, and I need to perform this kind of basic I/O operations
without extracting real files to temporary  directories (and then make
the I/O on the file) because this would be highly inefficient.

Thank you very much!

-- 
Guido De Rosa

Re: [Dar-libdar_api] catalogue/directory: sorted set instead of vector

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

ja...@in... wrote:
[...]

> [..]
>> Yes, I can imagine... There has been some feedback on that point some
>> time ago (that triggered the new implementation by the way).
> 
> Oh well, I searched the mailing lists but obviously not thoroughly enough ;)

As far as I remember it was on dar-support mailing-list, rawly six
months ago.

> 
> [..]
>> Thank you for your feedback Thomas. I do change the 2.3.x releases only
>> for bugs to keep it stable (OK, we could open a long discussion on the
>> fact that this problem is or not a bug. For me it is not, the software
>> does what it is expected to do, slowly, yes). Instead next to come 2.4.0
>> is still under heavy development, thus any change can be brought there
>> with much more ease.
> 
> Sure it's not really a bug. So I'll try to wait for the 2.4.0 release then.
> 
> Just in case that turns out to be too problematic and I would really
> need a working patch for the 2.3.X-branch before that:
> 
> If I would keep the current ordering in the vector<nomme*> fils and  
> additionally add a fast lookup datastructure, to increase the  
> insert/search speed, that should work with the rest of the code,  
> right?

This is what I have done in the current development version. By the way,
You can check that you do not forgot anything in your implementation by
having a look at the current code in CVS Trunk and searching where is
used the map<> data structure of the "directory" class. Maybe that will
help, even if your implementation is different.

> As all
> the data structures are private?

It should not be necessary (nor it is recommanded) to have it public or
just protected.

> 
> Of course this would require additional memory,
> but that's not normally an issue in my case.
> 
>     Thanks for the feedback,
>         Thomas
> 
> 

Regards,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKpVsUpC5CI8gYGlIRArbOAJ94ECGUYIUuITHu9OFxFATkFtEWqwCdGdTa
8QnPy4qXooOioQ2RCElYJ94=
=9S/Z
-----END PGP SIGNATURE-----

Re: [Dar-libdar_api] catalogue/directory: sorted set instead of vector

[<ja...@in...>]

> Yes, this conflict with hard links and it may also lead the user to have
> to change of slice unnecessarily.

[... lots of explanations ... ]

> Now if you look at dar source code in the CVS trunk (the current
> development version), you will see that the hard link support has been
> mostly rewritten and does not have such main and subsequent entries to
> take care of hard linked inodes. And second, the directory class holds
> first a STL list in place of the STL vector you mentioned, and second it
> contains a STL map for fast lookup which points to the entry of the STL
> list.

[..]
> Yes, I can imagine... There has been some feedback on that point some
> time ago (that triggered the new implementation by the way).

Oh well, I searched the mailing lists but obviously not thoroughly enough ;)

[..]
> Thank you for your feedback Thomas. I do change the 2.3.x releases only
> for bugs to keep it stable (OK, we could open a long discussion on the
> fact that this problem is or not a bug. For me it is not, the software
> does what it is expected to do, slowly, yes). Instead next to come 2.4.0
> is still under heavy development, thus any change can be brought there
> with much more ease.

Sure it's not really a bug. So I'll try to wait for the 2.4.0 release then.

Just in case that turns out to be too problematic and I would really
need a working patch for the 2.3.X-branch before that:

If I would keep the current ordering in the vector<nomme*> fils and  
additionally add a fast lookup datastructure, to increase the  
insert/search speed, that should work with the rest of the code,  
right? As all
the data structures are private?

Of course this would require additional memory,
but that's not normally an issue in my case.

    Thanks for the feedback,
        Thomas

Re: [Dar-libdar_api] catalogue/directory: sorted set instead of vector

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello Thomas,

Thomas Jacob a écrit :
> Hey Dennis,
> 
> I have come across a performance issue in DAR 2.3.9 
> that occurs if you archive directories with a huge number of
> files in them (and possibly do on-the-fly-isolate,
> haven't yet tested wether this is required for the
> problem to occur).

I know this problem, yes,

> 
> Specifically, after gprofing the issue, the problem
> seems to happen in the directory class defined in
> catalogue.hpp/cpp which uses the STL vector template
> to store an unsorted list of nommes, and this
> list is often linearily searched to find out if
> a given name is already present in the list,
> with the obvious consequences if the list becomes
> large.
> 
> I've replaced that with the sorted STL set template + std::string
> < operator, which has much nicer performance characteristics
> in this case, but internally sorts the elements and thus
> when reading out the members of a directory
> the order of elements will no longe be the same as when
> they were added to this data structure.
> 
> Could this cause problems with some of DAR's features?

Yes, this conflict with hard links and it may also lead the user to have
to change of slice unnecessarily.

> More specifically, is the order of the elements in the directory class
> relevant for at least one part of DAR?

the order is important first for restoration to avoid having the user
been asked slice N , then N+1, then N again, then N+1 and so on.

The order is also important for hard links, as in the 2.3.x code, the
first entry corresponding to an hard linked inode holds all the
necessary information of the inode. Subsequent entries related to this
hard linked inode just contain a reference to the first one. If when dar
reads the catalogue from the archive (thus before restoring any file)
such subsequent entry is met before the main entry, dar will not be able
to properly build the catalogue in memory. It is thus important to write
the archive catalogue --- in particular a given directory contents ---
in a well defined order, that's it the order the inode have been read
from filesystem.

> If not, would you be interested
> in a patch? Any specific requirements for a patch (at the moment
> I am simply replacing the implementation without any extra autoconf
> or runtime options to control this feature)

Now if you look at dar source code in the CVS trunk (the current
development version), you will see that the hard link support has been
mostly rewritten and does not have such main and subsequent entries to
take care of hard linked inodes. And second, the directory class holds
first a STL list in place of the STL vector you mentioned, and second it
contains a STL map for fast lookup which points to the entry of the STL
list.

For the test I had done, the performance was really improved (I remember
a factor of 40 on some large directories, but I was several month ago,
my memory may be inaccurate and some new features implemented since that
time may have change that deal). For now I am still adding features
testing them after each implementation in as much as possible aspect
they impact but I will focus on global behavior and feature interactions
 when the pre-release phase will start. I have expected a major release
(2.4.0) for the end of the year, but I'm afraid I will probably be
late... so maybe we will only have the pre-release phase before the end
of the year and the release some months after that.

> 
> In any case, the performance impact is fairly major if you archive
> a single directory with a million empty files with the following config

Yes, I can imagine... There has been some feedback on that point some
time ago (that triggered the new implementation by the way).

> (standard dar-2.3.9 --enable-debug --enable-profiling --enable-mode=64):
> 
> --create /tmp/dar_test
> --on-fly-isolate /tmp/dar_test_iso
> --fs-root DAR_Test/ # 1 million empty files (named 0 .. 999999)
> --slice 500M
> 
> It takes about 6 minutes to run this with my patched version,
> the unpatched version is still running (since about 2 days
> now), I am letting it finish to see what the difference in memory
> usage is.
> 
> Sure, having a million files in a directory, particularily in a Linux
> ext3 directory, is not such a clever idea, but customers are well
> customers ;)

Yes! I fully Agree! By the way, old filesystems (like first versions of
ext2) had also this behavior of heavy penalty on large directories, for
that reason many software like split their cache directory in a several
directory level tree.

> 
>     Thomas
> 
> 

Thank you for your feedback Thomas. I do change the 2.3.x releases only
for bugs to keep it stable (OK, we could open a long discussion on the
fact that this problem is or not a bug. For me it is not, the software
does what it is expected to do, slowly, yes). Instead next to come 2.4.0
is still under heavy development, thus any change can be brought there
with much more ease.

Kind Regards,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKpAaPpC5CI8gYGlIRAlkiAJ0f5lRLjNEFJwHZJzXkxO8SKmfCAwCgr7gO
VJ4QixJmCgX8Lz2XlgGRbtI=
=o0BG
-----END PGP SIGNATURE-----

[Dar-libdar_api] catalogue/directory: sorted set instead of vector

[Thomas J. <ja...@in...>]

Hey Dennis,

I have come across a performance issue in DAR 2.3.9 
that occurs if you archive directories with a huge number of
files in them (and possibly do on-the-fly-isolate,
haven't yet tested wether this is required for the
problem to occur).

Specifically, after gprofing the issue, the problem
seems to happen in the directory class defined in
catalogue.hpp/cpp which uses the STL vector template
to store an unsorted list of nommes, and this
list is often linearily searched to find out if
a given name is already present in the list,
with the obvious consequences if the list becomes
large.

I've replaced that with the sorted STL set template + std::string
< operator, which has much nicer performance characteristics
in this case, but internally sorts the elements and thus
when reading out the members of a directory
the order of elements will no longe be the same as when
they were added to this data structure.

Could this cause problems with some of DAR's features?
More specifically, is the order of the elements in the directory class
relevant for at least one part of DAR? If not, would you be interested
in a patch? Any specific requirements for a patch (at the moment
I am simply replacing the implementation without any extra autoconf
or runtime options to control this feature)

In any case, the performance impact is fairly major if you archive
a single directory with a million empty files with the following config
(standard dar-2.3.9 --enable-debug --enable-profiling --enable-mode=64):

--create /tmp/dar_test
--on-fly-isolate /tmp/dar_test_iso
--fs-root DAR_Test/ # 1 million empty files (named 0 .. 999999)
--slice 500M

It takes about 6 minutes to run this with my patched version,
the unpatched version is still running (since about 2 days
now), I am letting it finish to see what the difference in memory
usage is.

Sure, having a million files in a directory, particularily in a Linux
ext3 directory, is not such a clever idea, but customers are well
customers ;)

    Thomas

Re: [Dar-libdar_api] compiling the 64 bit libdar in AIX 5.3 platform

[Thomas J. <ja...@in...>]

On Sun, 2009-01-25 at 23:21 +0530, Sivakumar Selvam wrote:
> 
> I ran through gdb and got the following output. 
>  
> (gdb) r
> Starting
> program: /usr/sony/iTRx/siva/64bit/dec26/dar-2.3.6/src/dar_suite/.libs/dar
> Program received signal SIGILL, Illegal instruction.
> 0x0000000000000000 in ?? ()
> (gdb) bt
> #0  0x0000000000000000 in ?? ()
> #1  0x09000000011a638c in reg_frame ()
> from /usr/sony/iTRx/siva/64bit/dec26/dar-2..3.6/src/libdar/.libs/libdar.a(libdar.so.4)
> #2  0x09000000011a6490 in _GLOBAL__FI_libdar_so ()
> 
> from /usr/sony/iTRx/siva/64bit/dec26/dar-2.3.6/src/libdar/.libs/libdar.a(libdar.so.4)
> #3  0x09fffffff0001bd4 in mod_init1 () from /usr/ccs/bin/usla64
> #4  0x09fffffff000290c in usl_init_mods () from /usr/ccs/bin/usla64
> #5  0x09fffffff000280c in usl_exec_init_mods ()
> from /usr/ccs/bin/usla64
> #6  0x000000010000029c in __start ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> (gdb)

Sorry but I have not clue about compiling & linking stuff under AIX, all
I can see from this backtrace is that the segfault seems to occur before
any libdar code is called so it's probably a compilation/linking issue.