dar-libdar_api — discussion area for developpers that want to use the libdar API

[Dar-libdar_api] dar_manager API

[Johnathan B. <jk...@co...>]

Hi Denis,

I hope things are well with you. I'm settled in the U.S. and now have some 
time to think about dar/kdar. Glad to see you've made the move to have 
dar_manager in the API. While it looks like you haven't made an official 
announcement yet, I'll comment and ask questions anyway.

The API includes functions for manipulating the path to the dar binary. 
Shouldn't the database object just call libdar functions directly through the 
API?

add_archive(...) requires a catalogue reference. It seems akward from a 
developer point of view: macro_tools.hpp has to be included in order to use 
the "macro_tools_get_catalogue_from()" helper to get the catalogue. This 
could be called from within add_archive() directly, no?

Cheers,
JB
-- 
Johnathan K. Burchill, Ph.D.
jk...@co...

[Dar-libdar_api] Re: Error reading EA

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

Denis Corbin wrote:
| Wesley Leggette wrote:
| | On Mon, 2005-03-21 at 21:20 +0100, Denis Corbin wrote:
| |
| | Johnathan Burchill wrote:
| | | Hi Denis,
| |
| | Hello Johnathan,
| |
| | |
| | | From the dar-support list archive, message of 2005-03-01 12:30:
| | |
| | |
| | |>| They explictly mention the security namespace but I don"t know
at all
| | |>| if this is standard.
| | |
| | |
| | |
| | |>I don"t know neither. I will look at this soon.
| | |
| | |
| | | On my Slackware 10.1 box, "man 5 attr" shows 4 namespaces:
| | |
| | |        "Currently the security, system, trusted, and user extended
| | |        attribute  classes  are  defined as described below. Addi-
| | |        tional classes may be added in the future."
| | |
| | | The fact that more may be added in future makes me think there must be
| | a way
| | | to implement EA support in libdar in a more general way, that is
| | independent
| | | of specific namespaces...
| | |
| |
| | Yes, this is the object of a request for evolution, and of course I
| | don't want to have to change dar upon each new namespace that could
| | appear in the future. Historically (may 2002) when I read the
| | documentation provided by Andreas Gruenbacher (Author of EA/ACL) only
| | two namespaces were defined and it seemed there were no need to have
| | some more... :-)
| |
|[...]
|
| My current idea is to make namespace transparent to dar. the -u and -U
| options would change their meaning becoming a mask option (like -Y and
| -Z for example) but applied to the whole EA name. For example -u
| "system.*" would exclude all EA of the system namespace while -U
| "security.*" would include only the security namespace. One could also
| restore or save only a particular set of EA using several -u or -U
| basing the selection on namespace and/or name of the EA.
|
| This implies API changes of course.


| [...]

This is now implemented under the CVS trunk. The documentation, man
pages, API tutorial and doxygen comments are up to date.

Rawly, this has been done as explained above, EA namespace is now
transparent to dar, for libdar, a mask replaces the ea_root and ea_user
switchs to defines which EA to consider. For dar, the -u and -U options
have changed and now take an argument.

Cheers,
Denis.



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFCfgpHpC5CI8gYGlIRAmGFAKC9Grn6I80afJJwejDiw+XwMbIHUwCgtcyG
4DWa5sUsU+Ji80XXEzoK3cQ=
=zxwO
-----END PGP SIGNATURE-----

[Dar-libdar_api] Re: Progress reporting

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Johnathan Burchill wrote:
| On Tuesday 29 March 2005 01:38, Johnathan Burchill wrote:
|
|>Hi Denis,

Hello Johnathan,

I understand your need. To my point of view, this is not really like
user interaction. I would rather transform the statistics structure in a
class. Then an object of that class would be given by the API caller as
argument of each operations, this object is incremented by libdar code
as it is done actually with the statistics structure, but as this would
be now a class and as it is not a returned value but a IN/OUT parameter,
it is know by the caller and can be consulted at will during the libdar
operation (this assumes one thread for the libdar operation another for
consulting the counters).

Why this way? Because this solves the problem for libdar to know when to
send information to the API caller. Instead this is the API caller that
fetches this informations when it needs (it can done by any mean like a
timer, a GUI event and so on). It also avoids having libdar slowed by
this type of report done too often and/or ignored by the caller.

So concretely, we could have:

class statistics
{
	public:
	 void set_treated(const infinint & val);
~         void set_hard_links(const infinint & val);
~          ....
~         infinint get_treated() const;
~         infinint get_hard_links() const;
~          ....

~         void clear();
	 void get_photo(infinint & treated, infinint & hard_links, ...) const;

	private:
	  infinint treated;
~          infinint hard_links;
~          ...
}

but each set/get method would use a mutex to access the corresponding
variable.

and we could have a typical call to libdar this way:

~     statistics stats;
~     stats.clear();

~     <thread one>
~         op_create(...., stats);
~              // "stats" argument is passed as: statistics & stats

~     <thread two>
~         while(...)
~         {
~             stats.get_photo(....);
~             display_counters_to_user(...);
~             sleep(a_certain_time);
~         }



|>
|
|
| [...]
|
|
Cheers,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFCSwf9pC5CI8gYGlIRAtlDAJ9OeJ/kS6RQGckcIFuAlQg1GncJGwCfXZgr
IdwNPLafDYzCNFBYDiYSNl0=
=fmkn
-----END PGP SIGNATURE-----

[Dar-libdar_api] Re: Error reading EA

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Wesley Leggette wrote:
| On Thu, 2005-03-24 at 22:56 +0100, Denis Corbin wrote:
|
|[...]
|
|> I should stress that what I'm suggesting only uses the EA code, but the
|> "encapsulated" flag indicates it is not a real EA. More below.

Yes, but if this "encapsulated" is note real EA how could it use real EA
code? So we have different action depending on the domain of the EA
which is not the case today (dar's code is EA-domain independent).

|
|[...]
|
| Any objections are welcome :-)
|
|
|> I see your point and agree with it. That is exactly why I thought
|> storing these "non-standard" (non-Posix) values as "Attributes" would be
|> good. I.e., instead of extending the catalog entries to add more values
|> for each system, if an archive is created on a specific system, it can
|> store those non-standard values as "fake" attributes (thus, the
|> "encapsulated" meant that they are not really attributes that would be
|> restored on a Linux system, they are just values that OS specific
|> implementations could restore if they wanted).

Whatever is the way theses filesystem specific informations are stored
in the archive (through a "encapsulated" EA or on their own beside other
inode informations), we still have the problem of specific code for
different filesystem. Let's take the read-only notion as example: Under
Unix this is stored with the permission (user/group/other write access
off), under DOS this is a file attribute. How to restore a r--r--rw-
file under DOS then?

Now you can extrapolate to Macintosh forks, and so one. Storing
filesystem specific information has heavy consequences in portability
and probably also in stability of the application, as much as more
different filesytems are handled in their specificities. Because there
will not be a generic way to handle a file whatever is the filesystem it
comes from and it goes to (which what we have today).

|
|> So, here's how I would envision it:
|
|> You have libdar on WIN32. You back up some files, and they all have
|> these "encapsulated" attributes with them. But of course, they are
|> really just optional non-standard information.
|
|> Then, you take that archive to a linux machine. There, maybe the fact
|> that these WIN32 values are present is displayed to the user, or maybe
|> not. Regardless, they are not restored to the hard drive. Instead, just
|> the standard Posix values are.

what about the read-only property?

|
|> Then, maybe you take this same archive to another WIN32 machine. Here,
|> these "encapsulated" values could be restored. See?

Yes I see. So you should have as much encapsulated domaine as there is
filesystem with handled specificities. Assuming you save the 'sync' ext2
attribute (which is not the case today), as it is ext2 specific, you
should have to store this information in an EA of an "encapsulated"
domain, but not the same "encapsulated" domain as the one handling the
System/Archive/Read-Only attributes found in FAT filesystem. What about
the ext2 'dump' attribute then, it is the FAT's "Archive" attribute
equivalent. Which domain would it be stored in?

This also raise the filesystem detection problem. How to know which
fileystem is used in a portable way? Because of course you must know
which "encapsulated domain has to/can be restored. Moreover it must be
detected file by file, if for example you mount a fat16 partition on a
ext2 filesystem, and also have an iso9660...

|
|> So it's an extensible system where all non-standard information would be
|> stored in key/value pairs. Any values that an implementation doesn't
|> understand can be ignored, and any missing values will be made up from
|> the posix standard data (like r-x would equal the read-only flag).

The idea to have a set of attributes that is used by thoses
implementation that know about it an ignored by others implies several
implementations.

now, having different applications, is what we have for HTML when a
browser ignores marks it does not know. As you can see with HTML, it
does not displays the same way from browser to browser, this has minor
importance because the target is the human eye and human can interprete
what it sees.

But Dar's target restoring files is not human eye, but computer's
filesystems, and behind theses, are computer applications that use
theses files. Filesystems as well as computer applications do not have
the same ability to interprete what the use. I am affraid that one saved
file could result in slightly different restored files depending on
filesystem it is restored on and depending on dar's implementation and
operating system used.

Rewriting dar to natively use windows system is one thing. Making a
different implementation to add extensions is very different. This is
what Microsoft did with HTML, that finally ruined HTML standard as much
web site's use Microsoft's extensions, you know, those web sites where
you can see: "your browser is not supported, you need to use Microsoft's
IE version 6.2". This is quite the opposite of the original idea of
HTML, where it was planed to be browser and system independant.

|
|> This is sort of how RAR and ZIP do it, but they used fixed extensions. I
|> think this is more flexible.
|

do ZIP and RAR save FAT attributes? do they save ressource forks? Do
they save ext2 file attributes? I am not sure of that.

Even if they do, what would be the use of having a cross platform
application if they only can restore in a portable manner what dar
handles today?

|
|> As far as Mac OS X is concerned: Yes, the forks would have to be saved.
|> Note also that NTFS uses alternate data streams (which are very similiar
|> to forks). Thus, if dar was able to save forks, it would kill two birds
|> with one stone.
|
|> I propose adding full multiple data stream support. Basically, how this
|> would work is similiar to the ideas presented in Reiserfs 4. Each file
|> acts as a sort of "directory" containing multiple other entries. I would
|> propose a special "stream" entree type, and perhaps the EOD entree could
|> end a listing of streams after a file_stream entry, or something like
|> that.
|
|> As far as all that goes, I should note that the reason I've been writing
|> up a detailed specification of dar edition 3 (and I'm going to write the
|> edition 4 specs soon) is that I want to propose changes to incorporate
|> the features I mention above. I intend to be crystal clear about what
|> the changes would be and what the API would be to access them. I am
|> working on documenting the private API for the same reason.


So we would end with many different implementations, one working for Mac
OS (saving forks), one for DOS/windows (saving DOS attributes) one for
NTFS and so on. Why then not using existing backup tools on theses
systems, why taking dar for that? There is no more need of portability,
which is what dar brings.

|
|
|
| |
| |> So, these values would have the "namespace" stored within the key name.
| |> But if you change the namespace method to be more generic... This
may be
| |> a bit of overkill.
| |
| |
| |> Offtopic here, but I have also been giving some thought on file
streams.
| |> As you know, Mac OS X uses multiple file streams, but NTFS does as
well.
| |> I've been coming up with a plan to support these things (and of course
| |> I'd be implementing them). Let me know if you're interested in
| |> discussing this.
|
| As soon as we can have a standard feature, with a standard API I will
| say yes. Else, we will loose time an energy, and will make dar less
| stable (having to consider many different cases...).
|
|
|> Noting my work on documentation, yes, I agree fully. My main efforts
|> right now have been focused on documenting the file format and the
|> private API for just this reason. After the current API is documented,

wasn't it? ;-)

|> I
|> will document the proposed changes. This way, we can come to some
|> consensus before any implementation. And of course, in the worst case,
|> at least there will be some useful documents of the current system ;)
|
|
|> Well, thanks,
|> Wesley
|
|[...]

Cheers,
Denis.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFCSwNYpC5CI8gYGlIRAgh5AKCA9LYzixfjDVwZQGHbqMFTtjcxdwCeLR0I
r6p+1dnNtNw4gsQBpXIMfCo=
=hVPI
-----END PGP SIGNATURE-----

Re: [Dar-libdar_api] Progress reporting

[Johnathan B. <jk...@us...>]

On Tuesday 29 March 2005 01:38, Johnathan Burchill wrote:
> Hi Denis,
>

[...]

> Another way could be to change
>
> void libdar::user_interaction::warning( const std::string & message )
>
> to
>
> void libdar::user_interaction::warning( const libdar::message & info )

whoops, that should be

 void libdar::user_interaction::warning( const libdar::info & information )


>
> where info is
>
> struct info
> {
>     std::string message;
>     bool progress_update;
>     libdar::statistics stats;
> }

[...]

>
> Cheers,
> JB

=2D-=20
Johnathan K. Burchill, Ph.D.
jk...@us...

[Dar-libdar_api] Progress reporting

[Johnathan B. <jk...@us...>]

Hi Denis,

I'm interested in being able to get progress on libdar operations while the=
y=20
happen. Something like a visual representation of the libdar::statistics=20
structure as the counters are being incremented would be helpful.

Since libdar already provides this structure at the end of the operation, i=
t=20
seems it would be possible to store it in the user_interaction abstract=20
class, giving the user access to it at will, perhaps via internal accessor=
=20
methods involving mutexes. One could argue that providing the user with=20
operation stats is a form of user interaction anyway. So you could then get=
=20
rid of the separate "libdar::statistics" parameter in each archive operatio=
n=20
method.

Another way could be to change=20

void libdar::user_interaction::warning( const std::string & message )

to=20

void libdar::user_interaction::warning( const libdar::message & info )

where info is=20

struct info
{
    std::string message;
    bool progress_update;
    libdar::statistics stats;
}

(or change it to void libdar::user_interaction::warning( const std::string =
&=20
message, const libdar::statistics & stats ) )

Then I could do the following in KDar:

KDarInteraction::warning( const libdar::info info)
{
    if ( info.progress_update )
    {
        m_inodesProcessed =3D info.stats.treated;
        progressBar->update( (int) (m_inodesProcessed*100/totalInodes) );
    }
}

where totalInodes is computed by running the creation operation in dry-run=
=20
mode beforehand or concurrently as a separate, but much faster, thread (to =
be=20
done by the client).

What do you think?

Cheers,
JB

=2D-=20
Johnathan K. Burchill, Ph.D.
jk...@us...

Re: [Dar-libdar_api] Re: Error reading EA

[Wesley L. <li...@ka...>]

On Thu, 2005-03-24 at 22:56 +0100, Denis Corbin wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>=20
> Wesley Leggette wrote:
> | On Mon, 2005-03-21 at 21:20 +0100, Denis Corbin wrote:
> |
> | Johnathan Burchill wrote:
> | | Hi Denis,
> |
> | Hello Johnathan,
> |
> | |
> | | From the dar-support list archive, message of 2005-03-01 12:30:
> | |
> | |
> | |>| They explictly mention the security namespace but I don"t know at a=
ll
> | |>| if this is standard.
> | |
> | |
> | |
> | |>I don"t know neither. I will look at this soon.
> | |
> | |
> | | On my Slackware 10.1 box, "man 5 attr" shows 4 namespaces:
> | |
> | |        "Currently the security, system, trusted, and user extended
> | |        attribute  classes  are  defined as described below. Addi-
> | |        tional classes may be added in the future."
> | |
> | | The fact that more may be added in future makes me think there must b=
e
> | a way
> | | to implement EA support in libdar in a more general way, that is
> | independent
> | | of specific namespaces...
> | |
> |
> | Yes, this is the object of a request for evolution, and of course I
> | don't want to have to change dar upon each new namespace that could
> | appear in the future. Historically (may 2002) when I read the
> | documentation provided by Andreas Gruenbacher (Author of EA/ACL) only
> | two namespaces were defined and it seemed there were no need to have
> | some more... :-)
> |
> |
> |
> |> This was a concern for me as well. I am working on implementing native
> |> win32 support. Here's what I envisioned (at that time I knew only of
> |> user, system, and trusted--not security):
> |
> |> 0x00    user (insert)
> |> 0x80    system
> |> 0x40    (delete mode)
> |> 0x20    encapsulated
> |> 0x10    trusted
> |
>=20
> My current idea is to make namespace transparent to dar. the -u and -U
> options would change their meaning becoming a mask option (like -Y and
> - -Z for example) but applied to the whole EA name. For example -u
> "system.*" would exclude all EA of the system namespace while -U
> "security.*" would include only the security namespace. One could also
> restore or save only a particular set of EA using several -u or -U
> basing the selection on namespace and/or name of the EA.
>=20
> This implies API changes of course.
>=20
> |
> |> What I mean by "encapsulated" is that I wanted to store additional
> |> attributes in this mechanism. I had in mind "virtualizing" FAT and NTF=
S
> |> attributes like this:
> |
> |> Key                         Description
> |> -----                       --------------
> |> FAT.ATTRIB:BYTE             Basic FAT attributes byte
> |> FAT.TIME:C                  FAT creation time
> |> NTFS.ATTRIB:COMPRESSED      NTFS file compressed flag
> |> NTFS.ATTRIB:ENCRYPTED       NTFS file encrypted flag
> |> ...
> |> NTFS.TIME:C                 NTFS creation time in FILETIME format
> |> ...
> |
>=20
> How to stay portable then? Permission already carry the readonly flag
> (which is a fat attribute), time is also carried in dar's archive, (last
> modification time, last access time), creation time is not portable, but
> can be stored as last modification time for DOS systems. While file
> format is a display problem not a storage problem (the same information
> has not to be saved twice).
>=20
> I don't think the EA is a good place to store filesystem specific
> information. EA has a quite standard definition and API, what if one day
> the "encapsulation" namespace is used somewhere else. In another side,
> dar has not to understand the EA meaning or role in the filesystem.

I should stress that what I'm suggesting only uses the EA code, but the
"encapsulated" flag indicates it is not a real EA. More below.

>=20
> If dar must store the compressed attribute and restore it it must be
> done beside this, as new fields. (ext2/3 have several fields, ntfs, fat
> have their own fields, macintosh filesystem have a more complex own
> structure (the forks)).
>=20
> There is a choice to make: stay generic and system independant, thus
> drop some filesystem specificities but allow inter filesystem exchanges,
> or have the knowledge of all the possible filesystem on Earth, and spend
> 90% of the time maintaing dar to fit each filesystem evolutions, and
> loose time considering what would come if one want to restore a file
> from a filesystem to another, does this or this attribute (in a very
> general meaning) has to be casted to a more or less corresponding one on
> the target filesystem?
>=20
> As you can guess now, my choice is the generic way. Following standards
> leads to portability. For that reason I have dropped ext2 attributes
> from dar's abilities, nobody complained, yet. Look at dar's venerable
> grand brother: tar. It has become the most popular tool used for so long
> time... does it knows of all of the filesystems specificities? no. And
> is is still used today.
>=20
> Any objections are welcome :-)

I see your point and agree with it. That is exactly why I thought
storing these "non-standard" (non-Posix) values as "Attributes" would be
good. I.e., instead of extending the catalog entries to add more values
for each system, if an archive is created on a specific system, it can
store those non-standard values as "fake" attributes (thus, the
"encapsulated" meant that they are not really attributes that would be
restored on a Linux system, they are just values that OS specific
implementations could restore if they wanted).

So, here's how I would envision it:

You have libdar on WIN32. You back up some files, and they all have
these "encapsulated" attributes with them. But of course, they are
really just optional non-standard information.

Then, you take that archive to a linux machine. There, maybe the fact
that these WIN32 values are present is displayed to the user, or maybe
not. Regardless, they are not restored to the hard drive. Instead, just
the standard Posix values are.

Then, maybe you take this same archive to another WIN32 machine. Here,
these "encapsulated" values could be restored. See?

So it's an extensible system where all non-standard information would be
stored in key/value pairs. Any values that an implementation doesn't
understand can be ignored, and any missing values will be made up from
the posix standard data (like r-x would equal the read-only flag).

This is sort of how RAR and ZIP do it, but they used fixed extensions. I
think this is more flexible.


As far as Mac OS X is concerned: Yes, the forks would have to be saved.
Note also that NTFS uses alternate data streams (which are very similiar
to forks). Thus, if dar was able to save forks, it would kill two birds
with one stone.

I propose adding full multiple data stream support. Basically, how this
would work is similiar to the ideas presented in Reiserfs 4. Each file
acts as a sort of "directory" containing multiple other entries. I would
propose a special "stream" entree type, and perhaps the EOD entree could
end a listing of streams after a file_stream entry, or something like
that.

As far as all that goes, I should note that the reason I've been writing
up a detailed specification of dar edition 3 (and I'm going to write the
edition 4 specs soon) is that I want to propose changes to incorporate
the features I mention above. I intend to be crystal clear about what
the changes would be and what the API would be to access them. I am
working on documenting the private API for the same reason.


>=20
> |
> |> So, these values would have the "namespace" stored within the key name=
.
> |> But if you change the namespace method to be more generic... This may =
be
> |> a bit of overkill.
> |
> |
> |> Offtopic here, but I have also been giving some thought on file stream=
s.
> |> As you know, Mac OS X uses multiple file streams, but NTFS does as wel=
l.
> |> I've been coming up with a plan to support these things (and of course
> |> I'd be implementing them). Let me know if you're interested in
> |> discussing this.
>=20
> As soon as we can have a standard feature, with a standard API I will
> say yes. Else, we will loose time an energy, and will make dar less
> stable (having to consider many different cases...).

Noting my work on documentation, yes, I agree fully. My main efforts
right now have been focused on documenting the file format and the
private API for just this reason. After the current API is documented, I
will document the proposed changes. This way, we can come to some
consensus before any implementation. And of course, in the worst case,
at least there will be some useful documents of the current system ;)


Well, thanks,
Wesley


>=20
> |
> |> Thanks,
> |> Wesley
> |
> |
> |
> |
> |
> |
> | | Cheers,
> | | JB
> |
> | Cheers,
> | Denis.
> |
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.6 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>=20
> iD8DBQFCQzeWpC5CI8gYGlIRAoTFAKCvti6MW74l6MtX503oSjxL6g5LIACfa6sA
> G148kInM7tAOSu1SBy7l/ts=3D
> =3Dn49n
> -----END PGP SIGNATURE-----
>=20
>=20
>=20
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=3D6595&alloc_id=3D14396&op=3Dclick
> _______________________________________________
> Dar-libdar_api mailing list
> Dar...@li...
> https://lists.sourceforge.net/lists/listinfo/dar-libdar_api
--=20
Wesley Leggette <li...@ka...>

GPG Key:         http://www.kaylix.net/kaylix.asc or http://pgp.mit.edu
GPG Fingerprint: 9B6F 19FB 5296 5E6C 21FE  7614 2A20 5688 F848 9BDD

[Dar-libdar_api] Re: Error reading EA

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Wesley Leggette wrote:
| On Mon, 2005-03-21 at 21:20 +0100, Denis Corbin wrote:
|
| Johnathan Burchill wrote:
| | Hi Denis,
|
| Hello Johnathan,
|
| |
| | From the dar-support list archive, message of 2005-03-01 12:30:
| |
| |
| |>| They explictly mention the security namespace but I don"t know at all
| |>| if this is standard.
| |
| |
| |
| |>I don"t know neither. I will look at this soon.
| |
| |
| | On my Slackware 10.1 box, "man 5 attr" shows 4 namespaces:
| |
| |        "Currently the security, system, trusted, and user extended
| |        attribute  classes  are  defined as described below. Addi-
| |        tional classes may be added in the future."
| |
| | The fact that more may be added in future makes me think there must be
| a way
| | to implement EA support in libdar in a more general way, that is
| independent
| | of specific namespaces...
| |
|
| Yes, this is the object of a request for evolution, and of course I
| don't want to have to change dar upon each new namespace that could
| appear in the future. Historically (may 2002) when I read the
| documentation provided by Andreas Gruenbacher (Author of EA/ACL) only
| two namespaces were defined and it seemed there were no need to have
| some more... :-)
|
|
|
|> This was a concern for me as well. I am working on implementing native
|> win32 support. Here's what I envisioned (at that time I knew only of
|> user, system, and trusted--not security):
|
|> 0x00    user (insert)
|> 0x80    system
|> 0x40    (delete mode)
|> 0x20    encapsulated
|> 0x10    trusted
|

My current idea is to make namespace transparent to dar. the -u and -U
options would change their meaning becoming a mask option (like -Y and
- -Z for example) but applied to the whole EA name. For example -u
"system.*" would exclude all EA of the system namespace while -U
"security.*" would include only the security namespace. One could also
restore or save only a particular set of EA using several -u or -U
basing the selection on namespace and/or name of the EA.

This implies API changes of course.

|
|> What I mean by "encapsulated" is that I wanted to store additional
|> attributes in this mechanism. I had in mind "virtualizing" FAT and NTFS
|> attributes like this:
|
|> Key                         Description
|> -----                       --------------
|> FAT.ATTRIB:BYTE             Basic FAT attributes byte
|> FAT.TIME:C                  FAT creation time
|> NTFS.ATTRIB:COMPRESSED      NTFS file compressed flag
|> NTFS.ATTRIB:ENCRYPTED       NTFS file encrypted flag
|> ...
|> NTFS.TIME:C                 NTFS creation time in FILETIME format
|> ...
|

How to stay portable then? Permission already carry the readonly flag
(which is a fat attribute), time is also carried in dar's archive, (last
modification time, last access time), creation time is not portable, but
can be stored as last modification time for DOS systems. While file
format is a display problem not a storage problem (the same information
has not to be saved twice).

I don't think the EA is a good place to store filesystem specific
information. EA has a quite standard definition and API, what if one day
the "encapsulation" namespace is used somewhere else. In another side,
dar has not to understand the EA meaning or role in the filesystem.

If dar must store the compressed attribute and restore it it must be
done beside this, as new fields. (ext2/3 have several fields, ntfs, fat
have their own fields, macintosh filesystem have a more complex own
structure (the forks)).

There is a choice to make: stay generic and system independant, thus
drop some filesystem specificities but allow inter filesystem exchanges,
or have the knowledge of all the possible filesystem on Earth, and spend
90% of the time maintaing dar to fit each filesystem evolutions, and
loose time considering what would come if one want to restore a file
from a filesystem to another, does this or this attribute (in a very
general meaning) has to be casted to a more or less corresponding one on
the target filesystem?

As you can guess now, my choice is the generic way. Following standards
leads to portability. For that reason I have dropped ext2 attributes
from dar's abilities, nobody complained, yet. Look at dar's venerable
grand brother: tar. It has become the most popular tool used for so long
time... does it knows of all of the filesystems specificities? no. And
is is still used today.

Any objections are welcome :-)

|
|> So, these values would have the "namespace" stored within the key name.
|> But if you change the namespace method to be more generic... This may be
|> a bit of overkill.
|
|
|> Offtopic here, but I have also been giving some thought on file streams.
|> As you know, Mac OS X uses multiple file streams, but NTFS does as well.
|> I've been coming up with a plan to support these things (and of course
|> I'd be implementing them). Let me know if you're interested in
|> discussing this.

As soon as we can have a standard feature, with a standard API I will
say yes. Else, we will loose time an energy, and will make dar less
stable (having to consider many different cases...).

|
|> Thanks,
|> Wesley
|
|
|
|
|
|
| | Cheers,
| | JB
|
| Cheers,
| Denis.
|
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFCQzeWpC5CI8gYGlIRAoTFAKCvti6MW74l6MtX503oSjxL6g5LIACfa6sA
G148kInM7tAOSu1SBy7l/ts=
=n49n
-----END PGP SIGNATURE-----

Re: [Dar-libdar_api] Re: Error reading EA

[Wesley L. <li...@ka...>]

On Mon, 2005-03-21 at 21:20 +0100, Denis Corbin wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>=20
> Johnathan Burchill wrote:
> | Hi Denis,
>=20
> Hello Johnathan,
>=20
> |
> | From the dar-support list archive, message of 2005-03-01 12:30:
> |
> |
> |>| They explictly mention the security namespace but I don"t know at all
> |>| if this is standard.
> |
> |
> |
> |>I don"t know neither. I will look at this soon.
> |
> |
> | On my Slackware 10.1 box, "man 5 attr" shows 4 namespaces:
> |
> |        "Currently the security, system, trusted, and user extended
> |        attribute  classes  are  defined as described below. Addi-
> |        tional classes may be added in the future."
> |
> | The fact that more may be added in future makes me think there must be
> a way
> | to implement EA support in libdar in a more general way, that is
> independent
> | of specific namespaces...
> |
>=20
> Yes, this is the object of a request for evolution, and of course I
> don't want to have to change dar upon each new namespace that could
> appear in the future. Historically (may 2002) when I read the
> documentation provided by Andreas Gruenbacher (Author of EA/ACL) only
> two namespaces were defined and it seemed there were no need to have
> some more... :-)


This was a concern for me as well. I am working on implementing native
win32 support. Here's what I envisioned (at that time I knew only of
user, system, and trusted--not security):

0x00    user (insert)
0x80    system
0x40    (delete mode)
0x20    encapsulated
0x10    trusted


What I mean by "encapsulated" is that I wanted to store additional
attributes in this mechanism. I had in mind "virtualizing" FAT and NTFS
attributes like this:

Key                         Description
-----                       --------------
FAT.ATTRIB:BYTE             Basic FAT attributes byte
FAT.TIME:C                  FAT creation time
NTFS.ATTRIB:COMPRESSED      NTFS file compressed flag
NTFS.ATTRIB:ENCRYPTED       NTFS file encrypted flag
...
NTFS.TIME:C                 NTFS creation time in FILETIME format
...


So, these values would have the "namespace" stored within the key name.
But if you change the namespace method to be more generic... This may be
a bit of overkill.


Offtopic here, but I have also been giving some thought on file streams.
As you know, Mac OS X uses multiple file streams, but NTFS does as well.
I've been coming up with a plan to support these things (and of course
I'd be implementing them). Let me know if you're interested in
discussing this.

Thanks,
Wesley





>=20
> | Cheers,
> | JB
>=20
> Cheers,
> Denis.
>=20
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.6 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>=20
> iD8DBQFCPyySpC5CI8gYGlIRAl42AJ9X0ec9lCTDPYcdlYAlP4fJooVD6QCfX6OQ
> 7jReRYF6e2zuiU+w/cSGEdQ=3D
> =3DJJw2
> -----END PGP SIGNATURE-----
>=20
>=20
>=20
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=3D6595&alloc_id=3D14396&op=3Dclick
> _______________________________________________
> Dar-libdar_api mailing list
> Dar...@li...
> https://lists.sourceforge.net/lists/listinfo/dar-libdar_api
--=20
Wesley Leggette <li...@ka...>

GPG Key:         http://www.kaylix.net/kaylix.asc or http://pgp.mit.edu
GPG Fingerprint: 9B6F 19FB 5296 5E6C 21FE  7614 2A20 5688 F848 9BDD

[Dar-libdar_api] Re: Error reading EA

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Johnathan Burchill wrote:
| Hi Denis,

Hello Johnathan,

|
| From the dar-support list archive, message of 2005-03-01 12:30:
|
|
|>| They explictly mention the security namespace but I don"t know at all
|>| if this is standard.
|
|
|
|>I don"t know neither. I will look at this soon.
|
|
| On my Slackware 10.1 box, "man 5 attr" shows 4 namespaces:
|
|        "Currently the security, system, trusted, and user extended
|        attribute  classes  are  defined as described below. Addi-
|        tional classes may be added in the future."
|
| The fact that more may be added in future makes me think there must be
a way
| to implement EA support in libdar in a more general way, that is
independent
| of specific namespaces...
|

Yes, this is the object of a request for evolution, and of course I
don't want to have to change dar upon each new namespace that could
appear in the future. Historically (may 2002) when I read the
documentation provided by Andreas Gruenbacher (Author of EA/ACL) only
two namespaces were defined and it seemed there were no need to have
some more... :-)

| Cheers,
| JB

Cheers,
Denis.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFCPyySpC5CI8gYGlIRAl42AJ9X0ec9lCTDPYcdlYAlP4fJooVD6QCfX6OQ
7jReRYF6e2zuiU+w/cSGEdQ=
=JJw2
-----END PGP SIGNATURE-----

[Dar-libdar_api] Re: Error reading EA

[Johnathan B. <jk...@us...>]

Hi Denis,=20

=46rom the dar-support list archive, message of 2005-03-01 12:30:=20

> | They explictly mention the security namespace but I don"t know at all
> | if this is standard.
=20
> I don"t know neither. I will look at this soon.

On my Slackware 10.1 box, "man 5 attr" shows 4 namespaces:

       "Currently the security, system, trusted, and user extended
       attribute  classes  are  defined as described below. Addi-
       tional classes may be added in the future."

The fact that more may be added in future makes me think there must be a wa=
y=20
to implement EA support in libdar in a more general way, that is independen=
t=20
of specific namespaces...

Cheers,
JB
=2D-=20
Johnathan K. Burchill, Ph.D.
jk...@us...

Re: [Dar-libdar_api] Re: Archive info method

[Johnathan B. <jk...@us...>]

On Tuesday 18 January 2005 13:49, Denis Corbin wrote:
> Johnathan Burchill wrote:
> | Hi Denis,
>
> Hello Johnathan,
>
> | I've been thinking for awhile about a way to get info about a dar
>
> archive.
>
> | The 3.0 API returns info via the op_listing method, i.e. archive size,
> | # slices, version no., whether it's encrypted, # directories, #
> | regular files, etc.
> |
> | For example, a listing of an archive from the command line gives:
> |
> | Extracting contents of the archive...
> | Archive version format               : 04
> | Compression algorithm used           : bzip2
> | Scrambling or strong encryption used : yes
> | Catalogue size in archive            : 54380 bytes
> | Archive is composed of 1 file(s)
> | File size: 38243193 bytes
> |
> | CATALOGUE CONTENTS :
> |
> | total number of inode : 4211
> | saved inode           : 4211
> | distribution of inodes
> |  - directories        : 235
> |  - plain files        : 3976
> |  - symbolic links     : 0
> |  - named pipes        : 0
> |  - unix sockets       : 0
> |  - character devices  : 0
> |  - block devices      : 0
> | hard links informations
> |    number of file inode with hard links : 0
> |    total number of hard links           : 0
> | destroyed entries informations
> |    0 file(s) have been record as destroyed since backup of reference
> |
> | Then the user is asked whether to continue listing the archive.
> |
> | In KDar, we don't use the op_listing method, since the full listing is
> | never needed. We use the get_children_of method to build up the
> | archive tree listing on an "on-demand" basis.
>
> Yes, of course.
>
> | It would be nice to have a method for returning the archive info
> | without using the op_listing technique. Do you have a plan for this?
> | Perhaps all it would take is making the "get_header()" call a public
> | member of the archive class, instead of being private? Then each
> | developer can parse
>
> the
>
> | info using their own user_interaction class. Basically, I think the
> | details of how the archive info is presented to the user should be
> | abstracted away from the library and into the client applications,
>
> through
>
> | the user_interaction class.
>
> This should not be a problem. I add it in the "TODO" list.
>

Thanks.

> | Also, it would be useful to have a way to determine if an archive is
> | encrypted besides trial-and-error. Currently, in KDar you have to
>
> remember
>
> | whether an archive is encrypted, and if it is but you don't specify
> | the right encryption algorithm, the archive fails to open with an
> | error. The only way that I can see to make this transparent to the
> | user is for KDar to catch the error, and try opening the archive
> | again, but with a different encryption algorithm.
>
> Well, in the archives there is only a flag telling whether the archive
> has been encrypted or not. There is no information about the algorithm
> used (nor there is about the password :-) sorry just kidding). This is
> intentional. The flag is just used to be able to inform the user that in
> case of wrong encryption or password, dar may return that the archive
> has an incoherent structure. This warning does not shows for clear
> archives.
>
> | From this point of view, perhaps the crypto algorithm argument to the
> | "open" archive constructor is redundant. Couldn't the library query
> | the archive header, see what type of encryption algorithm is used, and
>
> request
>
> | a password through the user_interaction only if encryption is used?
>
> I initially thought that the user should have to record both chosen
> encryption algorithm and chosen password to decrypt a given archive.
> This is the reason why the cipher used is not stored in the archives.
> This makes just a bit more difficult to crack an archive. But you are
> right (assuming you would argue this way) this reason is very weak, and
> for the user, not having to remeber the encryption used is just more
> confortable, while it does not bring any secury problem. So it will be
> changed for user confort. :-)

Thanks! In fact, I agree that witholding the encryption type does not=20
improve the security substantially.

On the other hand, as DAR's author, it's your call. KDar could create a=20
hash of each archive, and keep a database of archives and encryption=20
types. Or, we could use EA to hold that information. Of course, having it=20
in the archive header is easiest for KDar and other clients, so that's=20
where my vote goes. :)

>
> | I realize this possibly involves changes to the api, so I don't expect
> | modifications for the upcoming release. These are some suggestions for
> | future enhancements.
>
> Of course.
>
> | Cheers,
> | JB
>
> Cheers,
> Denis.
>
> P.S.: have you seen kdar on slashdot ? :-)
>

Yeah, I saw KDar on slashdot. I'm quite pleased with the tutorial by Joe=20
Barr. Unfortunately he didn't describe the backup process. Also too bad he=
=20
reviewed the version with the individual file-restore bug, but I take=20
responsibility for that, and for some of the flak that it created. What=20
can I say? It was a regression, and I was enjoying a nice holiday in=20
Italy! :)

Most interesting was the general nature of the comments: "Making backups is=
=20
easy! I just do X, Y, Z (usually including tar...)." It is a "News for=20
nerds" forum, after all; but I suspect that there are plenty of people out=
=20
there who don't make backups for lack of an "easy" way to do it. It's good=
=20
to see that the article generated a lot of discussion about backups,=20
though!

There was also a tutorial on backups, which featured KDar, in the November=
=20
issue of Linux Format (a UK-based magazine). It was quite positive, too.

Cheers,
JB

>
>
> -------------------------------------------------------
> The SF.Net email is sponsored by: Beat the post-holiday blues
> Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
> It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
> _______________________________________________
> Dar-libdar_api mailing list
> Dar...@li...
> https://lists.sourceforge.net/lists/listinfo/dar-libdar_api

=2D-=20
Johnathan K. Burchill, Ph.D.
jk...@us...

[Dar-libdar_api] Re: Archive info method

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Johnathan Burchill wrote:
| Hi Denis,

Hello Johnathan,

|
| I've been thinking for awhile about a way to get info about a dar
archive.
| The 3.0 API returns info via the op_listing method, i.e. archive size, #
| slices, version no., whether it's encrypted, # directories, # regular
| files, etc.
|
| For example, a listing of an archive from the command line gives:
|
| Extracting contents of the archive...
| Archive version format               : 04
| Compression algorithm used           : bzip2
| Scrambling or strong encryption used : yes
| Catalogue size in archive            : 54380 bytes
| Archive is composed of 1 file(s)
| File size: 38243193 bytes
|
| CATALOGUE CONTENTS :
|
| total number of inode : 4211
| saved inode           : 4211
| distribution of inodes
|  - directories        : 235
|  - plain files        : 3976
|  - symbolic links     : 0
|  - named pipes        : 0
|  - unix sockets       : 0
|  - character devices  : 0
|  - block devices      : 0
| hard links informations
|    number of file inode with hard links : 0
|    total number of hard links           : 0
| destroyed entries informations
|    0 file(s) have been record as destroyed since backup of reference
|
| Then the user is asked whether to continue listing the archive.
|
| In KDar, we don't use the op_listing method, since the full listing is
| never needed. We use the get_children_of method to build up the archive
| tree listing on an "on-demand" basis.

Yes, of course.

|
| It would be nice to have a method for returning the archive info without
| using the op_listing technique. Do you have a plan for this? Perhaps all
| it would take is making the "get_header()" call a public member of the
| archive class, instead of being private? Then each developer can parse
the
| info using their own user_interaction class. Basically, I think the
| details of how the archive info is presented to the user should be
| abstracted away from the library and into the client applications,
through
| the user_interaction class.

This should not be a problem. I add it in the "TODO" list.

|
| Also, it would be useful to have a way to determine if an archive is
| encrypted besides trial-and-error. Currently, in KDar you have to
remember
| whether an archive is encrypted, and if it is but you don't specify the
| right encryption algorithm, the archive fails to open with an error.
| The only way that I can see to make this transparent to the user is for
| KDar to catch the error, and try opening the archive again, but with a
| different encryption algorithm.

Well, in the archives there is only a flag telling whether the archive
has been encrypted or not. There is no information about the algorithm
used (nor there is about the password :-) sorry just kidding). This is
intentional. The flag is just used to be able to inform the user that in
case of wrong encryption or password, dar may return that the archive
has an incoherent structure. This warning does not shows for clear archives.

|
| From this point of view, perhaps the crypto algorithm argument to the
| "open" archive constructor is redundant. Couldn't the library query the
| archive header, see what type of encryption algorithm is used, and
request
| a password through the user_interaction only if encryption is used?

I initially thought that the user should have to record both chosen
encryption algorithm and chosen password to decrypt a given archive.
This is the reason why the cipher used is not stored in the archives.
This makes just a bit more difficult to crack an archive. But you are
right (assuming you would argue this way) this reason is very weak, and
for the user, not having to remeber the encryption used is just more
confortable, while it does not bring any secury problem. So it will be
changed for user confort. :-)

|
| I realize this possibly involves changes to the api, so I don't expect
| modifications for the upcoming release. These are some suggestions for
| future enhancements.

Of course.

|
| Cheers,
| JB

Cheers,
Denis.

P.S.: have you seen kdar on slashdot ? :-)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFB7XZspC5CI8gYGlIRAuthAJ4kPVeL5JPvepf2UTjvMcoMae7pMgCgnYuQ
/O1/Hn/lpHuz2YDBF3lcHlY=
=MPUT
-----END PGP SIGNATURE-----

[Dar-libdar_api] Archive info method

[Johnathan B. <jk...@sh...>]

Hi Denis,

I've been thinking for awhile about a way to get info about a dar archive.=
=20
The 3.0 API returns info via the op_listing method, i.e. archive size, #=20
slices, version no., whether it's encrypted, # directories, # regular=20
files, etc.

=46or example, a listing of an archive from the command line gives:

Extracting contents of the archive...
Archive version format               : 04
Compression algorithm used           : bzip2
Scrambling or strong encryption used : yes
Catalogue size in archive            : 54380 bytes
Archive is composed of 1 file(s)
=46ile size: 38243193 bytes

CATALOGUE CONTENTS :

total number of inode : 4211
saved inode           : 4211
distribution of inodes
 - directories        : 235
 - plain files        : 3976
 - symbolic links     : 0
 - named pipes        : 0
 - unix sockets       : 0
 - character devices  : 0
 - block devices      : 0
hard links informations
   number of file inode with hard links : 0
   total number of hard links           : 0
destroyed entries informations
   0 file(s) have been record as destroyed since backup of reference

Then the user is asked whether to continue listing the archive.

In KDar, we don't use the op_listing method, since the full listing is=20
never needed. We use the get_children_of method to build up the archive=20
tree listing on an "on-demand" basis.

It would be nice to have a method for returning the archive info without=20
using the op_listing technique. Do you have a plan for this? Perhaps all=20
it would take is making the "get_header()" call a public member of the=20
archive class, instead of being private? Then each developer can parse the=
=20
info using their own user_interaction class. Basically, I think the=20
details of how the archive info is presented to the user should be=20
abstracted away from the library and into the client applications, through=
=20
the user_interaction class.

Also, it would be useful to have a way to determine if an archive is=20
encrypted besides trial-and-error. Currently, in KDar you have to remember=
=20
whether an archive is encrypted, and if it is but you don't specify the=20
right encryption algorithm, the archive fails to open with an error.
The only way that I can see to make this transparent to the user is for=20
KDar to catch the error, and try opening the archive again, but with a=20
different encryption algorithm.

=46rom this point of view, perhaps the crypto algorithm argument to the=20
"open" archive constructor is redundant. Couldn't the library query the=20
archive header, see what type of encryption algorithm is used, and request=
=20
a password through the user_interaction only if encryption is used?

I realize this possibly involves changes to the api, so I don't expect=20
modifications for the upcoming release. These are some suggestions for=20
future enhancements.

Cheers,
JB
=2D-=20
Johnathan K. Burchill, Ph.D.
Department of Physics and Astronomy
University of Calgary
2500 University Drive N.W.
Calgary, AB T2N 1N4
Canada

(403) 217-4286
jk...@sh...

[Dar-libdar_api] typo fix in API 3

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

I've noticed a bit late that the correct spelling is "cancellation" with
two 'l'. The current CVS is now updated with this change (better late
than never). :-)=)

Cheers,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBtfHjpC5CI8gYGlIRAqPLAJ0eDAxw0+1ULncP1cB/uE+NlWRu4QCfWGOf
1b82aZAYfk2dEo5DzPFr7NE=
=dfCh
-----END PGP SIGNATURE-----

[Dar-libdar_api] Re: Thread cancelation problem

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Johnathan Burchill wrote:
| Hi Denis,

Hello Johnathan,

|
| Sometimes thread cancellation does not work. For example, in KDar if I
test
| a relatively small archive, say a backup of the kdar source directory,
the
| thread cancellation request is ignored by libdar. Sometimes this happens
| with diff and isolate too.
|
| Also, if I do a dry-run (empty) restoration, the cancellation request is
| ignored.
|
| The issue seems lie with the null_file class. I see in going from version
| 1.8 to 1.9 in CVS that you no longer derive null_file from
| thread_cancellation, and you've removed the check_self_cancellation from
| the read and write methods. So if an archive is small enough to be
| completely in memory, then perhaps the thread cancellation check points
| will never be reached.
|
| When I went back to version 1.8 of null_file.hpp libdar responded very
| quickly to all cancellation requests.
|
| Was the change in null_file.hpp intentional? I couldn't find any notes on
| the change in the CVS log.

Yes it was intentional. The reason is that whatever the operation you
do, you alsways read data from a file. There is thus no need to have the
null file doing a second check_self_cancellation as the "fichier" class
used to read each file to backup, or used to read the archive to
test/list/compare/extract (even in dry-run execution) should do the
check regulary when data flow.

If an archive is small enough to be done in memory it must be less than
10 KB. More data will make at least two read/write operations thus at
least one thread cancellation check. In another hand, if the archive is
small, and execution short, thread cancellation may not be very useful.

By the way, I have noted a typo error (cancellation with one 'l') in all
that concern the class for thread cancellation. If you don't mind I will
not fix this mistake now?

The CVS log for null_file.hpp revision 1.9 is uncomplete you are right,
only four actions on five are listed (I usually try to keep it correct
an complete).

|
| I recommend reverting to the version 1.8 behaviour of null_file.hpp.
|

Well, I don't see why the fact to remove the thread cancelation from
null_file avoids thread cancelation to operate even in dry-run mode. But
I have quite not much time actually to spend with dar :-( so I will just
revert to 1.8 as you suggested. Still plane to have pre-release starting
very soon.


| Cheers,
| JB


Cheers,
Denis.

P.S.: CVS is now up to date with this change.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBsQErpC5CI8gYGlIRAsLmAJ467aR+Wo+miRV88IK61XptZ0as5ACdHJSm
ujAOZFzGVF32adLipTAuMHw=
=7upH
-----END PGP SIGNATURE-----

[Dar-libdar_api] Thread cancelation problem

[Johnathan B. <jk...@us...>]

Hi Denis,

Sometimes thread cancellation does not work. For example, in KDar if I test=
=20
a relatively small archive, say a backup of the kdar source directory, the=
=20
thread cancellation request is ignored by libdar. Sometimes this happens=20
with diff and isolate too.

Also, if I do a dry-run (empty) restoration, the cancellation request is=20
ignored.

The issue seems lie with the null_file class. I see in going from version=20
1.8 to 1.9 in CVS that you no longer derive null_file from=20
thread_cancellation, and you've removed the check_self_cancellation from=20
the read and write methods. So if an archive is small enough to be=20
completely in memory, then perhaps the thread cancellation check points=20
will never be reached.

When I went back to version 1.8 of null_file.hpp libdar responded very=20
quickly to all cancellation requests.

Was the change in null_file.hpp intentional? I couldn't find any notes on=20
the change in the CVS log.

I recommend reverting to the version 1.8 behaviour of null_file.hpp.

Cheers,
JB
=2D-=20
Johnathan K. Burchill, Ph.D.
jk...@us...

[Dar-libdar_api] Re: More unused variable warnings in libdar header

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Johnathan Burchill wrote:
| Hi Denis,

Hello Johnathan,

|
| I get these warning messages when compiling KDar:
|
| /usr/local/include/dar/user_interaction.hpp:125: warning: unused
parameter
| ` const std::string&flag'
| /usr/local/include/dar/user_interaction.hpp:125: warning: unused
parameter
| ` const std::string&perm'
| /usr/local/include/dar/user_interaction.hpp:125: warning: unused
parameter
| ` const std::string&uid'
| /usr/local/include/dar/user_interaction.hpp:125: warning: unused
parameter
| ` const std::string&gid'
| /usr/local/include/dar/user_interaction.hpp:125: warning: unused
parameter
| ` const std::string&size'
| /usr/local/include/dar/user_interaction.hpp:125: warning: unused
parameter
| ` const std::string&date'
| /usr/local/include/dar/user_interaction.hpp:125: warning: unused
parameter
| ` const std::string&filename'
| /usr/local/include/dar/user_interaction.hpp:125: warning: unused
parameter
| ` bool is_dir'
| /usr/local/include/dar/user_interaction.hpp:125: warning: unused
parameter
| ` bool has_children'
| linking kdar (libtool)

yes, this is normal, the virtual method "listing()" is implemented by an
empty body in the pure virtual class "user_interaction" (this is expected).

I have added some control code that uses all the arguments, so libdar
will be more explicit (in case a user application calls this method
without having overwritten it in the inherited class) and the C++
compilers should no more complain. :-)

By the way, with which compiler and with which options do you generate
theses warnings ?

|
| Cheers,
| JB

Cheers,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBoReSpC5CI8gYGlIRAhl2AKC+zh1IQJX+sqHTZ5sOPs+/mm1hJACcCg0P
rLLbOF5kGeKQG0aNxvDanMc=
=x6f3
-----END PGP SIGNATURE-----

[Dar-libdar_api] More unused variable warnings in libdar header

[Johnathan B. <jk...@us...>]

Hi Denis,

I get these warning messages when compiling KDar:

/usr/local/include/dar/user_interaction.hpp:125: warning: unused parameter=
=20
` const std::string&flag'
/usr/local/include/dar/user_interaction.hpp:125: warning: unused parameter=
=20
` const std::string&perm'
/usr/local/include/dar/user_interaction.hpp:125: warning: unused parameter=
=20
` const std::string&uid'
/usr/local/include/dar/user_interaction.hpp:125: warning: unused parameter=
=20
` const std::string&gid'
/usr/local/include/dar/user_interaction.hpp:125: warning: unused parameter=
=20
` const std::string&size'
/usr/local/include/dar/user_interaction.hpp:125: warning: unused parameter=
=20
` const std::string&date'
/usr/local/include/dar/user_interaction.hpp:125: warning: unused parameter=
=20
` const std::string&filename'
/usr/local/include/dar/user_interaction.hpp:125: warning: unused parameter=
=20
` bool is_dir'
/usr/local/include/dar/user_interaction.hpp:125: warning: unused parameter=
=20
` bool has_children'
linking kdar (libtool)

Cheers,
JB
=2D-=20
Johnathan K. Burchill, Ph.D.
jk...@us...

[Dar-libdar_api] Re: API documentation in headers?

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Denis Corbin wrote:
| Johnathan Burchill wrote:
|[...]
| |
| | I use it in the KDar source header files, and it would help with
| | development based on libdar to be able to browse the API easily, and see
| | the relationships between classes graphically, etc.
| |
| | If you like, I could send you a Doxyfile that I've modified slightly for
| | libdar.
|
| Yes, why not. However I will have a deep look in doxygen today.
|
|
Here it is, the "day" was a bit long, but that's done. I have documented
the libdar header files (maybe I will also document the command-line
files in dar_suite) and have detailed in particular the API concerned
symbols as much as I could.

Some other fixes you have reported, Johnathan, about MUTEX missing at
installation, and the warning you get compiling KDar has also been
integrated into CVS.

Cheers,
Denis.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBjmz4pC5CI8gYGlIRAvMwAJ4o2LgwWzwcne3KsoEuNQ9Tl8WKcACgjIzo
oCnMEusFvLpTzF3zPufFI+Y=
=QrM7
-----END PGP SIGNATURE-----

[Dar-libdar_api] Re: problem compiling KDar with libdar CVS

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Johnathan Burchill wrote:
| On Sunday 07 November 2004 04:18, Denis Corbin wrote:
|
|[...]
|>Thanks for the compliment :-) However I must check that the execution
|>speed is not too lowered by the simple but much frequent check done to
|>detect thread cancelation requests.
|>
|
|
| Here's a data point for you.
|
| On my Athlon-XP 1800+, 512 MB RAM, I created two archives of the kdar cvs
| source directory, using bzip2 compression. The two archives were created
| with the same settings, the only difference being that one was made with
| libdar 2.5 and the other with libdar 3.0.0 (from KDar). Both operations
| took 32 seconds to complete.

I did some tests too in my side, and I arrive to the same conclusion:
the thread cancelation feature does not introduces significant speed
reduction (I even saw a fit faster execution sometimes by 1% or less,
which may be due to mesure errors margin). :-)

So we keep thread cancelation this way.

|


Cheers,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBjmwNpC5CI8gYGlIRAjgAAKCvG946BIcGxwxKVAn0Ak2p6TwBkQCfRgaI
veSOWtUpQPxFMECB/GBydXU=
=jiln
-----END PGP SIGNATURE-----

[Dar-libdar_api] Re: problem compiling KDar with libdar CVS

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Johnathan Burchill wrote:
| Hi Denis,

Hello Johnathan,

|
|[...]
|>Yes you are right. user_interaction::pause() is only able to handle
|>boolean questions. Thus I had to add a new method to user_interaction
|>class:
|>
|>~   std::string get_string(const std::string & message, bool echo);
|>
|
|
| I'd like to get your opinion on confirming passwords.
|
| When a user needs to enter a password to encrypt a file, I think he/she
| should be asked to confirm what they entered. If the password is for
| decryption, no confirmation should be required.

Yes, that's right, but only when password is given interactively (not
from the command-line of configuration file for example  --- beside the
security aspect they raise).

|
| Is this something that the application or libdar should be concerned
about?
| If the application has to keep track of whether an archive is being
| created or read, that is okay, but means repetitive coding for everyone
| who uses libdar, and more complicated code at the application level. I'd
| rather let libdar decide whether to confirm passwords, with a method like
|
| std::string get_string( const std::string & message, bool echo, bool
| confirm );
|
| What do you think?

The problem here is that get_string() is called by libdar only. So the
application cannot tell libdar this way for a confirmation. I agree that
It may be more simple to make the confirmation code in libdar once and
for all other client programs. So, I would rather add a boolean argument
called "pass_confirm" (for example) to the "create" and "isolate"
archive constructors. This way the application decides whether password
confirmation is needed or not. And libdar will call once, twice or more
time the user_interaction::get_string() method to get two identical
passwords or just one.

|
| [...]
|
| JB
|

Cheers,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBjmqupC5CI8gYGlIRAlrNAJ4jC3w6+5fwhiN99/63pQ1ozIVeaACeIGZt
nthncMOA05GvE58tknElS7Y=
=8Ce5
-----END PGP SIGNATURE-----

Re: [Dar-libdar_api] Re: problem compiling KDar with libdar CVS

[Johnathan B. <jk...@us...>]

Hi Denis,

On Thursday 04 November 2004 10:47, Denis Corbin wrote:
> Johnathan Burchill wrote:
> | On Thursday 04 November 2004 02:27, Denis Corbin wrote:
> |>Johnathan Burchill wrote:
> |
> |[...]
> |
> |>| calls dialog.warning(...) if the archive is encrypted and no
> |>| password is given. Perhaps if "macro_tools_open_archive" could throw
> |>| an Epassword() exception, that could be caught by the application,
> |>| which could transparently ask the user for a password and try the
> |>| read again.
> |>
> |>I like the idea, but throwing an exception is not a reversible
> |> operation ~ from the stack point of view (you cannot continue
> |> execution where it has been thrown). I suppose you mean that libdar
> |> should rather use a the user_interaction::pause(...) method to ask
> |> the password to the user, it the pass argument was something like
> |> "bf:" or "scram:" where just the algorithm (the cypher) is given ?
> |
> | I didn't expect libdar to continue where it left off after issuing
> | Epassword(). Just that it would be a way to inform the application to
> | ask the user for a password and try again to read the archive from the
> | beginning.
> |
> | However, your idea of using user_interaction::pause looks promising. I
>
> did
>
> | not think of that one because the current implementation seems to be
> | able to handle boolean questions, not arbitrary responses.
>
> Yes you are right. user_interaction::pause() is only able to handle
> boolean questions. Thus I had to add a new method to user_interaction
> class:
>
> ~   std::string get_string(const std::string & message, bool echo);
>

I'd like to get your opinion on confirming passwords.

When a user needs to enter a password to encrypt a file, I think he/she=20
should be asked to confirm what they entered. If the password is for=20
decryption, no confirmation should be required.

Is this something that the application or libdar should be concerned about?=
=20
If the application has to keep track of whether an archive is being=20
created or read, that is okay, but means repetitive coding for everyone=20
who uses libdar, and more complicated code at the application level. I'd=20
rather let libdar decide whether to confirm passwords, with a method like

std::string get_string( const std::string & message, bool echo, bool=20
confirm );

What do you think?

[...]

JB

=2D-=20
Johnathan K. Burchill, Ph.D.
jk...@us...

Re: [Dar-libdar_api] Re: problem compiling KDar with libdar CVS

[Johnathan B. <jk...@sh...>]

On Sunday 07 November 2004 04:18, Denis Corbin wrote:
> Johnathan Burchill wrote:
> | Hi Denis,
>
> Hello Johnathan,
>
> | On Wednesday 03 November 2004 13:08, Denis Corbin wrote:
> |>Denis Corbin wrote:
> |>|[...]
> |>| I proposes the following API :
> |>|
> |>| ~    void cancel_thread(pthread_t tid);
> |>| ~    bool cancel_current(pthread_t & tid);
> |>| ~    void cancel_clear();
> |>|
> |>| where cancel_thread() is used to cancel the given thread. Which will
> |>| make libdar throw a Euser_abort exception, cleanly free memory and
> |>| release any mutex.
> |>|
> |>| cancel_current() return true and the tid of the thread that will be
> |>| canceled when execution will reach the next checkpoint, false if
> |>| there is not pending cancelation.
> |>|
> |>| cancel_clear() abort cancelation process if the thread has not
> |>| already terminated.
> |>|
> |>| Only one thread can be canceled at a time.
> |>|
> |>|
> |>|[...]
> |>
> |>Hello,
> |>
> |>[...]
> |
> | I've got KDar running on libdar API V3.0.0, CVS. Works great!
> | The thread cancelation works perfectly, my limited testing shows.
> | The response is very snappy, and the user does not have to wait or
>
> restart
>
> | the program to stop an operation anymore. I think that you have a good
> | implementation of thread cancelation. Quite elegant.
>
> Thanks for the compliment :-) However I must check that the execution
> speed is not too lowered by the simple but much frequent check done to
> detect thread cancelation requests.
>

Here's a data point for you.

On my Athlon-XP 1800+, 512 MB RAM, I created two archives of the kdar cvs 
source directory, using bzip2 compression. The two archives were created 
with the same settings, the only difference being that one was made with 
libdar 2.5 and the other with libdar 3.0.0 (from KDar). Both operations 
took 32 seconds to complete.

Here's the command-line equivalent of the operation:

dar -v -c /opt/user/backups/dar_backups/test-kdar-backup-20041107 
-R /opt/user/src/kdarcvs/kdar/ -w -s 4613734400 -D -y -m 150 -Z "*.Z" -Z 
"*.avi" -Z "*.bz2" -Z "*.gif" -Z "*.gz" -Z "*.jpg" -Z "*.mov" -Z "*.mpg" 
-Z "*.pbm" -Z "*.pdf" -Z "*.png" -Z "*.pnm" -Z "*.zip" -X "*.dar" -X "*~" 
-X ".*~"

When run with the statically-linked dar, here are the results:

API V2.5:
--------------------------------------------
 992 inode(s) saved
 with 0 hard link(s) recorded
 0 inode(s) not saved (no file change)
 0 inode(s) failed to save (filesystem error)
 33 files(s) ignored (excluded by filters)
 0 files(s) recorded as deleted from reference backup
 --------------------------------------------
 --------------------------------------------
 Total number of file considered: 1025

real    0m31.268s
user    0m26.900s
sys     0m0.980s

API V3.0.0 (dev):
--------------------------------------------
 992 inode(s) saved
 with 0 hard link(s) recorded
 0 inode(s) not saved (no file change)
 0 inode(s) failed to save (filesystem error)
 33 files(s) ignored (excluded by filters)
 0 files(s) recorded as deleted from reference backup
 --------------------------------------------
 --------------------------------------------
 Total number of file considered: 1025

real    0m31.703s
user    0m27.170s
sys     0m0.850s

These are not rigorous benchmarks. In particular, the thread cancellation 
overhead might be compensated for some other speed improvement in API 3.0 
compared to API 2.5. Anyway, for what it's worth...

Cheers,
JB

> | One point about libdar's "make install": the config.h header that gets 
> | put 
> | into <installdir>/include/dar/ does not contain "#define MUTEX_WORKS
> | 1" when I configure libdar with pthreads, so I have to add it manually
> | to get KDar to compile.
>
> OK, I will fix that. Thanks for feedback.
>
> | [...]
> |
> | Cheers,
> | JB
>
> Cheers,
> Denis.
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Sybase ASE Linux Express Edition - download now for FREE
> LinuxWorld Reader's Choice Award Winner for best database on Linux.
> http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
> _______________________________________________
> Dar-libdar_api mailing list
> Dar...@li...
> https://lists.sourceforge.net/lists/listinfo/dar-libdar_api

-- 
Johnathan K. Burchill, Ph.D.
Department of Physics and Astronomy
University of Calgary
2500 University Drive N.W.
Calgary, AB T2N 1N4
Canada

(403) 217-4286
jk...@sh...

[Dar-libdar_api] Re: problem compiling KDar with libdar CVS

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Johnathan Burchill wrote:
| Hi Denis,

Hello Johnathan,

|
| On Wednesday 03 November 2004 13:08, Denis Corbin wrote:
|
|>Denis Corbin wrote:
|>|[...]
|>| I proposes the following API :
|>|
|>| ~    void cancel_thread(pthread_t tid);
|>| ~    bool cancel_current(pthread_t & tid);
|>| ~    void cancel_clear();
|>|
|>| where cancel_thread() is used to cancel the given thread. Which will
|>| make libdar throw a Euser_abort exception, cleanly free memory and
|>| release any mutex.
|>|
|>| cancel_current() return true and the tid of the thread that will be
|>| canceled when execution will reach the next checkpoint, false if there
|>| is not pending cancelation.
|>|
|>| cancel_clear() abort cancelation process if the thread has not already
|>| terminated.
|>|
|>| Only one thread can be canceled at a time.
|>|
|>|
|>|[...]
|>
|>Hello,
|>
|>[...]
|
|
| I've got KDar running on libdar API V3.0.0, CVS. Works great!
| The thread cancelation works perfectly, my limited testing shows.
| The response is very snappy, and the user does not have to wait or
restart
| the program to stop an operation anymore. I think that you have a good
| implementation of thread cancelation. Quite elegant.

Thanks for the compliment :-) However I must check that the execution
speed is not too lowered by the simple but much frequent check done to
detect thread cancelation requests.

|
| One point about libdar's "make install": the config.h header that gets
put
| into <installdir>/include/dar/ does not contain "#define MUTEX_WORKS 1"
| when I configure libdar with pthreads, so I have to add it manually to
get
| KDar to compile.

OK, I will fix that. Thanks for feedback.

|
| [...]
|
| Cheers,
| JB
|

Cheers,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBjgRvpC5CI8gYGlIRAmzaAJ98wVAgXkOHSSfbIZi0jcXmi+4ikQCgr++D
hCA1/6UqrHaFE8+vywcbO74=
=HMOb
-----END PGP SIGNATURE-----