dar-libdar_api — discussion area for developpers that want to use the libdar API

[Dar-libdar_api] last archive slice

[Johnathan B. <jk...@us...>]

Hi Denis,

In dar version 2.1.3, libdar mistakes the last slice number sometimes.

=46or example, create an archive with slice sizes of 3 MB. Say this results=
=20
in 6 slices. Now create the archive, same basename, same directory, and=20
allow the library to overwrite the existing archive, but use 5 MB slices.
Suppose it results in 3 slices.

Now try to list the archive. I get the following user interaction question:

"/opt/user/backups/dar_backups/test6.6.dar is a file from another set of=20
backup file, please provide the correct file."

Then the user has to delete slices, one by one starting with the highest=20
number, and retry the list command until it works.

Is this the expected behaviour? Shouldn't libdar see that the third slice,=
=20
in this example, would be the last archive slice? I have to admit, it's=20
confusing for a user who does this to know that the archive consists of=20
only three slices when six are on disk.

You wouldn't have to change the slice size to get this problem. Suppose you=
=20
did a full backup and got 12 CD-R-sized slices. Then during the course of=20
the following week, you delete a bunch of directories that you don't need=20
anymore, and do a full backup which has 11 CD-R-sized slices. Dar will=20
miss the fact that the last slice  #11, but will still ask the user for=20
the 12th slice.

I see the following solutions:

1) libdar deletes all slices on a disk if they have the same storage=20
directory and basename as the new one being created. It might do this=20
before creating the archive, or after creating the archive. Perhaps doing=20
this after will be more efficient, since you only delete slices that=20
haven't been overwritten, if there are any.

2) The search algorithm gets fixed to stop on the first slice with a "T"=20
type terminating character in the header. Am I correct in interpreting the=
=20
"T" to mean that the slice is the last one in the archive? I assume that=20
libdar figures out the last slice by relying on the filename, not the=20
header, although I haven't checked the sourcecode.

3) A combination of 1) and 2).

I see a similar problem when creating archives with KDar. I have a progress=
=20
bar that periodically updates which slice is currently being written, and=20
what that slice's size is. When the "pause" between slices option is=20
chosen, and libdar asks if it okay to continue writing the next slice,=20
first it writes the "non-terminating" character to the slice header, and=20
then waits for the user to answer the question. In the case where the user=
=20
is overwriting an older archive, the statusbar code runs through the files=
=20
on disk until it finds the one with the "T" in the header. This will be=20
the last one in the older archive, and the statusbar will jump to that=20
slice instead of the current one.=20

Perhaps libdar should write the "N" to the termination type only if the=20
user says it's okay to continue, and write an "I" to the termination type=20
if the user cancels. That way we know that the "last" archive slice is=20
actually invalid, i.e. the creation process was aborted, or failed in some=
=20
way.

=46or the status indicator, the best solution would be for libdar to have a=
=20
"currentSlice" method, which could be called at any time to determine the=20
slice number of the current one being written or read.

Should I file a bug report for any of this?

Cheers,
JB

=2D-=20
Johnathan K. Burchill, Ph.D.
jk...@us...

Re: [Dar-libdar_api] Re: archive methods?

[Johnathan B. <jk...@sh...>]

On July 17, 2004 09:46 am, Denis Corbin wrote:
> Christian Neumann wrote:
> | On Sat, 17 Jul 2004 16:22:50 +0200
> |
> | Denis Corbin <dar...@fr...> wrote:
> |>any objections ? comments ? ;-)
> |
> | My two cents: I would prefer a more object oriented API, because OO
> | programming is more convenient.. Of course it would be difficult for a
> | C programmer to understand the API, but at last it's a C++ library and
> | not a C library. And you can't use libdar in C programs, can you
> | (correct me if I'm wrong)? Non-OO C++ is just worse than plain C
> | (IMHO).
>
> that makes sens. So it we will go this way.
>

Sounds good. :!

JB

> | Regards,
> | Christian
>
> Cheers,
> Denis.
>
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by BEA Weblogic Workshop
> FREE Java Enterprise J2EE developer tools!
> Get your free copy of BEA WebLogic Workshop 8.1 today.
> http://ads.osdn.com/?ad_id=3D4721&alloc_id=3D10040&op=3Dclick
> _______________________________________________
> Dar-libdar_api mailing list
> Dar...@li...
> https://lists.sourceforge.net/lists/listinfo/dar-libdar_api

=2D-=20
Johnathan K. Burchill, Ph.D.
Department of Physics and Astronomy
University of Calgary
2500 University Drive N.W.
Calgary, AB T2N 1N4
Canada

(403) 217-4286
jk...@sh...

[Dar-libdar_api] Re: archive methods?

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Christian Neumann wrote:
| On Sat, 17 Jul 2004 16:22:50 +0200
| Denis Corbin <dar...@fr...> wrote:
|
|
|>any objections ? comments ? ;-)
|
| My two cents: I would prefer a more object oriented API, because OO
| programming is more convenient.. Of course it would be difficult for a C
| programmer to understand the API, but at last it's a C++ library and not
| a C library. And you can't use libdar in C programs, can you (correct me
| if I'm wrong)? Non-OO C++ is just worse than plain C (IMHO).

that makes sens. So it we will go this way.

|
| Regards,
| Christian

Cheers,
Denis.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFA+Um8pC5CI8gYGlIRAvwBAJ44il9f3jYMv9U4RW08B9CaM8PTdQCgrf1h
D9fvinB5BUK0IjOEUT8iSHY=
=F0Dc
-----END PGP SIGNATURE-----

Re: [Dar-libdar_api] Re: archive methods?

[Christian N. <chr...@al...>]

On Sat, 17 Jul 2004 16:22:50 +0200
Denis Corbin <dar...@fr...> wrote:

> any objections ? comments ? ;-)
My two cents: I would prefer a more object oriented API, because OO
programming is more convenient.. Of course it would be difficult for a C
programmer to understand the API, but at last it's a C++ library and not
a C library. And you can't use libdar in C programs, can you (correct me
if I'm wrong)? Non-OO C++ is just worse than plain C (IMHO).

Regards,
Christian

[Dar-libdar_api] Re: archive methods?

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Johnathan Burchill wrote:
| Hi Denis,

Hello Johnathan,

|
| Now that libdar uses the "archive" class, what is the possibility of
having
| the various libdar operations implemented as methods of the archive class?
|
| Instead of
|
| libdar::archive *theArchive;
|
| libdar::open( theArchive, ...)
| libdar::test( theArchive, ...)
| etc.
|
| what about having
|
| libdar::archive *theArchive( path, basename, etc.)
|
| theArchive->open();
| theArchive->test();
|
| Have you considered this?

yes, I did. I even intially wanted to implement things this way.

| I'm not sure what this would accomplish, besides
| quite a bit of possibly unnecessary code rewriting.

I don't think there is too much code rewriting for that be possible.

| It just ocurred to me
| that an archive is the main object for libdar, and the usual way in
c++ is
| to implement functions as methods.

yes, I agree.

|
| Please enlighten me if I am off base here. I'm not sure that I've thought
| this through very much. Sorry in advance if this has been documented
| elsewhere.

In fact, if I did not do this way, is to keep API a set of normal
functions, for those who could call libdar from a program written in C,
Pascal, or any other language. Of course some of the arguments are
objects. But, for a non C++ programmer it is maybe more intuitive to use
a pure C function than a method on an object.

This is the reason why I did not moved op_create and other methods as
members of archive class.

Another problem is the _noexcept variants of API calls. In particular
the open_archive_noexcept() call which would correspond to a constructor
for the archive class. constructor do not return error, so if it is not
possible to throw exception to report an error, how would it be possible
to report an error during the archive object construction ? OK, one
could make a fake constructor, that do nothing real, and a
open_archive_noexcept() method that (as it is now a normal method), can
return an error code status, but, I don't see much more interest
compared to normal functions.

any objections ? comments ? ;-)

|
| Cheers,
| JB
|

Cheers,
Denis.

P.S.:
Johnathan, I cannot figure out how to find your public key. It is not on
your web site (there is old obsolete one only), nor it is on any key
servers I tried.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFA+TY6pC5CI8gYGlIRAiwaAJ9y6KrSZRM1YOEtjzE+WKz8hqXQkQCeI4GW
lZj52GTaJKQ3gXrF7xT9uBI=
=I0Tl
-----END PGP SIGNATURE-----

[Dar-libdar_api] archive methods?

[Johnathan B. <jk...@us...>]

Hi Denis,

Now that libdar uses the "archive" class, what is the possibility of having=
=20
the various libdar operations implemented as methods of the archive class?

Instead of

libdar::archive *theArchive;

libdar::open( theArchive, ...)
libdar::test( theArchive, ...)
etc.

what about having

libdar::archive *theArchive( path, basename, etc.)

theArchive->open();
theArchive->test();

Have you considered this? I'm not sure what this would accomplish, besides=
=20
quite a bit of possibly unnecessary code rewriting. It just ocurred to me=20
that an archive is the main object for libdar, and the usual way in c++ is=
=20
to implement functions as methods.

Please enlighten me if I am off base here. I'm not sure that I've thought=20
this through very much. Sorry in advance if this has been documented=20
elsewhere.

Cheers,
JB

=2D-=20
Johnathan K. Burchill, Ph.D.
jk...@us...

[Dar-libdar_api] Re: which mask to use to prune a directory

[Denis C. <dar...@fr...>]

Denis Corbin wrote:
> Christian Neumann wrote:
> 
>[...]

> 
> I will look at this problem and fix it in CVS.

things are fixed in CVS in the following way.

for the original unordered mask construction, the

	not simple_mask(<argument>)

construction is used when -P <argument> is given on command line. While 
in the ordered mask construction, the following mask is built:

	not ( simple_mask(<argument>) or simple_mask(<argument>/*) )

This seems to work properly so far.

Cheers,
Denis.

[Dar-libdar_api] Re: which mask to use to prune a directory

[Denis C. <dar...@fr...>]

Christian Neumann wrote:
> Hi,

Hello,

> do the simple_path_mask or the exclude_dir_mask support wildcards
> (?/*)? As far as I understood the libdar code, they do not. But as
> dar's -P option supports wildcards...

exclude_dir_mask matches any file that is a subdir or equal to the mask 
string. No wildcard are available for that type of mask.

simple_path_mask matches any file that is a subdir or the mask of any 
file for which mask is a subdir. No wildcard are available for that type 
of mask.

simple_mask implements wildcard mask like on the command-line (glob 
expression).

 > Which mask(s) uses dar to
> implement the-P option? I thought of using two simple_masks (<dir> and
> <dir>/*), but I am not sure if this is the right way. 

well actually in CVS we have:
-g option uses simple_path_mask
-P option uses exclude_dir_mask
-I option uses simple_mask
-X option uses simple_mask

Note: -g is new and is equivalent to [list of paths] the opposit of -P

But in last release, we had:

[list of paths] used simple_path_mask
-P option used simple_mask
-I option uses simple_mask
-X option uses simple_mask

note for previous release which behaves properly, the -P <string> on 
command line, only implies the construction of simple_mask(<string>). 
This is not necessary to add a simple_mask(<string>/*) because as soon a 
directory is excluded no recursion is done in it.

it seems I have made a mistake while implementing the ordered mask 
feature (-am option), substituing simple_mask by exclude_dir_mask for -P 
option. This has some reasons. When doing unordered mask construction 
the correct mask for -P is simple_mask. While when doing ordered mask 
construction using simple_mask for -P brings some new problems:

for example: -P "usr/local" -g "usr/local/lib"

using ordered mask (-am option) it would build the following mask:

simple_path_mask("usr/local/lib") or not simple_mask("usr/local")

the "usr/local/man" for example, would then wrongly be included for 
saving because "usr/local/man" neither matches "usr/local/lib" nor 
"usr/local". Instead in original unordered mask construction, we would 
have got:

simple_path_mask("usr/local/lib") and not simple_mask("usr/local")

scanning the directory we would first reach "usr/local" which would 
match the second mask and make all this subdirectory tree be excluded 
(eventually being replaced by an empty directory in the archive if -D
was used), with no more recursion in it.

In ordered mask construction, the "usr/local" is first matched by 
simple_path_mask("usr/local/lib") because "usr/local/lib" is a subdir of 
  "usr/local". So never the [not simple("usr/local")] expression will match.

I will look at this problem and fix it in CVS.


> And does the -D
> option work with this solution? Also, is there any way to include
> wildcarded dirs (like include only config dirs: ".*") (I know that the
> parent dirs up to the root path have to be included. Any convenient
> solution?).

the -D option is independant of the type of masked used, as soon a 
directory is excluded by a filter and -D option is used, the directory 
it subsitued by an empty dir in the archive.


> 
> Regards,
> Christian
> 

Thanks for feedback,

Cheers,
Denis.

[Dar-libdar_api] which mask to use to prune a directory

[Christian N. <chr...@al...>]

Hi,
do the simple_path_mask or the exclude_dir_mask support wildcards
(?/*)? As far as I understood the libdar code, they do not. But as
dar's -P option supports wildcards... Which mask(s) uses dar to
implement the-P option? I thought of using two simple_masks (<dir> and
<dir>/*), but I am not sure if this is the right way. And does the -D
option work with this solution? Also, is there any way to include
wildcarded dirs (like include only config dirs: ".*") (I know that the
parent dirs up to the root path have to be included. Any convenient
solution?).

Regards,
Christian

[Dar-libdar_api] Re: listing callback

[Denis C. <dar...@fr...>]

Denis Corbin wrote:
> Christian Neumann wrote:
> 
>> Hi,
> 
> 
> Hello,
>[...]
> 
> from the listing callback you can know if a file is a directory from the 
> permission, but OK, this require parsing. So I will add the two 
> arguments you ask.
> 

This is now implemented, you can check it out from CVS. ;-)

>[...]

Cheers,
Denis.

[Dar-libdar_api] Re: listing callback

[Denis C. <dar...@fr...>]

Christian Neumann wrote:
> Hi,

Hello,

> it would be useful if the listing callback had a <bool has_children>
> and/or <bool is_directory> parameter, so that the program could show the
> user that some rows of the file/directory list have children (which can
> be optionally listed by clicking on them (the program would then call a
> get_children_of(the row))).

from the listing callback you can know if a file is a directory from the 
permission, but OK, this require parsing. So I will add the two 
arguments you ask.

Thanks for feedback,
> 
> Regards,
> Christian

Denis.

> 
> 
> -------------------------------------------------------
> This SF.Net email sponsored by Black Hat Briefings & Training.
> Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
> digital self defense, top technical experts, no vendor pitches, 
> unmatched networking opportunities. Visit www.blackhat.com

[Dar-libdar_api] Re: libdar polling, KDar forking...

[Denis C. <dar...@fr...>]

Johnathan Burchill wrote:
> I talked with a programmer friend last night (Mark) about interrupting 
> threads. A real programming challenge it seems. The apparent way to go is 
> to poll. In fact, the pthread_cancel() function, and the ENABLE and 
> DISABLE flags are a type of polling. The libdar would have to have polling 
> points to make it work. Mark explained that the stigma of polling comes 
> from asynchronous communication, as an example. But the real problem is to 
> ascertain how long you are willing to wait between polls. 

OK, I see.

> 
> As an example, suppose you have a process that loops through a list of 
> files, compressing each in turn. Let's say the program polls the outside 
> world once per iteration to see if it should cancel the operation. If the 
> compression time is small, then the polling interval is small, and the 
> cancel operation takes effect almost immediately. 
> 
> If, on the other hand, the compression takes a long time, say 30 seconds to 
> possibly several minutes for a large file or a slow computer, then the 
> user will have to wait that time before the execution polls again. This 
> might be an unacceptably long delay to cancel the operation.
> 
> In either case, most of the loop is involved in doing real work (the 
> compression in this case) with a very small part devoted to polling. 
> 
> So a tradeoff must be made between how complicated one wants to get in the 
> code --- perhaps by implementing the compression algorithm directly and 
> adding new polling points --- and how much delay is acceptable for things 
> like cancelling the file compression operation. If users typically 
> compress only small files, then the delay will be minimal and acceptable. 
> But if they compress large files, the delay associated with cancelling the 
> operation can be a problem. 

With night ideas come. Your description makes me thing about another 
possibility. I think in all operations done by dar, there is readings 
and writings to files (even to virtual files when running in dry-run 
mode). So instead of polling at each file boundary, the polling could be 
done at each read() or write() time. As the file I/O operation are 
wrapped in class,  --- in fact it is a stack of class, one compressing, 
another scrambling, yet another slicing, --- I could add a new class 
that has the only effect to poll a certain variable for cancelation 
while transparently transmitting the data to the upper or lower class in 
the stack. This class could take place near the bottom of this stack, 
and if a cancelation is seen, it would throw an Euser_abort exception. 
The use of exception here would have the advantage to free up allocated 
memory along the exception path.

>[...]
> 
> Another way would be to fork the process and either exec a new program 
> dedicated to the task, or just run the copy of the original process. 
> Communications between the parent and child could be established through a 
> pipe or shared memory. Then if the user cancels the operation, we just 
> kill the process, and don't worry a bit about dangling threads, locked 
> mutexes, and the lot. The advantage of this technique is that we are 
> guaranteed to kill the process quickly and get control back to the user. 
> At least as quickly as a <Ctrl>-C on the command-line.

yes, I agree.

> 
> The issue of whether threads are better than forked processes depends on 
> the application. In the case of the backup program, we might typically 
> just run a backup, or a restore, by itself. I could see where we are even 
> testing one archive while diffing another one. But there won't be more 
> than several libdar operations going on at once. So the extra overhead 
> associated with forking a kdar is not really a problem.
> 
> If I decided to go with the fork technique, I wouldn't have to worry about 
> how you implement the cancelling operation in libdar, and so our lines of 
> development could in principle be carried out independently from that 
> point of view.

yes.

> 
> I have to get educated on forking before I make the plunge into those 
> waters, but it may be a few weeks as I have some jobs to apply for and 
> other work-related priorities that will eat at my spare time. But I will 
> try to get the cancel operation implemented because it is important not to 
> leave the user "hanging" and frustrated when they want to stop an 
> operation.

Forking or not forking...  :-)

to resume, either you fork and have easy cancelation, but will have 
overhead in development (kdar only) and program execution about 
returning information about the processing job, or you keep threads 
which implies extra-development (libdar and kdar) to have 
cancellation,and keep easy user feedback about the running task.

I like the second one but I will require more time to be available to 
users. So I suggest one could start with a forking kdar waiting for a 
libdar that allows thread cancellation.

About spare time I have the same restriction actually, at least up to 
the end of july. So I will concentrate on the many little tasks I have 
to do in dar, as it is not easy to have a global sight of what you do 
when your work is interrupted too much time and get done over too long 
period.

> 
> JB


Denis.

[Dar-libdar_api] Re: [Dar-discussions] Re: Fwd: I would like to cancel archive creation

[Denis C. <dar...@fr...>]

Johnathan Burchill wrote:
> On June 28, 2004 03:02 pm, Johnathan Burchill wrote:
> 
>>>By the way, note that the mutex used by libdar (and common to all
>>>threads) are never released. If this is a problem it might be possible
>>>to add a call in the API to release them and in consequence forbid any
>>>further call to libdar. The mutex used do not take much memory place
>>>however.
>>
>>In other words, the mutexes are created on the stack?

in fact the mutex is a static variable of the special_alloc.cpp module 
it is pthread_mutex_init() initialized, but never pthread_mutex_destroy()ed.

>>
> 
> Sorry, I meant the heap.

Sure. Here if I don't make any mistake the mutex is created in the data 
segment, but the pthread_mutex_init may use the heap to activate the 
mutex...

> 
> JB

[Dar-libdar_api] listing callback

[Christian N. <chr...@al...>]

Hi,
it would be useful if the listing callback had a <bool has_children>
and/or <bool is_directory> parameter, so that the program could show the
user that some rows of the file/directory list have children (which can
be optionally listed by clicking on them (the program would then call a
get_children_of(the row))).

Regards,
Christian

[Dar-libdar_api] libdar polling, KDar forking...

[Johnathan B. <jk...@us...>]

I talked with a programmer friend last night (Mark) about interrupting=20
threads. A real programming challenge it seems. The apparent way to go is=20
to poll. In fact, the pthread_cancel() function, and the ENABLE and=20
DISABLE flags are a type of polling. The libdar would have to have polling=
=20
points to make it work. Mark explained that the stigma of polling comes=20
from asynchronous communication, as an example. But the real problem is to=
=20
ascertain how long you are willing to wait between polls.=20

As an example, suppose you have a process that loops through a list of=20
files, compressing each in turn. Let's say the program polls the outside=20
world once per iteration to see if it should cancel the operation. If the=20
compression time is small, then the polling interval is small, and the=20
cancel operation takes effect almost immediately.=20

If, on the other hand, the compression takes a long time, say 30 seconds to=
=20
possibly several minutes for a large file or a slow computer, then the=20
user will have to wait that time before the execution polls again. This=20
might be an unacceptably long delay to cancel the operation.

In either case, most of the loop is involved in doing real work (the=20
compression in this case) with a very small part devoted to polling.=20

So a tradeoff must be made between how complicated one wants to get in the=
=20
code --- perhaps by implementing the compression algorithm directly and=20
adding new polling points --- and how much delay is acceptable for things=20
like cancelling the file compression operation. If users typically=20
compress only small files, then the delay will be minimal and acceptable.=20
But if they compress large files, the delay associated with cancelling the=
=20
operation can be a problem.=20

=46or most KDar users, we might get away with a quick response most of the=
=20
time, and then occasionally one tries to cancel during an extended period=20
without a poll, and a few minute wait happens. This might be a trade-off=20
that we could live with, in which case I would continue to call the libdar=
=20
operations in separate threads.

One way to get around the problem is to call the thread_cancel function,=20
and immediately return control to the user, ignoring the fact that the=20
thread may still be running, and just let it finish on its own time. This=20
can be undesirable because the cancelled thread may take some considerable=
=20
time to stop, and will still potentially waste a lot of CPU cycles.=20
However, it may be the case that most of the time the operation stops=20
quickly and we're okay.

Another way would be to fork the process and either exec a new program=20
dedicated to the task, or just run the copy of the original process.=20
Communications between the parent and child could be established through a=
=20
pipe or shared memory. Then if the user cancels the operation, we just=20
kill the process, and don't worry a bit about dangling threads, locked=20
mutexes, and the lot. The advantage of this technique is that we are=20
guaranteed to kill the process quickly and get control back to the user.=20
At least as quickly as a <Ctrl>-C on the command-line.

The issue of whether threads are better than forked processes depends on=20
the application. In the case of the backup program, we might typically=20
just run a backup, or a restore, by itself. I could see where we are even=20
testing one archive while diffing another one. But there won't be more=20
than several libdar operations going on at once. So the extra overhead=20
associated with forking a kdar is not really a problem.

If I decided to go with the fork technique, I wouldn't have to worry about=
=20
how you implement the cancelling operation in libdar, and so our lines of=20
development could in principle be carried out independently from that=20
point of view.

I have to get educated on forking before I make the plunge into those=20
waters, but it may be a few weeks as I have some jobs to apply for and=20
other work-related priorities that will eat at my spare time. But I will=20
try to get the cancel operation implemented because it is important not to=
=20
leave the user "hanging" and frustrated when they want to stop an=20
operation.

JB
=2D-=20
Johnathan K. Burchill, Ph.D.
jk...@us...

Re: [Dar-libdar_api] Re: [Dar-discussions] Re: Fwd: I would like to cancel archive creation

[Johnathan B. <jk...@us...>]

On June 28, 2004 03:02 pm, Johnathan Burchill wrote:
> >
> > By the way, note that the mutex used by libdar (and common to all
> > threads) are never released. If this is a problem it might be possible
> > to add a call in the API to release them and in consequence forbid any
> > further call to libdar. The mutex used do not take much memory place
> > however.
>
> In other words, the mutexes are created on the stack?
>
Sorry, I meant the heap.

JB
=2D-=20
Johnathan K. Burchill, Ph.D.
jk...@us...

Re: [Dar-libdar_api] Re: [Dar-discussions] Re: Fwd: I would like to cancel archive creation

[Johnathan B. <jk...@sh...>]

On June 28, 2004 01:29 pm, Denis Corbin wrote:
> Johnathan Burchill wrote:
> > On June 19, 2004 02:43 pm, Denis Corbin wrote:
> >>Johnathan Burchill wrote:
> >>>On June 19, 2004 09:59 am, Johnathan Burchill wrote:
> >
> >[...]
> >
> >>>Any comments?
> >>
> >>What standard mechanism over signal handlers can preempt a thread and
> >>let it do some arbitrary action ? I don't like polling. :-/
> >
> > Signal handlers are based on polling, right? When you say polling, do
> > you mean that within your libdar operation loop a conditional
> > statement checks to see if a signal has been set?
>
> no signal handlers are not based on polling: you associate a function
> (=3Dhandler) to a signal thanks to the signal() system call and when a
> process receive the given signal (either from the system or from another
> process thanks to the kill() system call) the given handler is run, and
> once finished the execution of the process continues transparently,
> unless the handler stops the process (thanks to the exit() standard call
> and so on).
>
> > What are the problems with polling? I am not familiar with the issues.
> > Portability? CPU cycles? ...?
>
> I don't like it because it makes algorithm more complex and less
> performant due to CPU cycle wasted checking someting has not changed
> most of the time.
>
> > I would suggest a public variable that can be set by the calling
> > thread,
> >
> > bool stop( false );
> >
> > and in the definition,
> >
> > while (...) //libdar::operation loop
> > {
> > 	if ( !stop )
> > 	{
> > 		...libdar operations...
> > 	}
> > 	else
> > 	{
> > 		...clean up premature stop...
> > 		...break out of operation loop
> > 	}
> > }
>
> I seems we must forget the signal mechanism, as described here :
> http://www.burningvoid.com/iaq/posix-signals.html
> "One important safety tip is that pthreads constructs like mutexes,"
> "condition variables, etc. do not have defined behavior inside signal"
> "handlers. So you cannot use pthreads calls or synchronize with other"
> "threads while inside a signal handler."
>
> > Although this might present a problem when, say, you are testing two
> > separate archives at the same time, but you decide to cancel just one
> > of them. Instead, you could add an additional argument to the libdar
> > call:
> >
> > libdar::statistics st =3D libdar::op_test( theArchive, ... other
> > arguments ..., (bool *) stop );
> >
> > The (bool *) points to a protected or private member variable of the
> > calling thread, so we could cancel one operation, keeping others
> > going.
>
> I would prefer another direction. (reference:
> http://www.llnl.gov/computing/tutorials/workshops/workshop/pthreads/MAIN
>.html )
>   Based on the information found in the previous web site, I would
> prefer the following mechanism:
>
> before entering a critical section (bounded by a mutex) in libdar, set
> the cancel state to DISABLE thanks to the pthread_setcancelstate() call.
>
> after exiting a critical section (still in libdar) set the cancel state
> back to ENABLE.
>
> If a multi threaded program wants to cancel a thead while calling a
> libdar routine, it would just need to call the pthread_cancel() routine
> on the corresponding thread.
>
> If the libdar code is out of a critical section the thread exits
> immediatly, if it is inside a critical section the thread exits once
> cancel state is enabled back after the critical section.

=46rom the man page for pthread_cancel(), it seems that with the  cancel=20
state set to DISABLE, the thread will ignore the cancel call, not defer=20
it. So it would be hit-and-miss with a user trying to cancle the thread,=20
depending on whether the thread is in some critical part or not.

To defer the cancel call, you set the cancel type to defer. Then your code=
=20
tests for the cancel signal at various points in the non-critical parts,=20
the so-called "cancel points".

Do I interpret that man page correctly?

>
> This way the global mutex objects stay in coherent state and thus it is
> possible to safely cancel a thread calling libdar routine.
>
> By the way, note that the mutex used by libdar (and common to all
> threads) are never released. If this is a problem it might be possible
> to add a call in the API to release them and in consequence forbid any
> further call to libdar. The mutex used do not take much memory place
> however.
>

In other words, the mutexes are created on the stack?

>  >[...]
> >
> > JB
>
> What's your objections if any ? ;-)

Just so I understand this, would I call pthread_create, with the name
of the libdar operation function as an argument? Or have you implemented=20
the libdar operations now as threads?

>
> I have at least one for now : it does not handle the allocated memory
> from the thread by libdar.

According to the pthread_cancel manpage,=20

       When a thread eventually honors a cancellation request, it
       performs as  if  pthread_exit(PTHREAD_CANCELED)  has  been
       called at that point: all cleanup handlers are executed in
       reverse order, finalization functions for  thread-specific
       data  are  called,  and finally the thread stops executing
       with   the    return    value    PTHREAD_CANCELED.     See
       pthread_exit(3) for more information.

Could the cleanup handlers handle the allocated memory?

>
> Cheers,
> Denis.
>

Cheers,
JB
=2D-=20
Johnathan K. Burchill, Ph.D.
Department of Physics and Astronomy
University of Calgary
2500 University Drive N.W.
Calgary, AB T2N 1N4
Canada

(403) 217-4286
jk...@sh...

[Dar-libdar_api] Re: [Dar-discussions] Re: Fwd: I would like to cancel archive creation

[Denis C. <dar...@fr...>]

Johnathan Burchill wrote:
> On June 19, 2004 02:43 pm, Denis Corbin wrote:
> 
>>Johnathan Burchill wrote:
>>
>>>On June 19, 2004 09:59 am, Johnathan Burchill wrote:
>>>
>[...]
>>
>>>Any comments?
>>
>>What standard mechanism over signal handlers can preempt a thread and
>>let it do some arbitrary action ? I don't like polling. :-/
>>
> 
> 
> Signal handlers are based on polling, right? When you say polling, do you 
> mean that within your libdar operation loop a conditional statement checks 
> to see if a signal has been set?

no signal handlers are not based on polling: you associate a function 
(=handler) to a signal thanks to the signal() system call and when a 
process receive the given signal (either from the system or from another 
process thanks to the kill() system call) the given handler is run, and 
once finished the execution of the process continues transparently, 
unless the handler stops the process (thanks to the exit() standard call 
and so on).

> 
> What are the problems with polling? I am not familiar with the issues. 
> Portability? CPU cycles? ...?

I don't like it because it makes algorithm more complex and less 
performant due to CPU cycle wasted checking someting has not changed 
most of the time.

> 
> I would suggest a public variable that can be set by the calling thread,
> 
> bool stop( false );
> 
> and in the definition, 
> 
> while (...) //libdar::operation loop
> {
> 	if ( !stop )
> 	{
> 		...libdar operations...
> 	}
> 	else
> 	{
> 		...clean up premature stop...
> 		...break out of operation loop
> 	}
> }
> 

I seems we must forget the signal mechanism, as described here : 
http://www.burningvoid.com/iaq/posix-signals.html
"One important safety tip is that pthreads constructs like mutexes,"
"condition variables, etc. do not have defined behavior inside signal"
"handlers. So you cannot use pthreads calls or synchronize with other" 
"threads while inside a signal handler."

> Although this might present a problem when, say, you are testing two 
> separate archives at the same time, but you decide to cancel just one of 
> them. Instead, you could add an additional argument to the libdar call:
> 
> libdar::statistics st = libdar::op_test( theArchive, ... other 
> arguments ..., (bool *) stop );
> 
> The (bool *) points to a protected or private member variable of the 
> calling thread, so we could cancel one operation, keeping others going.

I would prefer another direction. (reference: 
http://www.llnl.gov/computing/tutorials/workshops/workshop/pthreads/MAIN.html 
)
  Based on the information found in the previous web site, I would 
prefer the following mechanism:

before entering a critical section (bounded by a mutex) in libdar, set 
the cancel state to DISABLE thanks to the pthread_setcancelstate() call.

after exiting a critical section (still in libdar) set the cancel state 
back to ENABLE.

If a multi threaded program wants to cancel a thead while calling a 
libdar routine, it would just need to call the pthread_cancel() routine 
on the corresponding thread.

If the libdar code is out of a critical section the thread exits 
immediatly, if it is inside a critical section the thread exits once 
cancel state is enabled back after the critical section.

This way the global mutex objects stay in coherent state and thus it is 
possible to safely cancel a thread calling libdar routine.

By the way, note that the mutex used by libdar (and common to all 
threads) are never released. If this is a problem it might be possible 
to add a call in the API to release them and in consequence forbid any 
further call to libdar. The mutex used do not take much memory place 
however.

 >[...]
> 
> JB

What's your objections if any ? ;-)

I have at least one for now : it does not handle the allocated memory 
from the thread by libdar.

Cheers,
Denis.

[Dar-libdar_api] Re: [Dar-discussions] Re: problem testing infinint value

[Denis C. <dar...@fr...>]

Johnathan Burchill wrote:
> On June 25, 2004 02:31 pm, Denis Corbin wrote:
> 
>>Johnathan Burchill wrote:
>>
>>>On June 24, 2004 01:36 pm, Denis Corbin wrote:
>>
>>[...]
>[...]
> 
> 
> #include <config.h> 
> 
> kdar has its own config.h, which is generated by ./configure, and it 
> contains definitions of type sizes, among other things.
> 
> [...]
> 

Ah, OK I see ! :-)

> JB

Cheers,
Denis.

[Dar-libdar_api] Re: [Dar-discussions] Re: Fwd: I would like to cancel archive creation

[Johnathan B. <jk...@sh...>]

On June 19, 2004 02:43 pm, Denis Corbin wrote:
> Johnathan Burchill wrote:
> > On June 19, 2004 09:59 am, Johnathan Burchill wrote:
> >>Hi Denis,
>
> Hello Johnathan,
>
> >>I received this from a kdar user who asks whether it is possible to
> >>cancel a libdar operation. My first attempt will be to add the
> >> "dry-run" feature to the creation process. However, this is not a
> >> "quick" test of the backup, I think it takes a time comparable to the
> >> actual backup. Is this right?
>
> I think this is faster, but I have not made real comparison ;-)
>
> >>As far as interrupting the creation process, now that libdar is
> >>thread-safe, would there be a way to send it a "kill" signal, to get
> >> it to stop what it is doing? A couple of people have asked for this
> >> feature.
>
> does a SIGTERM handler would help ? There is a mutex to release, file
> descriptors to close, and memory to release. I suspect Linux closes file
> descriptors let openned by the thread, but it cannot obviously be done
> the same for mutex (which are shared by all threads). For other system I
> don't know what's the behavior about file descriptors.
>
> > QT has a method for terminating a thread, but this warning comes with
> > the documentation:
> >
> > Warning: This function is dangerous, and its use is discouraged. The
> > thread can be terminate at any point in its code path. Threads can be
> > terminated while modifying data. There is no chance for the thread to
> > cleanup after itself, unlock any held mutexes, etc. In short, use this
> > function only if absolutely necessary. (from
> > http://doc.trolltech.com/3.3/qthread.html)
> >
> > For the command line dar, a <Ctrl>-C will interrupt the libdar
> > operation. I'm guessing that the terminate method is the equivalent
> > for the QT GUI version. In dar-2.1.3, the library is not thread safe,
> > so I only run one operation at a time, and killing the operation
> > shouldn't be problematic as there are no locked mutexes. This should
> > be like hitting <Ctrl>-C. But the new threadsafe library may have
> > issues with this technique.
>
> yes.
>
> > Any comments?
>
> What standard mechanism over signal handlers can preempt a thread and
> let it do some arbitrary action ? I don't like polling. :-/
>

Signal handlers are based on polling, right? When you say polling, do you=20
mean that within your libdar operation loop a conditional statement checks=
=20
to see if a signal has been set?

What are the problems with polling? I am not familiar with the issues.=20
Portability? CPU cycles? ...?

I would suggest a public variable that can be set by the calling thread,

bool stop( false );

and in the definition,=20

while (...) //libdar::operation loop
{
	if ( !stop )
	{
		...libdar operations...
	}
	else
	{
		...clean up premature stop...
		...break out of operation loop
	}
}

Although this might present a problem when, say, you are testing two=20
separate archives at the same time, but you decide to cancel just one of=20
them. Instead, you could add an additional argument to the libdar call:

libdar::statistics st =3D libdar::op_test( theArchive, ... other=20
arguments ..., (bool *) stop );

The (bool *) points to a protected or private member variable of the=20
calling thread, so we could cancel one operation, keeping others going.

A problem with this technique is that it is inflexible. I'm not sure that=20
this is really significant though. Why would it be necessary for libdar to=
=20
be able to handle arbitrary signals?

JB
=2D-=20
Johnathan K. Burchill, Ph.D.
Department of Physics and Astronomy
University of Calgary
2500 University Drive N.W.
Calgary, AB T2N 1N4
Canada

(403) 217-4286
jk...@sh...

Re: [Dar-libdar_api] Re: [Dar-discussions] Re: problem testing infinint value

[Johnathan B. <jk...@sh...>]

On June 25, 2004 02:31 pm, Denis Corbin wrote:
> Johnathan Burchill wrote:
> > On June 24, 2004 01:36 pm, Denis Corbin wrote:
>
> [...]
>
> > Ah! The missing code was
> >
> > #ifdef HAVE_CONFIG_H
> > #include <config.h>
> > #endif
> >
> > And it has to be at the beginning of the .cpp files.
> > Thanks for the suggestion!
>
> Do you mean
>
>         #include <dar/config.h>
>
> ?

#include <config.h>=20

kdar has its own config.h, which is generated by ./configure, and it=20
contains definitions of type sizes, among other things.

[...]

JB
=2D-=20
Johnathan K. Burchill, Ph.D.
Department of Physics and Astronomy
University of Calgary
2500 University Drive N.W.
Calgary, AB T2N 1N4
Canada

(403) 217-4286
jk...@sh...

[Dar-libdar_api] Re: [Dar-discussions] Re: problem testing infinint value

[Denis C. <dar...@fr...>]

Johnathan Burchill wrote:
> On June 24, 2004 01:36 pm, Denis Corbin wrote:
> 
> 
[...]
> 
> Ah! The missing code was
> 
> #ifdef HAVE_CONFIG_H
> #include <config.h>
> #endif
> 
> And it has to be at the beginning of the .cpp files.
> Thanks for the suggestion!

Do you mean

        #include <dar/config.h>

?

> 
> [...]
> 
> 
>>for what it seems, the constant '1' in the expression, is cast as "long"
>>before being given to the limitint constructor. This is wrong because
>>there limitint is *unsigned* and cannot be built from *signed* integers.
> 
> 
> [...]
> 
>>To fix the problem, I suggest an explicit cast:
>>
>>	if (st.errored == libdar::infinint((U_I)1) plural = " ";
>>
>>U_I is a libdar macro, system independent, matching an unsigned integer.
>>(see "libdar/integers.hpp"), but if you prefer you can safely replace it
>>by "unsigned int".
> 
> 
> Signed long integers I can pass to the infinint constructor, and it work 
> okay. They must get cast implicitly by the library.

yes, probably.

> 
> [...]
> 
> 
>>Cheers,
>>Denis.
>>
>>P.S.: it should be more appropriate to post to lidar-api mailing-list
>>rather than to this general discussion mailing-list for such problems.
>>Thanks.
> 
> 
> No problem --- I should have considered this in the first place.

No problem ;-)

> 
> [...]
> 
> Cheers,
> JB
> 

Cheers,
Denis.

[Dar-libdar_api] Re: [Dar-discussions] Re: problem testing infinint value

[Johnathan B. <jk...@sh...>]

On June 24, 2004 01:36 pm, Denis Corbin wrote:

> Johnathan Burchill wrote:
> > Hi Denis,
>
> hello Johnathan,
>
> > This code has worked up until now.
> >
> > 		QString plural("s ");
> > 		libdar::statistics st=3Dop_test(theArchive, selection, subtree,
> > verbose); if (st.errored =3D=3D libdar::infinint(1)) plural =3D " ";
> >
> > It compiles fine (dar-2.1.3), but I get the following linker error:
> >
> >[...]
> >
> > *testArchiveThread.o(.text+0x2dd): In function
> > `testArchiveThread::run()': *: undefined reference to `void
> > libdar::limitint<unsigned long long>::limitint_from<long>(long)'
> > *collect2: ld returned 1 exit status
> > *Making all in doc
> > *gmake[2]: *** [kdar] Error 1
> > *gmake[2]: Target `all' not remade because of errors.
> > *
> >
> > The only change was to put the code into a separate .cpp file, which
> > compiles okay. When the code was inside my kdar.cpp file, there was no
> > problem during linkage.
>
> Have you checked the files included in kdar.cpp against those included
> in your separated file ? Has the compiler been changed ?

Ah! The missing code was

#ifdef HAVE_CONFIG_H
#include <config.h>
#endif

And it has to be at the beginning of the .cpp files.
Thanks for the suggestion!

[...]

>
> for what it seems, the constant '1' in the expression, is cast as "long"
> before being given to the limitint constructor. This is wrong because
> there limitint is *unsigned* and cannot be built from *signed* integers.

[...]
>
> To fix the problem, I suggest an explicit cast:
>
> 	if (st.errored =3D=3D libdar::infinint((U_I)1) plural =3D " ";
>
> U_I is a libdar macro, system independent, matching an unsigned integer.
> (see "libdar/integers.hpp"), but if you prefer you can safely replace it
> by "unsigned int".

Signed long integers I can pass to the infinint constructor, and it work=20
okay. They must get cast implicitly by the library.

[...]

> Cheers,
> Denis.
>
> P.S.: it should be more appropriate to post to lidar-api mailing-list
> rather than to this general discussion mailing-list for such problems.
> Thanks.

No problem --- I should have considered this in the first place.

[...]

Cheers,
JB

=2D-=20
Johnathan K. Burchill, Ph.D.
Department of Physics and Astronomy
University of Calgary
2500 University Drive N.W.
Calgary, AB T2N 1N4
Canada

(403) 217-4286
jk...@sh...

[Dar-libdar_api] Re: API / thread safe libdar

[Denis C. <dar...@fr...>]

Christian Neumann wrote:
> Hi.
> I just checked out the recent libdar cvs version. The user_interaction
> inheritance works for me, although I don't quite understand why one has
> to overwrite the clone() member or why it exists at all. Maybe I should
> revisit my book's chapters on this topic... 

well, that's a good and interesting question ! :-)

libdar needs to make temporary local copies of user_interaction objects. 
So libdar uses pointer to user_interaction which must be object of 
inherited class (user_interaction is a pure virtual class). A copy 
constructor will not work as it will duplicate only the use_interaction 
data not the data of any inherited class.

So, thanks to the clone() virtual method libdar am able to completely 
duplicate (clone) an given object which is only known through its parent 
class user_interaction.

If you have better way of doing so, my ears are all open ! :-)


> When are you going to release 2.1.4? As my program uses the new API, it
> won't be released before the next unstable libdar.

well, to respect the versionning of dar, the next release that will have 
this new API will be 2.2.0, 2.1.4 will only be the next version that 
only fixes bugs from 2.1.3, but does not add any feature.

When ? Well, it depends on my free time, I want to add encryption and 
multi-language support mainly, plus several other interesting features. 
Once done, there will be an API stabilization phase (where feedback from 
API user is very welcome), then pre-release (usually one month) and 
finally release.

> Another question: dar's TODO says that you will have to choose the
> language of dar at compile time. Is this correct? Won't you use gettext?

I plane to use gettext, the text you read is very old, it was before I 
was aware of gettext... I must fix that :-) I still must read 
documentation about gettext... as you see the '>' are here to help me 
remind the features I have to implement for next release, I don't say I 
will implement all but it would be nice to do so.

Releasing too often make me waste my time, in particular I must spend a 
lot of time in checking against portability issues, and building binary 
packages.

> 
> Regards,
> Christian
> 

Cheers,
Denis.

Re: [Dar-libdar_api] Re: API / thread safe libdar

[Christian N. <chr...@al...>]

Hi.
I just checked out the recent libdar cvs version. The user_interaction
inheritance works for me, although I don't quite understand why one has
to overwrite the clone() member or why it exists at all. Maybe I should
revisit my book's chapters on this topic... 
When are you going to release 2.1.4? As my program uses the new API, it
won't be released before the next unstable libdar.
Another question: dar's TODO says that you will have to choose the
language of dar at compile time. Is this correct? Won't you use gettext?

Regards,
Christian