dar-libdar_api — discussion area for developpers that want to use the libdar API

Re: [Dar-libdar_api] Re: problem compiling KDar with libdar CVS

[Johnathan B. <jk...@us...>]

Hi Denis,

On Wednesday 03 November 2004 13:08, Denis Corbin wrote:
> Denis Corbin wrote:
> |[...]
> | I proposes the following API :
> |
> | ~    void cancel_thread(pthread_t tid);
> | ~    bool cancel_current(pthread_t & tid);
> | ~    void cancel_clear();
> |
> | where cancel_thread() is used to cancel the given thread. Which will
> | make libdar throw a Euser_abort exception, cleanly free memory and
> | release any mutex.
> |
> | cancel_current() return true and the tid of the thread that will be
> | canceled when execution will reach the next checkpoint, false if there
> | is not pending cancelation.
> |
> | cancel_clear() abort cancelation process if the thread has not already
> | terminated.
> |
> | Only one thread can be canceled at a time.
> |
> |
> |[...]
>
> Hello,
>
> This is implemented and tested. CVS is up to date. If you have no
> objections, The API is now frozen (version 3) for the release 2.2.0 that
> will come in december I hope. The points that remains to do before the
> pre-release phase are :
> - documentation (API, user guide, etc.)
> - password/key from file
> - password/key from keyboard out of command-line
> - some few other strong encryption algorithms
> - LZO compression
>
> the two password/key features do not have any impact on the API (these
> are command-line related feature). The LZO and the new cypher support
> only add some more values in the "enum compression" and "enum
> crypto_algo" enumeration types.
>
> The pre-release target date is the beginning of december. Until that
> time, it is still time to fix API problem you could meet, so please give
> your feedback before pre-release phase.

I've got KDar running on libdar API V3.0.0, CVS. Works great!
The thread cancelation works perfectly, my limited testing shows.
The response is very snappy, and the user does not have to wait or restart=
=20
the program to stop an operation anymore. I think that you have a good=20
implementation of thread cancelation. Quite elegant.

One point about libdar's "make install": the config.h header that gets put=
=20
into <installdir>/include/dar/ does not contain "#define MUTEX_WORKS 1"=20
when I configure libdar with pthreads, so I have to add it manually to get=
=20
KDar to compile.

As an aside, here is a tip for people who want to implement thread=20
cancellation in QT:

Note that to cancel a libdar operation, you need to pass the ID of the=20
thread to cancel to libdar::cancel_thread(). You have to get this ID from=20
the "run()" method of the QThread, not it's constructor or anywhere else.=20
i.e. put "pthread_self()" inside run() and store it with a protected=20
member variable. This is not well documented by Trolltech, and it had me=20
puzzled for many hours today...

[...]

Cheers,
JB

=2D-=20
Johnathan K. Burchill, Ph.D.
jk...@us...

[Dar-libdar_api] Re: problem compiling KDar with libdar CVS

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Johnathan Burchill wrote:
| On Thursday 04 November 2004 02:27, Denis Corbin wrote:
|
|>Johnathan Burchill wrote:
|>
|[...]
|>
|>| calls dialog.warning(...) if the archive is encrypted and no password
|>| is given. Perhaps if "macro_tools_open_archive" could throw an
|>| Epassword() exception, that could be caught by the application, which
|>| could transparently ask the user for a password and try the read
|>| again.
|>
|>I like the idea, but throwing an exception is not a reversible operation
|>~ from the stack point of view (you cannot continue execution where it
|>has been thrown). I suppose you mean that libdar should rather use a the
|>user_interaction::pause(...) method to ask the password to the user, it
|>the pass argument was something like "bf:" or "scram:" where just the
|>algorithm (the cypher) is given ?
|
|
| I didn't expect libdar to continue where it left off after issuing
| Epassword(). Just that it would be a way to inform the application to ask
| the user for a password and try again to read the archive from the
| beginning.
|
| However, your idea of using user_interaction::pause looks promising. I
did
| not think of that one because the current implementation seems to be able
| to handle boolean questions, not arbitrary responses.

Yes you are right. user_interaction::pause() is only able to handle
boolean questions. Thus I had to add a new method to user_interaction class:

~   std::string get_string(const std::string & message, bool echo);

the message argument shall be displayed to the user and the user answer
be the returned value of the method. The echo argument is the "echo
local" value. In brief, if libdar set it to true the user answer can be
displayed, if libdar set it to false while calling get_string() all that
the user types must be hidden (like when entering passwords).

The inherited class like the user_interaction_callback must implement
get_string(). For this later class this implies a new callback function,
but nothing extraordinary there. Dar is updated and used this behavior.

So if the pass is an empty string while encryption cypher is set, this
mechanism is used to ask the pass to the user. Same thing when reading
an archive, if the archive uses encryption while a cypher encryption is
set and the pass argument is an empty string, libdar will ask the user
for a password through the same mechanisme.


I have added another check when reading an archive that is encrypted. If
the no cypher has been set (crypto_none) libdar will abort and refuse
too perform the operation as obviously this will lead to failure to read
the archive.

|
| The user_interaction::pause method is nice, because the application
can use
| whatever external program or library it wants to interact with the user.
| For example, I think KDE or QT has a library call for securely requesting
| passwords.
|
| The application would have to know how user_interaction::pause is to be
| used, though, right? Libdar would have to indicate that a particular
| user_interaction::pause call is to be for passwords, and not a yes/no
| response.

Yes. I realized this just after my previous mail.

|
|
|

Cheers,
Denis.

P.S.: CVS is up to date. Last feature added has been tested and seems to
work fine so far.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBimsxpC5CI8gYGlIRAvsRAKC0NVROXpnuXshNPFdH3ezS2K/y6wCgnQWf
8VPx0+AD3AOJA1xBMaLoCF4=
=182c
-----END PGP SIGNATURE-----

Re: [Dar-libdar_api] Re: problem compiling KDar with libdar CVS

[Johnathan B. <jk...@sh...>]

On Thursday 04 November 2004 02:27, Denis Corbin wrote:
> Johnathan Burchill wrote:
>
> Hello Johnathan,
>
> | On Wednesday 03 November 2004 13:08, Denis Corbin wrote:
> |>Denis Corbin wrote:
> |>|[...]
> |
> |[...]
> | As far as I can tell from the latest cvs API, when I read an existing
> | archive with the archive "read" constructor, I need to give it a
> | "const std::string &pass" argument, i.e. the password. So it seems
> | that the application has to know beforehand whether the user is
> | opening an encrypted archive or an unencrypted archive.
>
> In my point of view, the user has to know if the archive he tends to
> open is encrypted or not and if so it has to know too the password. The
> pass argument may be an empty string, or any other value, the given
> value will only be used if the archive has a flag set in the header of
> the archive telling that the archive has been encrypted (no information
> about the algorithm used or the block size used --- or the password of
> course --- is present in the header). Note that the pass has the
> structure described in the dar man page (option -K), "[algo:]pass" .
> where algo is bf, blowfish, scram or scrambling (more will follow I hope
> before the end of november) with "bf" = "blowfish", "scram" =
> "scrambling". If the password must contain a ':' the algo must be
> specified (in the form <algo>:<password_with_column>), so you can safely
> always specify the cypher used.
>
> A warning is issued if an encrypted archive is openned, because the
> diagnostic of a wrong password, is not easy to diagnostic neither for
> the user nor for libdar.
>
> | I see that archive::archive calls "macro_tools_open_archive" which in
>
> turn
>
> | calls dialog.warning(...) if the archive is encrypted and no password
> | is given. Perhaps if "macro_tools_open_archive" could throw an
> | Epassword() exception, that could be caught by the application, which
> | could transparently ask the user for a password and try the read
> | again.
>
> I like the idea, but throwing an exception is not a reversible operation
> ~ from the stack point of view (you cannot continue execution where it
> has been thrown). I suppose you mean that libdar should rather use a the
> user_interaction::pause(...) method to ask the password to the user, it
> the pass argument was something like "bf:" or "scram:" where just the
> algorithm (the cypher) is given ?

I didn't expect libdar to continue where it left off after issuing 
Epassword(). Just that it would be a way to inform the application to ask 
the user for a password and try again to read the archive from the 
beginning. 

However, your idea of using user_interaction::pause looks promising. I did 
not think of that one because the current implementation seems to be able 
to handle boolean questions, not arbitrary responses.

The user_interaction::pause method is nice, because the application can use 
whatever external program or library it wants to interact with the user. 
For example, I think KDE or QT has a library call for securely requesting 
passwords.

The application would have to know how user_interaction::pause is to be 
used, though, right? Libdar would have to indicate that a particular 
user_interaction::pause call is to be for passwords, and not a yes/no 
response.

>
> This is what I was looking for, to implement the password out of
> command-line for dar. Libdar could interactively ask the user for
> password when the user gives "-K bf:" or maybe just "-K :" where no
> password is given and the cypher is blowfish or where nothing is given
> at all, so the default cypher (bf actually) is used and no password is
> used.
>
> Thanks for the idea ! :-)
>
> | Or, is there a way in the API to check the archive, before opening it,
> | to see whether it is encrypted?
>
> well, it may have implications for the user if openning the archive once
> just to see if the archive is encrypted and a second time to open it for
> real. Implications can be extra change of disk for example.
>
> | Cheers,
> | JB
>
> Cheers,
> Denis.
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Sybase ASE Linux Express Edition - download now for FREE
> LinuxWorld Reader's Choice Award Winner for best database on Linux.
> http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
> _______________________________________________
> Dar-libdar_api mailing list
> Dar...@li...
> https://lists.sourceforge.net/lists/listinfo/dar-libdar_api

-- 
Johnathan K. Burchill, Ph.D.
Department of Physics and Astronomy
University of Calgary
2500 University Drive N.W.
Calgary, AB T2N 1N4
Canada

(403) 217-4286
jk...@sh...

[Dar-libdar_api] Re: problem compiling KDar with libdar CVS

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Denis Corbin wrote:
|[...]


I made a confusion between the API and the command-line syntax. Sorry.
The API has three different arguments :
~   crypto_algo crypto,
~   const std::string & pass,
~   U_32 crypto_size,

the pass and crypto_size arguments are used only if crypto is different
than crypto_none

| Note that the pass has the
| structure described in the dar man page (option -K), "[algo:]pass" .
| where algo is bf, blowfish, scram or scrambling (more will follow I hope
| before the end of november) with "bf" = "blowfish", "scram" =
| "scrambling". If the password must contain a ':' the algo must be
| specified (in the form <algo>:<password_with_column>), so you can safely
| always specify the cypher used.
|
|[...]

here too below, I made the same confusion. To correct the idea, if pass
is an empty string and crypto is not crypto_none then libdar could ask
the password to the user thanks to the user_interaction::pause(...) method.

|
| I like the idea, but throwing an exception is not a reversible operation
| ~ from the stack point of view (you cannot continue execution where it
| has been thrown). I suppose you mean that libdar should rather use a the
| user_interaction::pause(...) method to ask the password to the user, it
| the pass argument was something like "bf:" or "scram:" where just the
| algorithm (the cypher) is given ?
|
|[...]
|
| Cheers,
| Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBigKVpC5CI8gYGlIRAm37AJ0YtxRGIk3npMOplMNqhQGl+hlbDACfbYga
gTjklmjFWYag/+88z7FiUbM=
=LEIW
-----END PGP SIGNATURE-----

[Dar-libdar_api] Re: API documentation in headers?

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Johnathan Burchill wrote:
| Hi Denis,
|

Hello Johnathan,

| Regarding your plans to document the latest cvs API, have you considered
| moving to a doxygen style of header documentation? Is this your intention
| with respect to html API documentation?

This was not my intention, but that's a very good idea. :-) It seems to
be a very powerful tool for what is description on the web.

|
| I use it in the KDar source header files, and it would help with
| development based on libdar to be able to browse the API easily, and see
| the relationships between classes graphically, etc.
|
| If you like, I could send you a Doxyfile that I've modified slightly for
| libdar.

Yes, why not. However I will have a deep look in doxygen today.

|
| Cheers,
| JB

Cheers,
Denis.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBifcrpC5CI8gYGlIRAteHAJ9T5QOl9k//hlTv8e2Cg96UKpnigQCgymUL
fEo8zP6Clf+OZ5tXd1SkHkM=
=vxg0
-----END PGP SIGNATURE-----

[Dar-libdar_api] Re: problem compiling KDar with libdar CVS

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Johnathan Burchill wrote:

Hello Johnathan,

| On Wednesday 03 November 2004 13:08, Denis Corbin wrote:
|
|>Denis Corbin wrote:
|>|[...]
|[...]
| As far as I can tell from the latest cvs API, when I read an existing
| archive with the archive "read" constructor, I need to give it a "const
| std::string &pass" argument, i.e. the password. So it seems that the
| application has to know beforehand whether the user is opening an
| encrypted archive or an unencrypted archive.

In my point of view, the user has to know if the archive he tends to
open is encrypted or not and if so it has to know too the password. The
pass argument may be an empty string, or any other value, the given
value will only be used if the archive has a flag set in the header of
the archive telling that the archive has been encrypted (no information
about the algorithm used or the block size used --- or the password of
course --- is present in the header). Note that the pass has the
structure described in the dar man page (option -K), "[algo:]pass" .
where algo is bf, blowfish, scram or scrambling (more will follow I hope
before the end of november) with "bf" = "blowfish", "scram" =
"scrambling". If the password must contain a ':' the algo must be
specified (in the form <algo>:<password_with_column>), so you can safely
always specify the cypher used.

A warning is issued if an encrypted archive is openned, because the
diagnostic of a wrong password, is not easy to diagnostic neither for
the user nor for libdar.

|
| I see that archive::archive calls "macro_tools_open_archive" which in
turn
| calls dialog.warning(...) if the archive is encrypted and no password is
| given. Perhaps if "macro_tools_open_archive" could throw an Epassword()
| exception, that could be caught by the application, which could
| transparently ask the user for a password and try the read again.

I like the idea, but throwing an exception is not a reversible operation
~ from the stack point of view (you cannot continue execution where it
has been thrown). I suppose you mean that libdar should rather use a the
user_interaction::pause(...) method to ask the password to the user, it
the pass argument was something like "bf:" or "scram:" where just the
algorithm (the cypher) is given ?

This is what I was looking for, to implement the password out of
command-line for dar. Libdar could interactively ask the user for
password when the user gives "-K bf:" or maybe just "-K :" where no
password is given and the cypher is blowfish or where nothing is given
at all, so the default cypher (bf actually) is used and no password is used.

Thanks for the idea ! :-)

|
| Or, is there a way in the API to check the archive, before opening it, to
| see whether it is encrypted?

well, it may have implications for the user if openning the archive once
just to see if the archive is encrypted and a second time to open it for
real. Implications can be extra change of disk for example.

|
| Cheers,
| JB
|
|

Cheers,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBifXmpC5CI8gYGlIRArVsAJ9cIMUSRMbDISUFOTKf0oFFAbrCXgCePP/4
8u5/JizTPNvTX56PZM/OrVA=
=+UQ4
-----END PGP SIGNATURE-----

[Dar-libdar_api] API documentation in headers?

[Johnathan B. <jk...@us...>]

Hi Denis,

Regarding your plans to document the latest cvs API, have you considered 
moving to a doxygen style of header documentation? Is this your intention 
with respect to html API documentation?

I use it in the KDar source header files, and it would help with 
development based on libdar to be able to browse the API easily, and see 
the relationships between classes graphically, etc.

If you like, I could send you a Doxyfile that I've modified slightly for 
libdar.

Cheers,
JB
-- 
Johnathan K. Burchill, Ph.D.
jk...@us...

Re: [Dar-libdar_api] Re: problem compiling KDar with libdar CVS

[Johnathan B. <jk...@sh...>]

On Wednesday 03 November 2004 13:08, Denis Corbin wrote:
> Denis Corbin wrote:
> |[...]
> | I proposes the following API :
> |
> | ~    void cancel_thread(pthread_t tid);
> | ~    bool cancel_current(pthread_t & tid);
> | ~    void cancel_clear();
> |
> | where cancel_thread() is used to cancel the given thread. Which will
> | make libdar throw a Euser_abort exception, cleanly free memory and
> | release any mutex.
> |
> | cancel_current() return true and the tid of the thread that will be
> | canceled when execution will reach the next checkpoint, false if there
> | is not pending cancelation.
> |
> | cancel_clear() abort cancelation process if the thread has not already
> | terminated.
> |
> | Only one thread can be canceled at a time.
> |
> |
> |[...]
>
> Hello,
>
> This is implemented and tested. CVS is up to date. If you have no
> objections, The API is now frozen (version 3) for the release 2.2.0 that
> will come in december I hope. The points that remains to do before the
> pre-release phase are :
> - documentation (API, user guide, etc.)
> - password/key from file
> - password/key from keyboard out of command-line
> - some few other strong encryption algorithms
> - LZO compression
>
> the two password/key features do not have any impact on the API (these
> are command-line related feature). The LZO and the new cypher support
> only add some more values in the "enum compression" and "enum
> crypto_algo" enumeration types.

As far as I can tell from the latest cvs API, when I read an existing 
archive with the archive "read" constructor, I need to give it a "const 
std::string &pass" argument, i.e. the password. So it seems that the 
application has to know beforehand whether the user is opening an 
encrypted archive or an unencrypted archive.

I see that archive::archive calls "macro_tools_open_archive" which in turn 
calls dialog.warning(...) if the archive is encrypted and no password is 
given. Perhaps if "macro_tools_open_archive" could throw an Epassword() 
exception, that could be caught by the application, which could 
transparently ask the user for a password and try the read again.

Or, is there a way in the API to check the archive, before opening it, to 
see whether it is encrypted?

Cheers,
JB

>
> The pre-release target date is the beginning of december. Until that
> time, it is still time to fix API problem you could meet, so please give
> your feedback before pre-release phase.
>
> For short about API, waiting for an up to date documentation, all calls
> you need are in libdar.hpp and archive.hpp. The comments in the theses
> header files should help you understand the changes brought from API 2.
>
> Cheers,
> Denis.
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Sybase ASE Linux Express Edition - download now for FREE
> LinuxWorld Reader's Choice Award Winner for best database on Linux.
> http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
> _______________________________________________
> Dar-libdar_api mailing list
> Dar...@li...
> https://lists.sourceforge.net/lists/listinfo/dar-libdar_api

-- 
Johnathan K. Burchill, Ph.D.
Department of Physics and Astronomy
University of Calgary
2500 University Drive N.W.
Calgary, AB T2N 1N4
Canada

(403) 217-4286
jk...@sh...

[Dar-libdar_api] Re: problem compiling KDar with libdar CVS

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Denis Corbin wrote:
|[...]
| I proposes the following API :
|
| ~    void cancel_thread(pthread_t tid);
| ~    bool cancel_current(pthread_t & tid);
| ~    void cancel_clear();
|
| where cancel_thread() is used to cancel the given thread. Which will
| make libdar throw a Euser_abort exception, cleanly free memory and
| release any mutex.
|
| cancel_current() return true and the tid of the thread that will be
| canceled when execution will reach the next checkpoint, false if there
| is not pending cancelation.
|
| cancel_clear() abort cancelation process if the thread has not already
| terminated.
|
| Only one thread can be canceled at a time.
|
|
|[...]

Hello,

This is implemented and tested. CVS is up to date. If you have no
objections, The API is now frozen (version 3) for the release 2.2.0 that
will come in december I hope. The points that remains to do before the
pre-release phase are :
- - documentation (API, user guide, etc.)
- - password/key from file
- - password/key from keyboard out of command-line
- - some few other strong encryption algorithms
- - LZO compression

the two password/key features do not have any impact on the API (these
are command-line related feature). The LZO and the new cypher support
only add some more values in the "enum compression" and "enum
crypto_algo" enumeration types.

The pre-release target date is the beginning of december. Until that
time, it is still time to fix API problem you could meet, so please give
your feedback before pre-release phase.

For short about API, waiting for an up to date documentation, all calls
you need are in libdar.hpp and archive.hpp. The comments in the theses
header files should help you understand the changes brought from API 2.

Cheers,
Denis.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBiTqjpC5CI8gYGlIRAsV2AJ9wlNk64pXl0w8lOxRcAH2bqIUYLgCfQW7O
JlONqEWXWWBxFeg7hyKUnj8=
=a07r
-----END PGP SIGNATURE-----

[Dar-libdar_api] Re: problem compiling KDar with libdar CVS

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Johnathan Burchill wrote:
| On Monday 01 November 2004 11:48, Denis Corbin wrote:
|
|>Denis Corbin wrote:
|>| Hello Johnathan,
|>|
|>|[...]
|>|
|>| Things to review in the API are :
|>| - op_create & other functions must become methods of the archive class
|>| (for a more correct C++ design)
|>
|>this has been done today (CVS). Still needs documentation.
|>
|>| - what about thread cancelation ? Do we assume kdar will fork() rather
|>| than require a more complex mechanism to cancel a thread ?
|
|
| One weekend a while back I attempted to get the fork() working, with no
| success. There was some problem with forking a qt application and I did
| not understand how to fix it. I'd much prefer to stick with the thread
| cancellation technique.
|

OK

|
|>If the thread cancelation does not implies any change, then we are very
|
|
| It depends how you implement it. You once suggested a wrapper class that
| polls for a signal and throws an Euser_abort() if there is a cancellation
| request. What signal would it look for? Would it be independent of any of
| the pthread functions? How would the application set the signal?

the implementation I am working on does not rely on Unix signaling.

|
| Another way is to use cancellation points suitably located througout the
| code (see "man pthread_cancel"). As far as I can tell, what is needed is
| #include <pthread.h> and a bunch of pthread_testcancel()'s.
| You need so-called cancellation cleanup handlers for unlocking mutexes
and
| freeing memory (see "man pthread_cleanup_push" and the examples therein).
|
| It may require quite a bit of work. Also, I don't know whether it is
| directly portable to the cygwin version of libdar.

where there will not be pthread mutex available there will not be thread
cancelation available (and probably the system will not support
multi-threading neither). All is activated from the configuration script.

|
|
|>near the definitive API specifications for version 3 so we could
|>"freeze" the API. Some last feature will only add a type in enumeration
|>(LZO compression, RSA encryption, etc.), but class, function or methods
|>definitions should not change at all.
|
|
| Making the threads cancellable should not affect the API in any
significant
| way. Since the various libdar operations are now archive methods, you
| should have an "archive::cancel_operation()" method that toggles a
boolean
| which triggers the "Euser_abort()" exception between the filesystem reads
| or writes.

This is not possible this way, I proposes the following API :

~    void cancel_thread(pthread_t tid);
~    bool cancel_current(pthread_t & tid);
~    void cancel_clear();

where cancel_thread() is used to cancel the given thread. Which will
make libdar throw a Euser_abort exception, cleanly free memory and
release any mutex.

cancel_current() return true and the tid of the thread that will be
canceled when execution will reach the next checkpoint, false if there
is not pending cancelation.

cancel_clear() abort cancelation process if the thread has not already
terminated.

Only one thread can be canceled at a time.


|
| Now that I'm beginning to understand your wrapper class idea, I rather
like
| it.

;-)

The mechanisme is the following. When cancel_thread is called, it put
the requested tid in memory of a thread_cancelation object. Some key
classes (sar, tuyau, fichier and null_file) now inherit from the
thread_cancelation class. Thus they dispose of a method to check if the
thread Id (tid) they are run from has not been designed for cancelation.
If so the method throws the Euser_abort execption, which has the final
consequence of cleanly returning from the current libdar call.

the sar, tuyau fichier and null_file are the one used as low layer
objects in archive operation. The thread_cancelation checkpoint is put
in their read and write functions. To not overload the checkpoint
process I have limited the implementation to have a simple check. The
drawback is that only one thread can be canceled at a time (need to wait
for the cancelation has complete to as for a new cancelation).

Implementation is finished, I am testing it, and should upgrade CVS
tomorrow if all is fine.

|
| Cheers,
| JB
|

Cheers,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBiBDWpC5CI8gYGlIRAsCVAJ44wNiSKbn6rq4f5btWK0oLvW0JCACggSTi
m1PO5FMg5IlbLC9T4jjNPNE=
=lukb
-----END PGP SIGNATURE-----

Re: [Dar-libdar_api] Re: problem compiling KDar with libdar CVS

[Johnathan B. <jk...@sh...>]

On Monday 01 November 2004 11:48, Denis Corbin wrote:
> Denis Corbin wrote:
> | Hello Johnathan,
> |
> |[...]
> |
> | Things to review in the API are :
> | - op_create & other functions must become methods of the archive class
> | (for a more correct C++ design)
>
> this has been done today (CVS). Still needs documentation.
>
> | - what about thread cancelation ? Do we assume kdar will fork() rather
> | than require a more complex mechanism to cancel a thread ?

One weekend a while back I attempted to get the fork() working, with no=20
success. There was some problem with forking a qt application and I did=20
not understand how to fix it. I'd much prefer to stick with the thread=20
cancellation technique.

>
> If the thread cancelation does not implies any change, then we are very

It depends how you implement it. You once suggested a wrapper class that=20
polls for a signal and throws an Euser_abort() if there is a cancellation=20
request. What signal would it look for? Would it be independent of any of=20
the pthread functions? How would the application set the signal?

Another way is to use cancellation points suitably located througout the=20
code (see "man pthread_cancel"). As far as I can tell, what is needed is=20
#include <pthread.h> and a bunch of pthread_testcancel()'s.
You need so-called cancellation cleanup handlers for unlocking mutexes and=
=20
freeing memory (see "man pthread_cleanup_push" and the examples therein).

It may require quite a bit of work. Also, I don't know whether it is=20
directly portable to the cygwin version of libdar.

> near the definitive API specifications for version 3 so we could
> "freeze" the API. Some last feature will only add a type in enumeration
> (LZO compression, RSA encryption, etc.), but class, function or methods
> definitions should not change at all.

Making the threads cancellable should not affect the API in any significant=
=20
way. Since the various libdar operations are now archive methods, you=20
should have an "archive::cancel_operation()" method that toggles a boolean=
=20
which triggers the "Euser_abort()" exception between the filesystem reads=20
or writes.

Now that I'm beginning to understand your wrapper class idea, I rather like=
=20
it.

Cheers,
JB

>
> My current actions are the following in order of priority:
> - documentation (review all in HTML format, API documentation first)
> - finish strong encryption (password from interactive question, or from
> file, some few other strong encryptions algorithms)
> - LZO compression if time allows it (all development stop at end of
> November for 2.2.0 pre-release phase)
>
> | Cheers,
> | Denis.
>
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Sybase ASE Linux Express Edition - download now for FREE
> LinuxWorld Reader's Choice Award Winner for best database on Linux.
> http://ads.osdn.com/?ad_id=3D5588&alloc_id=3D12065&op=3Dclick
> _______________________________________________
> Dar-libdar_api mailing list
> Dar...@li...
> https://lists.sourceforge.net/lists/listinfo/dar-libdar_api

=2D-=20
Johnathan K. Burchill, Ph.D.
Department of Physics and Astronomy
University of Calgary
2500 University Drive N.W.
Calgary, AB T2N 1N4
Canada

(403) 217-4286
jk...@sh...

[Dar-libdar_api] Re: problem compiling KDar with libdar CVS

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Denis Corbin wrote:
| Hello Johnathan,
|
|[...]
|
| Things to review in the API are :
| - op_create & other functions must become methods of the archive class
| (for a more correct C++ design)

this has been done today (CVS). Still needs documentation.

| - what about thread cancelation ? Do we assume kdar will fork() rather
| than require a more complex mechanism to cancel a thread ?

If the thread cancelation does not implies any change, then we are very
near the definitive API specifications for version 3 so we could
"freeze" the API. Some last feature will only add a type in enumeration
(LZO compression, RSA encryption, etc.), but class, function or methods
definitions should not change at all.

My current actions are the following in order of priority:
- - documentation (review all in HTML format, API documentation first)
- - finish strong encryption (password from interactive question, or from
file, some few other strong encryptions algorithms)
- - LZO compression if time allows it (all development stop at end of
November for 2.2.0 pre-release phase)

|
| Cheers,
| Denis.
|
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBhoT9pC5CI8gYGlIRApa6AKCJAoCIASYsnjIon7/SolGXqbR9GgCeObOm
1b4v7uWQmtWL6ll7Y3m5PwE=
=FtiH
-----END PGP SIGNATURE-----

[Dar-libdar_api] Re: problem compiling KDar with libdar CVS

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello Johnathan,

There is two points I see about the API:

- - first, since I have added multi-language support (the gettext function
comes from here), I have not yet reviewed the general installation
process, maybe there is some points to review about installed header
files...(this will be done before pre-release phase, which target is
december 2004).

- - second, there is still changes to come in the API as discussed in this
forum (see details bellow). I will look at this soon, as I am about to
finish the implementation about the strong encryption features.

Before pre-release phase, I will send a mail to this mailing list for an
API review before release. I expect it before half november.

Things to review in the API are :
- - op_create & other functions must become methods of the archive class
(for a more correct C++ design)
- - what about thread cancelation ? Do we assume kdar will fork() rather
than require a more complex mechanism to cancel a thread ?

Cheers,
Denis.


Johnathan Burchill wrote:
| On Saturday 30 October 2004 01:00, Johnathan Burchill wrote:
|
|>Hi Denis,
|>
|>I've decided it is time to port KDar over to the new libdar API3. Libdar
|>installed without obvious problems. However, KDar's configure script
|>chokes, and spits out the following log messages:
|>
|>configure:32503: checking dar/libdar.hpp usability
|>configure:32515: g++ -c -Wnon-virtual-dtor -Wno-long-long -Wundef -Wall
|>-pedantic
|>-W -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -ansi
|>-D_XOPEN_SOURCE=500
|>-D_BSD_SOURCE -Wcast-align -Wconversion -DNDEBUG -DNO_DEBUG -O2 -O2
|>-pipe -march=a
|>thlon-xp -mmmx -msse -m3dnow -fno-exceptions -fno-check-new -fexceptions
|>-fexcepti
|>ons  -DQT_THREAD_SUPPORT  -D_REENTRANT conftest.cc >&5
|>In file included from /usr/local/include/dar/storage.hpp:29,
|>                 from /usr/local/include/dar/real_infinint.hpp:38,
|>                 from /usr/local/include/dar/infinint.hpp:30,
|>                 from /usr/local/include/dar/compressor.hpp:30,
|>                 from /usr/local/include/dar/libdar.hpp:32,
|>                 from conftest.cc:148:
|>/usr/local/include/dar/erreurs.hpp: In constructor
|>   `libdar::Ememory::Ememory(const std::string&)':
|>/usr/local/include/dar/erreurs.hpp:67: `gettext' undeclared (first use
|>this function)
|>/usr/local/include/dar/erreurs.hpp:67: (Each undeclared identifier is
|>reported only once for each function it appears in.)
|>configure:32521: $? = 1
|>
|>I am not familiar with gettext, but it seems that the function should be
|>declared somewhere.
|>
|>Any ideas?
|
|
| I've just upgraded to Slackware 10, and the problem disappeared. There
| might have been a glibc version problem or mismatch somewhere.
|
| Cheers,
| JB
|
|
|>Cheers,
|>JB
|
|
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBhiHapC5CI8gYGlIRAtL8AJsFEvg12nous1lYzEVfZWkAgpWWCgCfUyGx
VMPGVO1YnQWYccsMJjwjbl8=
=aU4m
-----END PGP SIGNATURE-----

Re: [Dar-libdar_api] problem compiling KDar with libdar CVS

[Johnathan B. <jk...@sh...>]

On Saturday 30 October 2004 01:00, Johnathan Burchill wrote:
> Hi Denis,
>
> I've decided it is time to port KDar over to the new libdar API3. Libdar
> installed without obvious problems. However, KDar's configure script
> chokes, and spits out the following log messages:
>
> configure:32503: checking dar/libdar.hpp usability
> configure:32515: g++ -c -Wnon-virtual-dtor -Wno-long-long -Wundef -Wall
> -pedantic
> -W -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -ansi
> -D_XOPEN_SOURCE=3D500
> -D_BSD_SOURCE -Wcast-align -Wconversion -DNDEBUG -DNO_DEBUG -O2 -O2
> -pipe -march=3Da
> thlon-xp -mmmx -msse -m3dnow -fno-exceptions -fno-check-new -fexceptions
> -fexcepti
> ons  -DQT_THREAD_SUPPORT  -D_REENTRANT conftest.cc >&5
> In file included from /usr/local/include/dar/storage.hpp:29,
>                  from /usr/local/include/dar/real_infinint.hpp:38,
>                  from /usr/local/include/dar/infinint.hpp:30,
>                  from /usr/local/include/dar/compressor.hpp:30,
>                  from /usr/local/include/dar/libdar.hpp:32,
>                  from conftest.cc:148:
> /usr/local/include/dar/erreurs.hpp: In constructor
>    `libdar::Ememory::Ememory(const std::string&)':
> /usr/local/include/dar/erreurs.hpp:67: `gettext' undeclared (first use
> this function)
> /usr/local/include/dar/erreurs.hpp:67: (Each undeclared identifier is
> reported only once for each function it appears in.)
> configure:32521: $? =3D 1
>
> I am not familiar with gettext, but it seems that the function should be
> declared somewhere.
>
> Any ideas?

I've just upgraded to Slackware 10, and the problem disappeared. There=20
might have been a glibc version problem or mismatch somewhere.

Cheers,
JB

>
> Cheers,
> JB

=2D-=20
Johnathan K. Burchill, Ph.D.
Department of Physics and Astronomy
University of Calgary
2500 University Drive N.W.
Calgary, AB T2N 1N4
Canada

(403) 217-4286
jk...@sh...

[Dar-libdar_api] problem compiling KDar with libdar CVS

[Johnathan B. <jk...@us...>]

Hi Denis,

I've decided it is time to port KDar over to the new libdar API3. Libdar=20
installed without obvious problems. However, KDar's configure script=20
chokes, and spits out the following log messages:

configure:32503: checking dar/libdar.hpp usability
configure:32515: g++ -c -Wnon-virtual-dtor -Wno-long-long -Wundef -Wall=20
=2Dpedantic
=2DW -Wpointer-arith -Wmissing-prototypes -Wwrite-strings -ansi=20
=2DD_XOPEN_SOURCE=3D500
=2DD_BSD_SOURCE -Wcast-align -Wconversion -DNDEBUG -DNO_DEBUG -O2 -O2 -pipe=
=20
=2Dmarch=3Da
thlon-xp -mmmx -msse -m3dnow -fno-exceptions -fno-check-new -fexceptions=20
=2Dfexcepti
ons  -DQT_THREAD_SUPPORT  -D_REENTRANT conftest.cc >&5
In file included from /usr/local/include/dar/storage.hpp:29,
                 from /usr/local/include/dar/real_infinint.hpp:38,
                 from /usr/local/include/dar/infinint.hpp:30,
                 from /usr/local/include/dar/compressor.hpp:30,
                 from /usr/local/include/dar/libdar.hpp:32,
                 from conftest.cc:148:
/usr/local/include/dar/erreurs.hpp: In constructor
   `libdar::Ememory::Ememory(const std::string&)':
/usr/local/include/dar/erreurs.hpp:67: `gettext' undeclared (first use this
   function)
/usr/local/include/dar/erreurs.hpp:67: (Each undeclared identifier is=20
reported only once for each function it appears in.)
configure:32521: $? =3D 1

I am not familiar with gettext, but it seems that the function should be=20
declared somewhere.

Any ideas?

Cheers,
JB

=2D-=20
Johnathan K. Burchill, Ph.D.
jk...@us...

[Dar-libdar_api] Re: Retrieve file history inside dar catalogue

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thierry Pierron wrote:
| Hi all !

Hello Thierry,

|
| I found dar some weeks ago, and I found it quite useful. I used to
play with
| afbackup, but I'll be quite happy to replace it by dar.
|
| Okay, but I just would like to have a little utility that can show me
| contents of dar catalogue created by dar_manager, with history of each
| files archived.

dar_manager can give you the information of a given file with the -f option.

|
| Is it possible with libdar to read and retrieve file information (at least
| size, perm, date, and filename of course) and each version ?

dar_manager database are not in libdar but in dar_manager own code. It
has not been yet moved to libdar unlike most of the dar code.

|
| If not is it possible to read directly (without help of libdar) the .dmd
| file ? I don't see any description of that file in the archive of dar.

dar_manager is the only command that is aware of dmd files (dar_manager
database).

|
| I try "dar_manager -u0 -B <arch>.dmd", but it shows me only last filename
| information, with no additional information.

a dar_manager database does not contain all information about a given
inode, just the name, the dates and the archive where is has been taken
from.

|
| Well, actually I often got requests of users who have accidentaly damaged
| one of their precious file. They usually notice that mistake just a
few days
| later. With afbackup I have to do ugly hack to restore the correct
"version",
| and I'm quite fed up with this. That's why if I can get history in a
sort of
| tree view, it will greatly simplify my life.

I think dar_manager can save you a lot of time. Suppose a user named joe
tells you he has, by mistake, destroyed his ".login" file. Assuming as a
system administrator you regulary make backups of the system with dar
and each time update a dar_manager database named "system" you can have
an history of the requested file thanks to the following command :

	dar_manager -B system -f home/joe/.login

which will produce something like this :

Archive number |  Data      |  EA
- ---------------+------------+------------
~        1       Sat Dec  6 11:26:37 2003
~        9       Sat Dec  6 11:26:37 2003

joe's .login file has been saved in archive 1 and 9 and the modification
dates is given in second column. No EA have been saved (else the
corresponding date would be listed in thrid column).

you can then use dar on the 9th or the 1st backup to restore the
requested file:

	dar -x <archvie n°1> -R / -v home/joe/.login

if you just want to get the last version you can use dar_manager :

~        dar_manager -B system -r home/joe/.login

assuming you have given the database the following option to pass
to dar :
	dar_manager -B system -o -R / -v

which will then make the same thing as:

	dar -x <archive n°9> -R / -v home/joe/.login

The advantage of using dar_manager for retreiving the last version of a
given set of file, is that you have only one command to type that
receive the list of files to restore. Then dar_manager will call dar
several time if necessary with the different archive where are located
the last versions of requested files.

|
| Maybe KDar can do this, but it was rather painful to use (Moreover I'm not
| a KDE user)
|
| BTW, a little advice about DAR API. Why do you bother writing library for
| such tool ? It is much more simple to provide pipe-driven command,
just like
| aspell/ispell or "gdb --interpreter=mi". You don't have to painfully
provide
| bindings for every languages.

In the last news, there will be bindings only for C++. As you said it is
painful to make bindings to other languages, dar and libdar are written
in C++

Now why a libdar and a API ? To have a GUI first. As I don't find much
fun devlopping a GUI, this was for me the most simple way to let other
do it, having a clear interface between the different development teams.

Now, why a GUI ? Because end-users too want to use dar. Many feel more
confortable with graphical user interface than command line interface.
having a GUI does not remove anything the the dar command line applications.

| Okay, maybe that "trick" does not work very
| well on Windows

the graphical application that generates the command-line was in project
too. But seems to have been dropped, I cannot have any news from the
author.

| Well, it's just an idea, in case of one day you want to
| provide another bindinds. Just think of it, because it has many advantages
| over dynamic/static library.

for windows, Dar/libdar rely on Cygwin. It seems to work properly so far
(still have a pending bug that I am unable to reproduce).

|
| Regards,
| --
| Thierry Pierron
|

Cheers,
Denis.

|
|
|
| -------------------------------------------------------
| This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
| Use IT products in your business? Tell us what you think of them. Give us
| Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out
more
| http://productguide.itmanagersjournal.com/guidepromo.tmpl
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBZnAApC5CI8gYGlIRAptsAJ0Q5fKYljThamyv7tDgTXxywR3cRACgnJfO
AI4QG9/dS9VR4VZJzleWBTo=
=TyDy
-----END PGP SIGNATURE-----

[Dar-libdar_api] Retrieve file history inside dar catalogue

[Thierry P. <tpi...@b2...>]

Hi all !

I found dar some weeks ago, and I found it quite useful. I used to play with
afbackup, but I'll be quite happy to replace it by dar.

Okay, but I just would like to have a little utility that can show me
contents of dar catalogue created by dar_manager, with history of each
files archived.

Is it possible with libdar to read and retrieve file information (at least
size, perm, date, and filename of course) and each version ?

If not is it possible to read directly (without help of libdar) the .dmd
file ? I don't see any description of that file in the archive of dar.

I try "dar_manager -u0 -B <arch>.dmd", but it shows me only last filename
information, with no additional information.

Well, actually I often got requests of users who have accidentaly damaged
one of their precious file. They usually notice that mistake just a few days
later. With afbackup I have to do ugly hack to restore the correct "version",
and I'm quite fed up with this. That's why if I can get history in a sort of
tree view, it will greatly simplify my life.

Maybe KDar can do this, but it was rather painful to use (Moreover I'm not
a KDE user).

BTW, a little advice about DAR API. Why do you bother writing library for
such tool ? It is much more simple to provide pipe-driven command, just like
aspell/ispell or "gdb --interpreter=mi". You don't have to painfully provide
bindings for every languages. Okay, maybe that "trick" does not work very
well on Windows. Well, it's just an idea, in case of one day you want to
provide another bindinds. Just think of it, because it has many advantages
over dynamic/static library.

Regards,
--
Thierry Pierron

[Dar-libdar_api] Re: [Kdar-users] extra semicolons in header file are not portable

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

There is already a bug report (item [1009407] on Sourceforge) about dar
not compiling with gcc-3.4.x. This is known, and will be fixed soon.

Cheers,
Denis.

Marcel Meyer wrote:
| Hi all,
|
| Am Donnerstag, 9. September 2004 18:42 schrieb Johnathan Burchill:
|
|>Comments? The obvious solution is to remove the extra semicolons from the
|>header files.
|
|
| # cd /usr/include/dar
| # for DATEI in *; do cat $DATEI | sed 's/};/}/g' > $DATEI.tmp; mv
$DATEI.tmp
| $DATEI; done
| # grep '};' *
| -> no more such files
| # emerge kdar
| -> still same problem....
|
|  :-(
|
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBQK1SpC5CI8gYGlIRAiQ8AJ90kdpGnW3jJPxigjqqcFGp3/A+rACgp3wM
prKS/Qrhhgt+l1mNapTX+Ic=
=5BVG
-----END PGP SIGNATURE-----

[Dar-libdar_api] Re: [Kdar-users] extra semicolons in header file are not portable

[Marcel M. <me...@fs...>]

Hi all,

Am Donnerstag, 9. September 2004 18:42 schrieb Johnathan Burchill:
> Comments? The obvious solution is to remove the extra semicolons from the
> header files.

# cd /usr/include/dar
# for DATEI in *; do cat $DATEI | sed 's/};/}/g' > $DATEI.tmp; mv $DATEI.tm=
p=20
$DATEI; done
# grep '};' *
=2D> no more such files
# emerge kdar
=2D> still same problem....

 :-(

=2D-=20
Marcel Meyer
| Netzwerk- und Rechnerorganisation
| Fachschaft Mathematik/Physik/Informatik
| Technische Universit=E4t M=FCnchen

[Dar-libdar_api] extra semicolons in header file are not portable

[Johnathan B. <jk...@us...>]

=2D----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Denis,

I've gotten a couple of reports from users trying to compile KDar using gcc=
=20
3.4.1. The configure script fails on checking the usability of libdar.hpp=20
and on checking for libdar.so:

e.g.

configure:28461: checking for libdar
configure:28489: g++ -o conftest -Wnon-virtual-dtor -Wno-long-long -Wundef=
=20
=2D -Wall -pedantic -W -Wpointer-arith -Wmissing-prototypes -Wwrite-strings=
=20
=2D -ansi -D_XOPEN_SOURCE=3D500 -D_BSD_SOURCE -Wcast-align -Wconversion -O2=
=20
=2D -fno-exceptions -fno-check-new -fexceptions -fexceptions =20
=2D -DQT_THREAD_SUPPORT  -D_REENTRANT -ldar conftest.cc  >&5
cc1plus: warning: command line option "-Wmissing-prototypes" is valid for=20
Ada/C/ObjC but not for C++
In file included from /usr/local/include/dar/infinint.hpp:30,
                 from /usr/local/include/dar/generic_file.hpp:38,
                 from /usr/local/include/dar/compressor.hpp:30,
                 from /usr/local/include/dar/libdar.hpp:30,
                 from conftest.cc:50:
/usr/local/include/dar/real_infinint.hpp:110: error: extra semicolon
configure:28492: $? =3D 1
configure: program exited with status 1

The code near line 110 of real_infinint.hpp (in dar-2.1.4) is=20
        template <class T>void unstack(T &v) throw(Ememory, Erange, Ebug)
            { E_BEGIN; infinint_unstack_to(v); E_END("infinint::unstack",=20
typeid(v).name()); };

Apparently gcc-3.4.1 doesn't accept the last ";" when -pedantic is passed.

I checked into this and found the following site with some tips on making=20
code portable:

http://www.mozilla.org/hacking/portable-cpp.html

Tip #12 is relevant here:
http://www.mozilla.org/hacking/portable-cpp.html#no_extra_semi_colons

It specifically mentions gcc-3.4.

Comments? The obvious solution is to remove the extra semicolons from the=20
header files.

The only temporary fix I can think of is to make a patch available for dar=
=20
to allow other programs to link to the library with gcc-3.4. Unless the=20
users disable the "-pedantic" switch, but I don't have gcc-3.4 so I can't=20
test this at the moment.

Cheers,
JB
=2D --=20
Johnathan K. Burchill, Ph.D.
jk...@us...
=2D----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQFBQIfdrXMSOtiWrO4RAjdfAJ9URMP2kVlRye6bpGdb9E7rotiHGQCgim+c
rzJtxA52lZGbHnnzc727VN4=3D
=3Dywk6
=2D----END PGP SIGNATURE-----

[Dar-libdar_api] Re: last archive slice

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Johnathan Burchill wrote:
| On July 25, 2004 01:10 pm, Denis Corbin wrote:
|
|>Johnathan Burchill wrote:
|>| On July 24, 2004 11:22 pm, Denis Corbin wrote:
|
|[...]
|
|>I would even say that you may miss a middle slice while dar knows
|>nothing that it is missing and keeps working properly restoring a subset
|>of files. Where is then disk archive as conceptualized object ? ;-)
|>(still quibbling).
|
|
| Is not a file an object? I can delete a character from the middle and it
| remains a valid file. An archive with missing slices may fail some CRC
| checks, but it is still an archive with (some accessible) data.

in the context of common human language, you are right, I think, a file
is an object one can manipulate. An archive is an object too.

|
| Actually, I am not convinced that archives should not be considered
| objects. For example, libdar does not give the user the option of
| specifying different backup directories for each slice.

dar could, but the only difficulty is how to have a simple specification
(compatible with command line restriction) that can assign a directory
to each slice number. Would the effort worth that feature ?


| The archive is in
| one directory. Libdar does not care whether that one directory is spread
| over different times, all the slices are still in that directory. When
the
| user is asked to provide a missing slice, it has to be placed in that
| directory.

the user can also have slice in many different directories and use
symbolic link that makes them appear in a single directory.

| As far as the user is concerned, an archive may not be a
| self-contained object, what with some of it sent over email, some on
| floppy, some on disk. But whenever libdar interacts with a given archive,
| all of the slices have to be available from the same source.

once an archive openned, dar still needs the good will of the user to
perform its action to the end. Required interaction may be far more
complex than requesting that the user put the correct tape in the
driver, if we want dar solve all the problem in place of the user. Dar
could scan the entire disk for the needed slice, keep a database of any
known burnt disks, seen floppies, and request the correct one in time,
etc... but that's not the direction I would like to take for libdar or
dar. I am not interested in making people more assisted, I am interested
in real feature a backup software can have.

|
| I must consider this further before coming to a conclusion.

me too. :-)

| [...]
|
|
|>That should be correct to my point of view. But you know some user
|>already told me that dar was too talkative, in particular about the
|>initial warning telling that all question will abort the program.
|
|
| A clever user. ;)

lol

who knows... ;-)


|>
|
|
| Okay, I will wait and see how things go in the API. Thanks for the
| discussion!

Thanks to you too. I may be a bit strict about the place of dar
relatively to the user (keeping things simples and rely on user's brain
for they understand what they do), but I stay open to discussion,
arguments against arguments, the best idea get the deal.

|
| JB
|
|

Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBBV3tpC5CI8gYGlIRAtWlAJ4+fIYGYSl+Ke2P9k/j6XAABEAi+QCfW8k5
BLYjceLIhuYdKgOHJP1XkTQ=
=1tRN
-----END PGP SIGNATURE-----

Re: [Dar-libdar_api] Re: last archive slice

[Johnathan B. <jk...@sh...>]

On July 25, 2004 01:10 pm, Denis Corbin wrote:
> Johnathan Burchill wrote:
> | On July 24, 2004 11:22 pm, Denis Corbin wrote:
> |
> |[...]
> |
> |>This is a user mistake to hide under the same basename in the same
> |>directory slices of different archives.
> |>
> |>There is no use to keep last slices of an old archive, it only waste
> |>your disk space: If you miss the first slice, you won't be able to
> |>restore anything from your old archive. Else, if you definitively want
> |>to keep several archives with the same name, it is not a very good
> |> idea to mix their respective slices toghether in the same
> |> directory... ;-) no ?
> |
> | That is exactly the problem I was trying to address. The slices _are_
> | absolutely useless, and just waste disk space, so why not have libdar
> | remove them?
>
> For me, the problem in adding additionnal slice removal feature in
> libdar is that there is more risk to erase precious data. So I would
> prefer that the user take their responsibilities in managing their own
> data. But...

Certainly.

> >=20
> But, I propose that if slice overwriting is detected and other slices
> than the first are present in the same directory, dar will ask the user
> whether the other old slices must be removed first or not (this, in any
> case, no option to deactivate that question). Nothing will be done to
> ask the user to provide missing slices of the old archive that could be
> located on other directory, filesystems, floppy, hard disk, ...
>
> The situation where overwriting occures stays for me a user error.
> allowing the deletion of old slices in the current directory, fixes the
> not so intuitive behavior of lidbar when remains older slices in the
> given directory when the new archive has been created. For the rest,
> this is under the user responsibility to through away the old CD-R and
> to erase the old slices located on separated floppies, etc.

Ah, I had not considered this issue. I understand now that with regular,=20
self-contained, files, you can be sure you have deleted the entire=20
contents when unlinking them. With slices, they could be all over the=20
place, some on disk, some on CD, etc.You can't treat an archive as a=20
self-contained object then, except in the case where it has only one=20
slice.

>
> | [...]
> |
> |>| I see the following solutions:
> |>|
> |>| 1) libdar deletes all slices on a disk if they have the same storage
> |>| directory and basename as the new one being created. It might do
> |>| this before creating the archive, or after creating the archive.
> |>| Perhaps doing this after will be more efficient, since you only
> |>| delete slices that haven't been overwritten, if there are any.
> |>
> |>I do not agree. libdar deletes nothing. 'rm' command does. ('rm' or
> |>unlink() system call)
>
> I keep thinking that when the user sees a warning telling that a file is
> about to be overwritten, he must understand that the warning is not only
> for decoration nor it is for legal disclosure. If the user ignores the
> warning and continues with dar, he must assume his action and take his
> responsibilities.
>
> See tar for example: tar does never warn against archive overwriting (by
> default at least) and nobody complains, when an archive is lost this
> way.
>
> | Libdar does delete files on disk if the appropriate conditions are met
> | during a restoration, but I quibble :).
> |
> :-)
> :
> | But libdar does overwrite archive slices! Is this not a form of
> | deletion?
>
> yes, and that's already too much. As explained above. Do the same with
> tar, you will not get any warning at all, do you ?
>
> | Consider the echo command:
> |
> | echo "hello" > file1
> | echo "bye" > file1
> |
> | What is file1?
> |
> | cat file1
> | bye
> |
> | It is not, "byelo", yet this is the analogous behavior that dar does
> | when creating a new archive with slices, while replacing one that
> | already exists. As you intend this to be the behaviour, I suggest that
> | it is not the obvious or intuitive one, and at least should be
> | documented.
>
> Apologies
>
> | in advance if it is already in the manpage.
>
> you have not to apology, there is nothing in man page about all that a
> user can do, clever things or less clever ones. There is only
> documentation about what dar does.
>
> to take your example, using tar in place of dar,
>
> tar -cf file1 <some files>
> tar -cf file1 <some other files>
>
> So, what is in file1 ?
>
> cat file1
> <some other files>
>
> No, it is not "<some files><some other files>" as this would be more
> intuitive to be. Maybe you thing this normal ? Then, this it is normal
> too that when you overwrite 7 files, the other remaing files stay
> unchanged and are not deleted, are they ? ;-) Just to quibble, too. ;-)

My argument was based on the idea that an archive is a self-contained=20
object. Indeed, this does not seem to hold in the real world. :)

>
> | I conceptualize disk archives as objects, whether they are made up of
> | slices or not. Making a new backup in a directory that contains an
>
> archive
>
> | of the same name should replace the archive as a whole, not on a
> | slice-by-slice basis. I do not see anything "clever" or assuming about
> | that for libdar.
>
> because there is not conceptualization of disk archives as "objets",
> they is not a single entity when an archive is split in slices, for the
> user has the total freedom to dispose some *files* on floppy, some
> others on disk and why not send the remaing ones by email. They stay
> files.
>
> I would even say that you may miss a middle slice while dar knows
> nothing that it is missing and keeps working properly restoring a subset
> of files. Where is then disk archive as conceptualized object ? ;-)
> (still quibbling).

Is not a file an object? I can delete a character from the middle and it=20
remains a valid file. An archive with missing slices may fail some CRC=20
checks, but it is still an archive with (some accessible) data.

Actually, I am not convinced that archives should not be considered=20
objects. For example, libdar does not give the user the option of=20
specifying different backup directories for each slice. The archive is in=20
one directory. Libdar does not care whether that one directory is spread=20
over different times, all the slices are still in that directory. When the=
=20
user is asked to provide a missing slice, it has to be placed in that=20
directory. As far as the user is concerned, an archive may not be a=20
self-contained object, what with some of it sent over email, some on=20
floppy, some on disk. But whenever libdar interacts with a given archive,=20
all of the slices have to be available from the same source.

I must consider this further before coming to a conclusion.

>
> | An archive with one slice, made with a slice size of 0, would be
>
> completely
>
> | replaced by the new archive. By your logic you should overwrite the
> | initial portion of the old archive with the new one, not completely
> | replace it.
>
> nop, because when dar opens a file in O_TRUNC mode (which implies the
> user has accepted to loose the data in the file) you loose the data in
> that file only (not in the other files).
> nop too, because, when you overwrite in a directory you do not replace
> all the files of that directory. Directory (that contain slices) do not
> act like plain files (that contain data). this is Unix feature ;-)
>
> | Same thing for when someone saves a file in a document processor. You
>
> don't
>
> | expect to have to first remove the one on disk before saving what is
> | in memory to the same filename.
>
> true, but when you overwrite an file you loose all its contents just
> after you have openned it. So there is not much difference in deleting
> it first before creating a new one with the same name. So such
> applications usually save the new content in a temporary files then swap
> the filenames. Do you wan't dar do the same ? :-)
>
> | In fact, in the API, libdar::archive is a class for which you can
> | instantiate archive objects. The objects aren't individual slices,
> | they are entire archives. Overwriting an old archive object with a new
> | archive object with the same name, same directory, should completely
> | replace the old archive object.
>
> implementation is one thing that is hidden to the user. You can have
> many different implementations for the same behavior, if dar was
> implemented else, no user would see the difference and your argument
> would not stay valid.
>
> |[...]
> |
> |
> | I understand your point now.
> |
> | I am not suggesting though that libdar just stop reporting that there
> | are extra useless slices, I am suggesting that it do something to stop
> | that situation from happening. I will make a weaker request then, that
> | at
>
> least
>
> | libdar warn the user of this situation during the creation process.
> |
> | For instance, would it not make sense for libdar to report during the
> | creation process that there are extra useless slices, report which
> | ones they are, and ask the user whether they should be removed, or
> | warn the user that they should be removed manually? You eliminate the
>
> assumption by
>
> | asking the question.
>
> That should be correct to my point of view. But you know some user
> already told me that dar was too talkative, in particular about the
> initial warning telling that all question will abort the program.

A clever user. ;)

>
> | After all, you do ask if your user intends to get into the "endless
> | loop" by backing up a directory that contains the archive, with no
> | exclusion file filters. You don't let it happen without the question.
>
> yes, because the user may stay a long time before understanding what's
> the matter. Some user may just thing dar is broken and only waste disk
> space and memory. This is only to avoid having wasting time for support
> about that common situation. B-)=3D)
>
> |>I don't like MS-office like programs that tend to have the prentention
> |>to be more clever than their users. They either get much restrictive
> |> and blindly forbid operation needed by more clever users than the
> |> program developers, or automatically do stupid things or have stupid
> |> questions to the user, that even less clever users find borring.
> |
> | I wholeheartedly agree! :)
> |
> :)
> :
> |[...]
> |
> |>To my point of view, this is here again a user mistake. Once you
> |>overwrite the first slice of the old archive, all the other slices of
> |>this archives are useless. Why not removing first all the slices of
> |> the old archive you have planned to overwrite ?
> |
> | I agree, if the user removes the old archive before creating the new
> | one, this situation will never arise in KDar.
> |
> | What I do not understand is why, given that you admit the remaining
>
> slices
>
> | become useless once the first one is overwritten, libdar does not
> | remove those useless extra slices automatically.
>
> In fact, the problem is different. If to find the last slice of an
> archive dar was more simple, and incrementally look slice by slice for a
> terminating slice, the remaining old slices would not cause a warning.
> If there were not a internal number to check against slice mixing of
> different archive, there would not be a warning, and dar would simply
> tell there is incoherent data structure in the archive.
>
> But it is not so simple, it looks in the current archive for the highest
> slice available opens it looking whether it is a terminating slice. It
> also check whether it is part of the same archive and if not reports the
> error to the user asking for help because it is unable (and does not
> want) to figure out what to do.

It makes sense to me now.

>
> If it was more simple and reported a incoherent data structure due to
> the mix of archive slices, do you thing user would complain anymore ?
>
> | Can you imagine some situation where, for a more clever user, those
> | extra slices are not useless?
>
> I don't pretend being more clever than dar's users. Maybe some of them
> could find something to do that I cannot even imagin.
>
> | Perhaps this is a stupid question! :)
>
> perhaps not ! :)
>
> |>| Perhaps libdar should write the "N" to the termination type only if
> |>| the user says it's okay to continue, and write an "I" to the
> |>| termination type if the user cancels. That way we know that the
> |>| "last" archive slice is actually invalid, i.e. the creation process
> |>| was aborted, or failed in
> |>
> |>some
> |>
> |>| way.
>
> I think this is too complicated, and not much useful. This assumes that
> the user has to agree at each slice creation, and that you can change
> afterward a byte (thus you cannot dump anymore an archive to a pipe).

Got it.

>
> |>perhaps, KDAR could propose a "remove old archive slice" option ? ;-)
> |>option, that can simply rely on the rm command or unlink system
> |> call...
> |
> | Agreed. Except I still think if you allow the user to overwrite an
>
> archive,
>
> | the entire archive should be replaced, not just individual slices.
>
> yes, I agree.
>
> |>| For the status indicator, the best solution would be for libdar to
> |>| have a "currentSlice" method, which could be called at any time to
> |>| determine the slice number of the current one being written or read.
> |>
> |>a new callback function ? ;-)
> |
> | There's no need for a callback function. The method just returns the
>
> slice
>
> | number, a libdar::infinint. I envision usage as such:
> |
> | //(start a new creation thread):
> |
> | libdar::archive =3D newArchive;
> | createArchiveThread *createThread =3D new
> | createArchiveThread( newArchive, ...);
> | createThread->start();
> |
> | //(occasionaly check the current slice number):
> | while ( createThread->running() )
> | {
> | 	libdar::infinint currentSlice =3D theArchive->currentSlice();
> | 	//report the current slice to the user
> | 	updateStatusBar( currentSlice );
> | 	//Sleep and check again
> | 	sleep( DURATION );
> | }
>
> well, this could be the subject of another discussion. "What do we
> consider when speaking about thread safe library ?".
>
> For me, this is when a given library proposes a set of functions and/or
> data structures that can be used in independant threads. This does not
> mean that a given data can be used transparently by several threads at
> the same time. In particular, think about an integer variable, in
> parallel programming you need to make critical section around any
> modification of this variable, to have only one thread modifying it at a
> given time. Now, replace the integer variable by an "archive" object:
> when it is executing an action (through a method like archive creation),
> that means it is run by a first thread, thus it should be in a critical
> section, thus no other thread should modify this same object, in
> particular another thread should not call a method of that same object.
>
> Instead, if you want to know the current slice, you just have --execute
> option that can return this information, at each slice creation, and
> pass this information from one thread to another thanks to a auxilliary
> variable (which may be accessed through a critical section).
>
> |>| Should I file a bug report for any of this?
> |>
> |>I don't think so, do you still ?
> |
> | I do see now that the issue is whether the user or the library should
> | be removing slices. Fair enough, your philosophy of design is keep it
> | simple and unassuming. Just because I cannot imagine any situation in
> | which the user will ever need those "useless" extra slices, doesn't
> | mean that someone else won't.
>
> Exactly. I'm happy to be understood ! :-)
>
> | I will implement a "do not overwrite entire archive" option, that
>
> defaults
>
> | to being off.
>
> well, maybe you will not have to... as explained at the beginning of my
> reply here.
>

Okay, I will wait and see how things go in the API. Thanks for the=20
discussion!

JB

> | [...]
> |
> |>Cheers,
> |>Denis.
> |
> | Cheers,
> | JB
>
> Cheers,
> Denis.
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by BEA Weblogic Workshop
> FREE Java Enterprise J2EE developer tools!
> Get your free copy of BEA WebLogic Workshop 8.1 today.
> http://ads.osdn.com/?ad_id=3D4721&alloc_id=3D10040&op=3Dclick
> _______________________________________________
> Dar-libdar_api mailing list
> Dar...@li...
> https://lists.sourceforge.net/lists/listinfo/dar-libdar_api

=2D-=20
Johnathan K. Burchill, Ph.D.
Department of Physics and Astronomy
University of Calgary
2500 University Drive N.W.
Calgary, AB T2N 1N4
Canada

(403) 217-4286
jk...@sh...

[Dar-libdar_api] Re: last archive slice

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Johnathan Burchill wrote:
| On July 24, 2004 11:22 pm, Denis Corbin wrote:
|
|[...]
|>This is a user mistake to hide under the same basename in the same
|>directory slices of different archives.
|>
|>There is no use to keep last slices of an old archive, it only waste
|>your disk space: If you miss the first slice, you won't be able to
|>restore anything from your old archive. Else, if you definitively want
|>to keep several archives with the same name, it is not a very good idea
|>to mix their respective slices toghether in the same directory... ;-) no
|>?
|
|
| That is exactly the problem I was trying to address. The slices _are_
| absolutely useless, and just waste disk space, so why not have libdar
| remove them?

For me, the problem in adding additionnal slice removal feature in
libdar is that there is more risk to erase precious data. So I would
prefer that the user take their responsibilities in managing their own
data. But...

But, I propose that if slice overwriting is detected and other slices
than the first are present in the same directory, dar will ask the user
whether the other old slices must be removed first or not (this, in any
case, no option to deactivate that question). Nothing will be done to
ask the user to provide missing slices of the old archive that could be
located on other directory, filesystems, floppy, hard disk, ...

The situation where overwriting occures stays for me a user error.
allowing the deletion of old slices in the current directory, fixes the
not so intuitive behavior of lidbar when remains older slices in the
given directory when the new archive has been created. For the rest,
this is under the user responsibility to through away the old CD-R and
to erase the old slices located on separated floppies, etc.

| [...]
|>
|>| I see the following solutions:
|>|
|>| 1) libdar deletes all slices on a disk if they have the same storage
|>| directory and basename as the new one being created. It might do this
|>| before creating the archive, or after creating the archive. Perhaps
|>| doing this after will be more efficient, since you only delete slices
|>| that haven't been overwritten, if there are any.
|>
|>I do not agree. libdar deletes nothing. 'rm' command does. ('rm' or
|>unlink() system call)

I keep thinking that when the user sees a warning telling that a file is
about to be overwritten, he must understand that the warning is not only
for decoration nor it is for legal disclosure. If the user ignores the
warning and continues with dar, he must assume his action and take his
responsibilities.

See tar for example: tar does never warn against archive overwriting (by
default at least) and nobody complains, when an archive is lost this way.

|
|
| Libdar does delete files on disk if the appropriate conditions are met
| during a restoration, but I quibble :).

:-)


|
| But libdar does overwrite archive slices! Is this not a form of deletion?

yes, and that's already too much. As explained above. Do the same with
tar, you will not get any warning at all, do you ?


| Consider the echo command:
|
| echo "hello" > file1
| echo "bye" > file1
|
| What is file1?
|
| cat file1
| bye
|
| It is not, "byelo", yet this is the analogous behavior that dar does when
| creating a new archive with slices, while replacing one that already
| exists. As you intend this to be the behaviour, I suggest that it is not
| the obvious or intuitive one, and at least should be documented.
Apologies
| in advance if it is already in the manpage.

you have not to apology, there is nothing in man page about all that a
user can do, clever things or less clever ones. There is only
documentation about what dar does.

to take your example, using tar in place of dar,

tar -cf file1 <some files>
tar -cf file1 <some other files>

So, what is in file1 ?

cat file1
<some other files>

No, it is not "<some files><some other files>" as this would be more
intuitive to be. Maybe you thing this normal ? Then, this it is normal
too that when you overwrite 7 files, the other remaing files stay
unchanged and are not deleted, are they ? ;-) Just to quibble, too. ;-)


|
| I conceptualize disk archives as objects, whether they are made up of
| slices or not. Making a new backup in a directory that contains an
archive
| of the same name should replace the archive as a whole, not on a
| slice-by-slice basis. I do not see anything "clever" or assuming about
| that for libdar.

because there is not conceptualization of disk archives as "objets",
they is not a single entity when an archive is split in slices, for the
user has the total freedom to dispose some *files* on floppy, some
others on disk and why not send the remaing ones by email. They stay files.

I would even say that you may miss a middle slice while dar knows
nothing that it is missing and keeps working properly restoring a subset
of files. Where is then disk archive as conceptualized object ? ;-)
(still quibbling).

|
| An archive with one slice, made with a slice size of 0, would be
completely
| replaced by the new archive. By your logic you should overwrite the
| initial portion of the old archive with the new one, not completely
| replace it.

nop, because when dar opens a file in O_TRUNC mode (which implies the
user has accepted to loose the data in the file) you loose the data in
that file only (not in the other files).
nop too, because, when you overwrite in a directory you do not replace
all the files of that directory. Directory (that contain slices) do not
act like plain files (that contain data). this is Unix feature ;-)

|
| Same thing for when someone saves a file in a document processor. You
don't
| expect to have to first remove the one on disk before saving what is in
| memory to the same filename.

true, but when you overwrite an file you loose all its contents just
after you have openned it. So there is not much difference in deleting
it first before creating a new one with the same name. So such
applications usually save the new content in a temporary files then swap
the filenames. Do you wan't dar do the same ? :-)

|
| In fact, in the API, libdar::archive is a class for which you can
| instantiate archive objects. The objects aren't individual slices, they
| are entire archives. Overwriting an old archive object with a new archive
| object with the same name, same directory, should completely replace the
| old archive object.

implementation is one thing that is hidden to the user. You can have
many different implementations for the same behavior, if dar was
implemented else, no user would see the difference and your argument
would not stay valid.

|[...]
|
|
| I understand your point now.
|
| I am not suggesting though that libdar just stop reporting that there are
| extra useless slices, I am suggesting that it do something to stop that
| situation from happening. I will make a weaker request then, that at
least
| libdar warn the user of this situation during the creation process.
|
| For instance, would it not make sense for libdar to report during the
| creation process that there are extra useless slices, report which ones
| they are, and ask the user whether they should be removed, or warn the
| user that they should be removed manually? You eliminate the
assumption by
| asking the question.

That should be correct to my point of view. But you know some user
already told me that dar was too talkative, in particular about the
initial warning telling that all question will abort the program.

|
| After all, you do ask if your user intends to get into the "endless loop"
| by backing up a directory that contains the archive, with no exclusion
| file filters. You don't let it happen without the question.

yes, because the user may stay a long time before understanding what's
the matter. Some user may just thing dar is broken and only waste disk
space and memory. This is only to avoid having wasting time for support
about that common situation. B-)=)

|
|
|>I don't like MS-office like programs that tend to have the prentention
|>to be more clever than their users. They either get much restrictive and
|>blindly forbid operation needed by more clever users than the program
|>developers, or automatically do stupid things or have stupid questions
|>to the user, that even less clever users find borring.
|
|
| I wholeheartedly agree! :)

:)

|[...]
|
|>To my point of view, this is here again a user mistake. Once you
|>overwrite the first slice of the old archive, all the other slices of
|>this archives are useless. Why not removing first all the slices of the
|>old archive you have planned to overwrite ?
|
|
| I agree, if the user removes the old archive before creating the new one,
| this situation will never arise in KDar.
|
| What I do not understand is why, given that you admit the remaining
slices
| become useless once the first one is overwritten, libdar does not remove
| those useless extra slices automatically.

In fact, the problem is different. If to find the last slice of an
archive dar was more simple, and incrementally look slice by slice for a
terminating slice, the remaining old slices would not cause a warning.
If there were not a internal number to check against slice mixing of
different archive, there would not be a warning, and dar would simply
tell there is incoherent data structure in the archive.

But it is not so simple, it looks in the current archive for the highest
slice available opens it looking whether it is a terminating slice. It
also check whether it is part of the same archive and if not reports the
error to the user asking for help because it is unable (and does not
want) to figure out what to do.

If it was more simple and reported a incoherent data structure due to
the mix of archive slices, do you thing user would complain anymore ?

|
| Can you imagine some situation where, for a more clever user, those extra
| slices are not useless?

I don't pretend being more clever than dar's users. Maybe some of them
could find something to do that I cannot even imagin.

|
| Perhaps this is a stupid question! :)

perhaps not ! :)

|
|
|>| Perhaps libdar should write the "N" to the termination type only if
|>| the user says it's okay to continue, and write an "I" to the
|>| termination type if the user cancels. That way we know that the "last"
|>| archive slice is actually invalid, i.e. the creation process was
|>| aborted, or failed in
|>
|>some
|>
|>| way.

I think this is too complicated, and not much useful. This assumes that
the user has to agree at each slice creation, and that you can change
afterward a byte (thus you cannot dump anymore an archive to a pipe).

|>
|>perhaps, KDAR could propose a "remove old archive slice" option ? ;-)
|>option, that can simply rely on the rm command or unlink system call...
|>
|
|
| Agreed. Except I still think if you allow the user to overwrite an
archive,
| the entire archive should be replaced, not just individual slices.

yes, I agree.

|
|
|>| For the status indicator, the best solution would be for libdar to
|>| have a "currentSlice" method, which could be called at any time to
|>| determine the slice number of the current one being written or read.
|>
|>a new callback function ? ;-)
|
|
| There's no need for a callback function. The method just returns the
slice
| number, a libdar::infinint. I envision usage as such:
|
| //(start a new creation thread):
|
| libdar::archive = newArchive;
| createArchiveThread *createThread = new
| createArchiveThread( newArchive, ...);
| createThread->start();
|
| //(occasionaly check the current slice number):
| while ( createThread->running() )
| {
| 	libdar::infinint currentSlice = theArchive->currentSlice();
| 	//report the current slice to the user
| 	updateStatusBar( currentSlice );
| 	//Sleep and check again
| 	sleep( DURATION );
| }


well, this could be the subject of another discussion. "What do we
consider when speaking about thread safe library ?".

For me, this is when a given library proposes a set of functions and/or
data structures that can be used in independant threads. This does not
mean that a given data can be used transparently by several threads at
the same time. In particular, think about an integer variable, in
parallel programming you need to make critical section around any
modification of this variable, to have only one thread modifying it at a
given time. Now, replace the integer variable by an "archive" object:
when it is executing an action (through a method like archive creation),
that means it is run by a first thread, thus it should be in a critical
section, thus no other thread should modify this same object, in
particular another thread should not call a method of that same object.

Instead, if you want to know the current slice, you just have --execute
option that can return this information, at each slice creation, and
pass this information from one thread to another thanks to a auxilliary
variable (which may be accessed through a critical section).

|
|
|>| Should I file a bug report for any of this?
|>
|>I don't think so, do you still ?
|>
|
|
| I do see now that the issue is whether the user or the library should be
| removing slices. Fair enough, your philosophy of design is keep it simple
| and unassuming. Just because I cannot imagine any situation in which the
| user will ever need those "useless" extra slices, doesn't mean that
| someone else won't.

Exactly. I'm happy to be understood ! :-)

|
| I will implement a "do not overwrite entire archive" option, that
defaults
| to being off.

well, maybe you will not have to... as explained at the beginning of my
reply here.

|
| [...]
|
|>Cheers,
|>Denis.
|
|
| Cheers,
| JB
|

Cheers,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBBAWTpC5CI8gYGlIRAkpVAKC/0+Y2/C5P6mG/2A+LctZFsgL2/gCeLg9Q
YP9HqzQG6Xhn+eU6jBqz44g=
=kHZe
-----END PGP SIGNATURE-----

Re: [Dar-libdar_api] Re: last archive slice

[Johnathan B. <jk...@sh...>]

On July 24, 2004 11:22 pm, Denis Corbin wrote:
> Johnathan Burchill wrote:
> | Hi Denis,
>
> Hello Johnathan,
>
> | In dar version 2.1.3, libdar mistakes the last slice number sometimes.
> |
> | For example, create an archive with slice sizes of 3 MB. Say this
> | results in 6 slices. Now create the archive, same basename, same
> | directory, and allow the library to overwrite the existing archive,
> | but use 5 MB slices. Suppose it results in 3 slices.
> |
> | Now try to list the archive. I get the following user interaction
>
> question:
> | "/opt/user/backups/dar_backups/test6.6.dar is a file from another set
> | of backup file, please provide the correct file."
> |
> | Then the user has to delete slices, one by one starting with the
> | highest number, and retry the list command until it works.
> |
> | Is this the expected behaviour?
>
> yes.
>
> | Shouldn't libdar see that the third slice,
> | in this example, would be the last archive slice? I have to admit,
> | it's confusing for a user who does this to know that the archive
> | consists of only three slices when six are on disk.
>
> This is a user mistake to hide under the same basename in the same
> directory slices of different archives.
>
> There is no use to keep last slices of an old archive, it only waste
> your disk space: If you miss the first slice, you won't be able to
> restore anything from your old archive. Else, if you definitively want
> to keep several archives with the same name, it is not a very good idea
> to mix their respective slices toghether in the same directory... ;-) no
> ?

That is exactly the problem I was trying to address. The slices _are_=20
absolutely useless, and just waste disk space, so why not have libdar=20
remove them?

>
> | You wouldn't have to change the slice size to get this problem.
>
> Suppose you
>
> | did a full backup and got 12 CD-R-sized slices. Then during the course
> | of the following week, you delete a bunch of directories that you
> | don't need anymore, and do a full backup which has 11 CD-R-sized
> | slices. Dar will miss the fact that the last slice  #11, but will
> | still ask the user for the 12th slice.
>
> what would be the 12th slice useful for ? why would it have to stay on
> disk for ?
>
> | I see the following solutions:
> |
> | 1) libdar deletes all slices on a disk if they have the same storage
> | directory and basename as the new one being created. It might do this
> | before creating the archive, or after creating the archive. Perhaps
> | doing this after will be more efficient, since you only delete slices
> | that haven't been overwritten, if there are any.
>
> I do not agree. libdar deletes nothing. 'rm' command does. ('rm' or
> unlink() system call)

Libdar does delete files on disk if the appropriate conditions are met=20
during a restoration, but I quibble :).

But libdar does overwrite archive slices! Is this not a form of deletion?
Consider the echo command:

echo "hello" > file1
echo "bye" > file1

What is file1?

cat file1
bye

It is not, "byelo", yet this is the analogous behavior that dar does when=20
creating a new archive with slices, while replacing one that already=20
exists. As you intend this to be the behaviour, I suggest that it is not=20
the obvious or intuitive one, and at least should be documented. Apologies=
=20
in advance if it is already in the manpage.

I conceptualize disk archives as objects, whether they are made up of=20
slices or not. Making a new backup in a directory that contains an archive=
=20
of the same name should replace the archive as a whole, not on a=20
slice-by-slice basis. I do not see anything "clever" or assuming about=20
that for libdar.

An archive with one slice, made with a slice size of 0, would be completely=
=20
replaced by the new archive. By your logic you should overwrite the=20
initial portion of the old archive with the new one, not completely=20
replace it.

Same thing for when someone saves a file in a document processor. You don't=
=20
expect to have to first remove the one on disk before saving what is in=20
memory to the same filename.

In fact, in the API, libdar::archive is a class for which you can=20
instantiate archive objects. The objects aren't individual slices, they=20
are entire archives. Overwriting an old archive object with a new archive=20
object with the same name, same directory, should completely replace the=20
old archive object.

>
> | 2) The search algorithm gets fixed to stop on the first slice with a
> | "T" type terminating character in the header. Am I correct in
> | interpreting
>
> the
>
> | "T" to mean that the slice is the last one in the archive? I assume
> | that libdar figures out the last slice by relying on the filename, not
> | the header, although I haven't checked the sourcecode.
>
> Assuming in libdar the mistakes of the users is not a good idea. I can
> already hear some users complaining that libdar does not remove unuseful
> slices of older archives, while it does not complain anymore that extra
> slices exist and do only waste disk space.

I understand your point now.=20

I am not suggesting though that libdar just stop reporting that there are=20
extra useless slices, I am suggesting that it do something to stop that=20
situation from happening. I will make a weaker request then, that at least=
=20
libdar warn the user of this situation during the creation process.=20

=46or instance, would it not make sense for libdar to report during the=20
creation process that there are extra useless slices, report which ones=20
they are, and ask the user whether they should be removed, or warn the=20
user that they should be removed manually? You eliminate the assumption by=
=20
asking the question.

After all, you do ask if your user intends to get into the "endless loop"=20
by backing up a directory that contains the archive, with no exclusion=20
file filters. You don't let it happen without the question.

>
> I don't like MS-office like programs that tend to have the prentention
> to be more clever than their users. They either get much restrictive and
> blindly forbid operation needed by more clever users than the program
> developers, or automatically do stupid things or have stupid questions
> to the user, that even less clever users find borring.

I wholeheartedly agree! :)

>
> | 3) A combination of 1) and 2).
>
> no again. If a user wants to clean its previous archive it is simple:
>
> ~  rm old_archive.*.dar
> ~  dar -c old_arhive ...
>
> nothing more nothing less. Simple, everybody understands, no hidden
> features from dar or libdar, no hidden surprises. (lib)dar is only a
> backup software.
>
> | I see a similar problem when creating archives with KDar. I have a
>
> progress
>
> | bar that periodically updates which slice is currently being written,
> | and what that slice's size is. When the "pause" between slices option
> | is chosen, and libdar asks if it okay to continue writing the next
> | slice, first it writes the "non-terminating" character to the slice
> | header, and then waits for the user to answer the question. In the
> | case where the
>
> user
>
> | is overwriting an older archive, the statusbar code runs through the
>
> files
>
> | on disk until it finds the one with the "T" in the header. This will
> | be the last one in the older archive, and the statusbar will jump to
> | that slice instead of the current one.
>
> To my point of view, this is here again a user mistake. Once you
> overwrite the first slice of the old archive, all the other slices of
> this archives are useless. Why not removing first all the slices of the
> old archive you have planned to overwrite ?

I agree, if the user removes the old archive before creating the new one,=20
this situation will never arise in KDar.

What I do not understand is why, given that you admit the remaining slices=
=20
become useless once the first one is overwritten, libdar does not remove=20
those useless extra slices automatically.=20

Can you imagine some situation where, for a more clever user, those extra=20
slices are not useless?=20

Perhaps this is a stupid question! :)=20

>
> | Perhaps libdar should write the "N" to the termination type only if
> | the user says it's okay to continue, and write an "I" to the
> | termination type if the user cancels. That way we know that the "last"
> | archive slice is actually invalid, i.e. the creation process was
> | aborted, or failed in
>
> some
>
> | way.
>
> perhaps, KDAR could propose a "remove old archive slice" option ? ;-)
> option, that can simply rely on the rm command or unlink system call...
>

Agreed. Except I still think if you allow the user to overwrite an archive,=
=20
the entire archive should be replaced, not just individual slices.

> | For the status indicator, the best solution would be for libdar to
> | have a "currentSlice" method, which could be called at any time to
> | determine the slice number of the current one being written or read.
>
> a new callback function ? ;-)

There's no need for a callback function. The method just returns the slice=
=20
number, a libdar::infinint. I envision usage as such:

//(start a new creation thread):

libdar::archive =3D newArchive;
createArchiveThread *createThread =3D new=20
createArchiveThread( newArchive, ...);=20
createThread->start();

//(occasionaly check the current slice number):
while ( createThread->running() )
{
	libdar::infinint currentSlice =3D theArchive->currentSlice();
	//report the current slice to the user
	updateStatusBar( currentSlice );
	//Sleep and check again
	sleep( DURATION );
}

>
> | Should I file a bug report for any of this?
>
> I don't think so, do you still ?
>

I do see now that the issue is whether the user or the library should be=20
removing slices. Fair enough, your philosophy of design is keep it simple=20
and unassuming. Just because I cannot imagine any situation in which the=20
user will ever need those "useless" extra slices, doesn't mean that=20
someone else won't.

I will implement a "do not overwrite entire archive" option, that defaults=
=20
to being off.

[...]
>
> Cheers,
> Denis.

Cheers,
JB

>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by BEA Weblogic Workshop
> FREE Java Enterprise J2EE developer tools!
> Get your free copy of BEA WebLogic Workshop 8.1 today.
> http://ads.osdn.com/?ad_id=3D4721&alloc_id=3D10040&op=3Dclick
> _______________________________________________
> Dar-libdar_api mailing list
> Dar...@li...
> https://lists.sourceforge.net/lists/listinfo/dar-libdar_api

=2D-=20
Johnathan K. Burchill, Ph.D.
Department of Physics and Astronomy
University of Calgary
2500 University Drive N.W.
Calgary, AB T2N 1N4
Canada

(403) 217-4286
jk...@sh...

[Dar-libdar_api] Re: last archive slice

[Denis C. <dar...@fr...>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Johnathan Burchill wrote:
| Hi Denis,

Hello Johnathan,

|
| In dar version 2.1.3, libdar mistakes the last slice number sometimes.
|
| For example, create an archive with slice sizes of 3 MB. Say this results
| in 6 slices. Now create the archive, same basename, same directory, and
| allow the library to overwrite the existing archive, but use 5 MB slices.
| Suppose it results in 3 slices.
|
| Now try to list the archive. I get the following user interaction
question:
|
| "/opt/user/backups/dar_backups/test6.6.dar is a file from another set of
| backup file, please provide the correct file."
|
| Then the user has to delete slices, one by one starting with the highest
| number, and retry the list command until it works.
|
| Is this the expected behaviour?

yes.

| Shouldn't libdar see that the third slice,
| in this example, would be the last archive slice? I have to admit, it's
| confusing for a user who does this to know that the archive consists of
| only three slices when six are on disk.

This is a user mistake to hide under the same basename in the same
directory slices of different archives.

There is no use to keep last slices of an old archive, it only waste
your disk space: If you miss the first slice, you won't be able to
restore anything from your old archive. Else, if you definitively want
to keep several archives with the same name, it is not a very good idea
to mix their respective slices toghether in the same directory... ;-) no ?

|
| You wouldn't have to change the slice size to get this problem.
Suppose you
| did a full backup and got 12 CD-R-sized slices. Then during the course of
| the following week, you delete a bunch of directories that you don't need
| anymore, and do a full backup which has 11 CD-R-sized slices. Dar will
| miss the fact that the last slice  #11, but will still ask the user for
| the 12th slice.

what would be the 12th slice useful for ? why would it have to stay on
disk for ?

|
| I see the following solutions:
|
| 1) libdar deletes all slices on a disk if they have the same storage
| directory and basename as the new one being created. It might do this
| before creating the archive, or after creating the archive. Perhaps doing
| this after will be more efficient, since you only delete slices that
| haven't been overwritten, if there are any.

I do not agree. libdar deletes nothing. 'rm' command does. ('rm' or
unlink() system call)

|
| 2) The search algorithm gets fixed to stop on the first slice with a "T"
| type terminating character in the header. Am I correct in interpreting
the
| "T" to mean that the slice is the last one in the archive? I assume that
| libdar figures out the last slice by relying on the filename, not the
| header, although I haven't checked the sourcecode.

Assuming in libdar the mistakes of the users is not a good idea. I can
already hear some users complaining that libdar does not remove unuseful
slices of older archives, while it does not complain anymore that extra
slices exist and do only waste disk space.

I don't like MS-office like programs that tend to have the prentention
to be more clever than their users. They either get much restrictive and
blindly forbid operation needed by more clever users than the program
developers, or automatically do stupid things or have stupid questions
to the user, that even less clever users find borring.

|
| 3) A combination of 1) and 2).

no again. If a user wants to clean its previous archive it is simple:

~  rm old_archive.*.dar
~  dar -c old_arhive ...

nothing more nothing less. Simple, everybody understands, no hidden
features from dar or libdar, no hidden surprises. (lib)dar is only a
backup software.

|
| I see a similar problem when creating archives with KDar. I have a
progress
| bar that periodically updates which slice is currently being written, and
| what that slice's size is. When the "pause" between slices option is
| chosen, and libdar asks if it okay to continue writing the next slice,
| first it writes the "non-terminating" character to the slice header, and
| then waits for the user to answer the question. In the case where the
user
| is overwriting an older archive, the statusbar code runs through the
files
| on disk until it finds the one with the "T" in the header. This will be
| the last one in the older archive, and the statusbar will jump to that
| slice instead of the current one.

To my point of view, this is here again a user mistake. Once you
overwrite the first slice of the old archive, all the other slices of
this archives are useless. Why not removing first all the slices of the
old archive you have planned to overwrite ?

|
| Perhaps libdar should write the "N" to the termination type only if the
| user says it's okay to continue, and write an "I" to the termination type
| if the user cancels. That way we know that the "last" archive slice is
| actually invalid, i.e. the creation process was aborted, or failed in
some
| way.

perhaps, KDAR could propose a "remove old archive slice" option ? ;-)
option, that can simply rely on the rm command or unlink system call...

|
| For the status indicator, the best solution would be for libdar to have a
| "currentSlice" method, which could be called at any time to determine the
| slice number of the current one being written or read.

a new callback function ? ;-)

|
| Should I file a bug report for any of this?

I don't think so, do you still ?

|
| Cheers,
| JB
|


Cheers,
Denis.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBA0OvpC5CI8gYGlIRAhTVAKC2e5HBV+gf0a7r/13u/5kdu+sTRQCePg08
y3Ne+36JhvyFZdVrkv2KncA=
=3y7l
-----END PGP SIGNATURE-----