dar-support — Any questions or problem using dar, this mailing-list is for you !

Re: [Dar-support] Dar on LTO

[John G. <jgo...@co...>]

Hi,

I just thought I'd add my 2 cents here.

Let me start from a different place... Bacula/Bareos are backup programs
designed for tape backup.  They both always spool backup data to disk
before writing it to tape.  For several reasons, but mainly because of
the shoeshining problem.  Now Bacula/Bareos are much more heavyweight
programs, designed for scheduling backups of many hosts on a network
with a centralized catalog.  They're great for that, but if you don't
need that, dar's featureset is a lot nicer.

So, why don't you spool the dar backups to disk?

Now, you may not have enough space to spool the entire backup.  But you
could use the slice options to define a maximum spool size.  Then, add
--execute to your dar command to point to a script.  That script would
then flush a given slice to tape.  It could be as simple as dd, or as
complex as mbuffer.  Then after it's written to tape, the script would
delete it from disk and continue with the next slice.

This will make you wind up with multiple tape files for a single dar
backup.  There are pros and cons to that.  The cons are that you may
have a more complex tape structure, particularly if you were planning to
put multiple backup sessions on one tape.  On the other hand, you will
have an easy way to seek to a given slice, so it may make partial
restores faster.  You can use --execute to interact with mt to seek to
particular tape files and such during restore automatically.  You don't
need to use dar_split; just use dar --slice.

Now then, other advice...

How many GBs can you give mbuffer's buffer?  If you can use -m 10G or -m
20G, that would help reduce shoeshining.  Use -P 99 and mbuffer won't
start writing until the buffer is 99% full, minimizing stop/start
cycles.  Adding a lot of RAM to the system, and using mbuffer -m 40G -P
99 may well reduce shoeshining to acceptable levels easily enough.

I don't see why slowing down the writes to tape would help.  I'd think
that would be harmful.

Also, turn off hardware compression and use dar compression.  Then you
can exactly predict how much space remains on a tape.

If you get into really complicated tape mechanics, then you may be
better suited for something like Bacula/Bareos.  They automate these
things, including splitting backups over tapes, tape recycling schemes,
etc.  The cost is a lot more heavyweight system; you need a SQL database
to hold their catalog (their version of dar_manager, which is required
for them to operate).  You've got to set up and manage daemons, etc.
But if you are talking multi-tape daily backups or fitting multiple
backups on a single tape and keeping track of it all, maybe it's worth
it.

If, on the other hand, you're preparing an archive -- that is, a
long-term storage copy of the data that maybe you refresh once a year --
dar is going to be nicer and gives you a lot more flexibility on
manipulating the archive.


John

On Tue, Sep 17 2024, dar...@sp... wrote:

> Dear Petr, Denis,
>
> I've followed in 2022 your thread about using DAR for backups on tape
> with much interest.
> But never got past the experimental stage.
>
> As you I like to pick it up again.
> And although I've slightly less demanding jobs (only 2x 8Tb compressed
> and a few million files)
> I still need several tapes to make a full backup and that will increase
> in future.
>
> One part of my data consists of a lot of hard-links made by 'dirvish' in
> differential mode.
> That makes for a very slow data rate using tar for backup to tape with
> shoe shining of the drive as consequence.
> I think DAR is a modern equivalent of tar and I like the catalogue
> option to mentioning one so I really want to use it.
>
> I have the same hardware (Tandberg LTO6) with a BTRFS raid1 and a ZFS
> raid1 on spinning rust.
> In my experiments I noticed that going beyond tar -b256 didn't gave any
> improvements.
> Using mbuffer I also tried a lot of different buffer sizes but because
> of the data part with a lot of hard-links
> it always emptied the buffer and the tape drive stopped. And big buffers
> take longer to refill so all in all longer backup times.
> The maximum data rate of this drive (400Mb/s compressed) is awful and
> only reachable with SSD's or better (and probably big files).
>
> For now I'm thinking of piping dar into mbuffer something like (very
> rudimentary) :
>
> /usr/bin/mbuffer <lot of options> -R160M -H -f -o /dev/st0 < <
> (/usr/local/bin/dar -c - -zlzh4 -R${rootpath} -@${catalogue}
> |/usr/local/bin/dar_split split_output )
>
> I can remember that there was a discussion 'End of Tape' was not
> reaching dar_split this way because of mbuffer.
> I like to use mbuffer because of it's logs showing the data rate.
> There are remarkable obervations to make looking at those logs and would
> regret loosing that possibility.
>
> As you can see I compress the data outside the drive and limit the write
> speed to 160Mb/s. Perhaps I will have to lower this even more.
> The idea is that by compressing it beforehand and using a buffer shoe
> shining is reduced to a minimum.
>
> My question is is there a good example how to use mbuffer, dar and
> dar_split in combination with a tape drive ?
> Is it possible to limit the size of dar_split chunks ?
>
> Thanks and with kind regards,
>
> Jonathan.

Re: [Dar-support] Dar on LTO

[Denis C. <dar...@fr...>]

On 17/09/2024 00:37, dar...@sp... wrote:
> Dear Petr, Denis,

Hi Jonathan,

> 
> I've followed in 2022 your thread about using DAR for backups on tape 
> with much interest.
> But never got past the experimental stage.
> 
> As you I like to pick it up again.
> And although I've slightly less demanding jobs (only 2x 8Tb compressed 
> and a few million files)
> I still need several tapes to make a full backup and that will increase 
> in future.
> 
> One part of my data consists of a lot of hard-links made by 'dirvish' in 
> differential mode.
> That makes for a very slow data rate using tar for backup to tape with 
> shoe shining of the drive as consequence.
> I think DAR is a modern equivalent of tar and I like the catalogue 
> option to mentioning one so I really want to use it.
> 
> I have the same hardware (Tandberg LTO6) with a BTRFS raid1 and a ZFS 
> raid1 on spinning rust.
> In my experiments I noticed that going beyond tar -b256 didn't gave any 
> improvements.
> Using mbuffer I also tried a lot of different buffer sizes but because 
> of the data part with a lot of hard-links
> it always emptied the buffer and the tape drive stopped. And big buffers 
> take longer to refill so all in all longer backup times.
> The maximum data rate of this drive (400Mb/s compressed) is awful and 
> only reachable with SSD's or better (and probably big files).
> 
> For now I'm thinking of piping dar into mbuffer something like (very 
> rudimentary) :
> 
> /usr/bin/mbuffer <lot of options> -R160M -H -f -o /dev/st0 < 
> <(/usr/local/bin/dar -c - -zlzh4 -R${rootpath} -@${catalogue} 
> |/usr/local/bin/dar_split split_output )

I'm not sure of the syntax: mbuffer << (dar | dar_split)
I would have naively thought about that instead:
	dar -c - ...| mbuffer ... | dar_split split_output /dev/st0

and at reading time:
   dar_split split_input /dev/st0 | mbuffer ... | dar -c - --sequential-read

> 
> I can remember that there was a discussion 'End of Tape' was not  > reaching dar_split this way because of mbuffer.

I couldn't find this discussion in the mailing-list archive... I have 
probably overlooked it somewhere. But if dar_split is in direct 
"contact" of the device inode (/dev/st0 here) this should not occur, 
while still having mbuffer controlling the speed of the byte flow to or 
from the tape, as above.

What you refer to may rather be the fact that dar_split, when reading 
from a tape, has no intelligence about the dar format and does not know 
when the archive has been completely read, so it will still wait for the 
next tape. Thus, you had to interrupt it manually when dar has finished. 
There is now a -c option to dar_split thanks to which you give the max 
number of tape to read from. And once this number of tape has been 
exhausted, dar_split ends nicely.

> I like to use mbuffer because of it's logs showing the data rate.
> There are remarkable obervations to make looking at those logs and would 
> regret loosing that possibility.

to my point of view, mbuffer is not incompatible with dar nor dar_split. 
So if you need mbuffer for its reach feature set, just use it with dar 
and dar_split. Still you may need to configure dar_split about the block 
size used to read or write data (see its -b option) according the the 
expected block size supported by your device.

> 
> As you can see I compress the data outside the drive and limit the write 
> speed to 160Mb/s. Perhaps I will have to lower this even more.
> The idea is that by compressing it beforehand and using a buffer shoe 
> shining is reduced to a minimum.

note about compression: since release 2.7.0 dar can perform compression 
leveraging multiple CPU cores (using multi-threading). But as 
compression is still performed file per file, the performance gain will 
depend on the average file size under backup and in consequence the 
compression block size used: Selecting a too small compression block 
will lead to a thread overhead that can lead to poor compression 
performances, while selecting a too large compression block will only 
lead files larger than this block size to use more than one CPU core, 
but compression performance should be slightly equivalent to the one you 
get not using multi-threading (see -z and -G options for details).

> 
> My question is is there a good example how to use mbuffer, dar and 
> dar_split in combination with a tape drive ?
> Is it possible to limit the size of dar_split chunks ?

It is possible to limit the amount of byte per system call (= size of 
the block the device will have to read or write at once, see -b option 
as mentioned above) and to limit the byte rate (see -r option). But if 
you plan to use mbuffer, you should avoid limiting dar_split rate, this 
will be done by mbuffer being piped between dar and dar_split as 
decribed above. And with mbuffer you would still have all the 
stats/indicators you like.

note that I have no more tape in my environment (and this for decades! 
dar came from that need to efficiently rely on disks :) ) Thus, I let 
those having more experience with tape drives to amend or complement my 
suggestions here.

> 
> Thanks and with kind regards,
> 
> Jonathan.
> 
> 
> 

Regards,
Denis

Re: [Dar-support] Dar on LTO

[<dar...@sp...>]

Dear Petr, Denis,

I've followed in 2022 your thread about using DAR for backups on tape 
with much interest.
But never got past the experimental stage.

As you I like to pick it up again.
And although I've slightly less demanding jobs (only 2x 8Tb compressed 
and a few million files)
I still need several tapes to make a full backup and that will increase 
in future.

One part of my data consists of a lot of hard-links made by 'dirvish' in 
differential mode.
That makes for a very slow data rate using tar for backup to tape with 
shoe shining of the drive as consequence.
I think DAR is a modern equivalent of tar and I like the catalogue 
option to mentioning one so I really want to use it.

I have the same hardware (Tandberg LTO6) with a BTRFS raid1 and a ZFS 
raid1 on spinning rust.
In my experiments I noticed that going beyond tar -b256 didn't gave any 
improvements.
Using mbuffer I also tried a lot of different buffer sizes but because 
of the data part with a lot of hard-links
it always emptied the buffer and the tape drive stopped. And big buffers 
take longer to refill so all in all longer backup times.
The maximum data rate of this drive (400Mb/s compressed) is awful and 
only reachable with SSD's or better (and probably big files).

For now I'm thinking of piping dar into mbuffer something like (very 
rudimentary) :

/usr/bin/mbuffer <lot of options> -R160M -H -f -o /dev/st0 < 
<(/usr/local/bin/dar -c - -zlzh4 -R${rootpath} -@${catalogue} 
|/usr/local/bin/dar_split split_output )

I can remember that there was a discussion 'End of Tape' was not 
reaching dar_split this way because of mbuffer.
I like to use mbuffer because of it's logs showing the data rate.
There are remarkable obervations to make looking at those logs and would 
regret loosing that possibility.

As you can see I compress the data outside the drive and limit the write 
speed to 160Mb/s. Perhaps I will have to lower this even more.
The idea is that by compressing it beforehand and using a buffer shoe 
shining is reduced to a minimum.

My question is is there a good example how to use mbuffer, dar and 
dar_split in combination with a tape drive ?
Is it possible to limit the size of dar_split chunks ?

Thanks and with kind regards,

Jonathan.

Re: [Dar-support] Can't update libthreadar

[Denis C. <dar...@fr...>]

Thank you Martin for your investigation and time

Have a good day,
Denis

On 03/06/2024 10:43, Marcin Wieczorek wrote:
> Hello Martin,
> 
> On 24/05/26 17:30, Dr. Martin Senftleben wrote:
>> .... FAILED
>> ==> Error: One or more files didn't pass the validity check!
> 
> This error is caused by an error in AUR package, which is Arch Linux
> User Repository. That package is maintained by the community.
> The only way to fix the issue is to contact @frekky at fre...@gm....
> 
> To fix the package yourself (locally):
> git clone https://aur.archlinux.org/libthreadar.git
> cd libthreadar
> updpkgsums
> makepkg -si
> 
> then you can use yay as you normally would to update other packages.
> 
> Best regards,
> Marcin Wieczorek
> 
> 

Re: [Dar-support] Can't update libthreadar

[Marcin W. <da...@ma...>]

Hello Martin,

On 24/05/26 17:30, Dr. Martin Senftleben wrote:
> .... FAILED
> ==> Error: One or more files didn't pass the validity check!

This error is caused by an error in AUR package, which is Arch Linux
User Repository. That package is maintained by the community.
The only way to fix the issue is to contact @frekky at fre...@gm....

To fix the package yourself (locally):
git clone https://aur.archlinux.org/libthreadar.git
cd libthreadar
updpkgsums
makepkg -si

then you can use yay as you normally would to update other packages.

Best regards,
Marcin Wieczorek

Re: [Dar-support] Can't update libthreadar

[Denis C. <dar...@fr...>]

On 26/05/2024 17:30, Dr. Martin Senftleben wrote:
> Hi,

Hi Martin,

> 
> I use Manjaro (testing) and just try to install the latest updates, 
> among them libthreadar. I get the following error message:
> 
> libthreadar-1.4.0.tar.gz ... FEHLGESCHLAGEN
> ==> FEHLER: Eine oder mehrere Dateien überstanden nicht die 
> Gültigkeits-Prüfung!
> 
> Translation:
> .... FAILED
> ==> Error: One or more files didn't pass the validity check!
> 
> What could be wrong?

The message does not tell what's the Manjaro distro building process 
find invalid in libthreadar-1.4.0.tar.gz source package.

Maybe there is a log file generated during the installation process?
This would help understanding.

> 
> Regards
> 
> Martin
> 

Regards,
Denis

[Dar-support] Can't update libthreadar

[Dr. M. S. <li...@dr...>]

Hi,

I use Manjaro (testing) and just try to install the latest updates, 
among them libthreadar. I get the following error message:

libthreadar-1.4.0.tar.gz ... FEHLGESCHLAGEN
==> FEHLER: Eine oder mehrere Dateien überstanden nicht die 
Gültigkeits-Prüfung!

Translation:
.... FAILED
==> Error: One or more files didn't pass the validity check!

What could be wrong?

Regards

Martin

Re: [Dar-support] Error Compiling on macOS

[Denis C. <dar...@fr...>]

Hi James,

OK super, thanks for your help!

Regards,
Denis

On 01/05/2024 18:46, James Pedersen wrote:
> Denis,
> 
> This error was not present when I compiled dar 2.7.15.RC1 on my other 
> MacOS Mojave laptop today.
> 
> James Pedersen
> 
> On Tue, Apr 30, 2024 at 6:20 AM Rolf Gebhardt <rg...@no... 
> <mailto:rg...@no...>> wrote:
> 
>     Hi Denis, hi James,
> 
>     first of all, I totally agree that adding the noexcept specification to
>     the smart_xxx class templates can not be a solution for the future.
>     It's
>     no more than a patch to get dar/libdar compiled on an outdated compiler
>     with outdated standard libraries on an outdateted operating system.
> 
>     And because - as I told you - I'm not very familiar with exception
>     specification and the ways they are used in C++ - as far as I
>     understood, the meaning "it's impossible that this function throws" is
>     not the only one - I can not contribute much to this discussion.
> 
>     The only thing I want to say was already mentioned in the first sentenc
>     above: It's a problem  with an outdated compiler with outdated standard
>     libraries on an outdateted operating system. MacOS Mojave is not
>     supported by apple any more and Apples development tools and libraries
>     either won't get any updates any more. The only reason to use it, is if
>     you have an old Mac that cannot be updated to newer MacOs-Versions.
> 
>     On the current version MacOS with the corresponding tools and libraries
>     dar compiles fine.
> 
>     @Denis: Just to prevent you from spending hours or days of your
>     valuable
>     time to solve problems of the past.
> 
>     Kind regards to all,
> 
>     Rolf
> 
> 
>     Am 27.04.24 um 21:15 schrieb Denis Corbin:
>      > On 25/04/2024 01:39, James Pedersen wrote:
>      >> Hi Denis and Rolf,
>      >
>      > Hi James,
>      >
>      >>
>      >> Thank you for getting back to me about this as well as for
>     suggesting
>      >> some solutions to this issue.
>      >>
>      >> Denis, following your advice, I removed the "noexcept" statement
>     from
>      >> the move assignment operator definition in a bunch of the .hpp
>     files
>      >> in src/libdar/, and then the errors went away.
>      >>
>      >> Rolf, I think that you have made a good observation here. In the
>      >> definition of the smart_pointer move assignment operator in
>      >> smart_pointer.hpp, it seems that it can throw exceptions because
>      >> /del_ref() /can throw exceptions. cat_nomme.hpp and cat_detruit.hpp
>      >> are subclasses of cat_entree.hpp, and hence it seems that they
>     would
>      >> both inherit the smart_pointer variable pdesc of cat_entree.hpp.
>      >> Since the move assignment operator of smart_pointer can throw
>      >> exception/s/, I would like to suggest that maybe the right solution
>      >> to this is to just remove the "noexcept" keyword for any move
>      >> assignment operator of a subclass of cat_entree.hpp?
>      >
>      > Yes, I agree with your analysis, smart_pointer/smart_node may throw
>      > exception thus the absence of noexcept keyword here is normal.
>      >
>      > I will go that way. first I have to reproduce the problem (which I
>      > failed under Devuan using clang, will try other distro/clang
>     version)
>      > to be sure the fix to be exhaustive.
>      >
>      >>
>      >> Anyways, thank you both for helping me with this,
>      >> James Pedersen
>      >>
>      >
>      > Regards,
>      > Denis
>      >>
>      >
>      >>
>      >>
>      >>
>      >>
>      >>
>      >
> 
> 

Re: [Dar-support] Error Compiling on macOS

[James P. <jam...@gm...>]

Denis,

This error was not present when I compiled dar 2.7.15.RC1 on my other MacOS
Mojave laptop today.

James Pedersen

On Tue, Apr 30, 2024 at 6:20 AM Rolf Gebhardt <rg...@no...> wrote:

> Hi Denis, hi James,
>
> first of all, I totally agree that adding the noexcept specification to
> the smart_xxx class templates can not be a solution for the future. It's
> no more than a patch to get dar/libdar compiled on an outdated compiler
> with outdated standard libraries on an outdateted operating system.
>
> And because - as I told you - I'm not very familiar with exception
> specification and the ways they are used in C++ - as far as I
> understood, the meaning "it's impossible that this function throws" is
> not the only one - I can not contribute much to this discussion.
>
> The only thing I want to say was already mentioned in the first sentenc
> above: It's a problem  with an outdated compiler with outdated standard
> libraries on an outdateted operating system. MacOS Mojave is not
> supported by apple any more and Apples development tools and libraries
> either won't get any updates any more. The only reason to use it, is if
> you have an old Mac that cannot be updated to newer MacOs-Versions.
>
> On the current version MacOS with the corresponding tools and libraries
> dar compiles fine.
>
> @Denis: Just to prevent you from spending hours or days of your valuable
> time to solve problems of the past.
>
> Kind regards to all,
>
> Rolf
>
>
> Am 27.04.24 um 21:15 schrieb Denis Corbin:
> > On 25/04/2024 01:39, James Pedersen wrote:
> >> Hi Denis and Rolf,
> >
> > Hi James,
> >
> >>
> >> Thank you for getting back to me about this as well as for suggesting
> >> some solutions to this issue.
> >>
> >> Denis, following your advice, I removed the "noexcept" statement from
> >> the move assignment operator definition in a bunch of the .hpp files
> >> in src/libdar/, and then the errors went away.
> >>
> >> Rolf, I think that you have made a good observation here. In the
> >> definition of the smart_pointer move assignment operator in
> >> smart_pointer.hpp, it seems that it can throw exceptions because
> >> /del_ref() /can throw exceptions. cat_nomme.hpp and cat_detruit.hpp
> >> are subclasses of cat_entree.hpp, and hence it seems that they would
> >> both inherit the smart_pointer variable pdesc of cat_entree.hpp.
> >> Since the move assignment operator of smart_pointer can throw
> >> exception/s/, I would like to suggest that maybe the right solution
> >> to this is to just remove the "noexcept" keyword for any move
> >> assignment operator of a subclass of cat_entree.hpp?
> >
> > Yes, I agree with your analysis, smart_pointer/smart_node may throw
> > exception thus the absence of noexcept keyword here is normal.
> >
> > I will go that way. first I have to reproduce the problem (which I
> > failed under Devuan using clang, will try other distro/clang version)
> > to be sure the fix to be exhaustive.
> >
> >>
> >> Anyways, thank you both for helping me with this,
> >> James Pedersen
> >>
> >
> > Regards,
> > Denis
> >>
> >
> >>
> >>
> >>
> >>
> >>
> >
>
>

Re: [Dar-support] Error Compiling on macOS

[Rolf G. <rg...@no...>]

Hi Denis, hi James,

first of all, I totally agree that adding the noexcept specification to 
the smart_xxx class templates can not be a solution for the future. It's 
no more than a patch to get dar/libdar compiled on an outdated compiler 
with outdated standard libraries on an outdateted operating system.

And because - as I told you - I'm not very familiar with exception 
specification and the ways they are used in C++ - as far as I 
understood, the meaning "it's impossible that this function throws" is 
not the only one - I can not contribute much to this discussion.

The only thing I want to say was already mentioned in the first sentenc 
above: It's a problem  with an outdated compiler with outdated standard 
libraries on an outdateted operating system. MacOS Mojave is not 
supported by apple any more and Apples development tools and libraries 
either won't get any updates any more. The only reason to use it, is if 
you have an old Mac that cannot be updated to newer MacOs-Versions.

On the current version MacOS with the corresponding tools and libraries 
dar compiles fine.

@Denis: Just to prevent you from spending hours or days of your valuable 
time to solve problems of the past.

Kind regards to all,

Rolf


Am 27.04.24 um 21:15 schrieb Denis Corbin:
> On 25/04/2024 01:39, James Pedersen wrote:
>> Hi Denis and Rolf,
>
> Hi James,
>
>>
>> Thank you for getting back to me about this as well as for suggesting 
>> some solutions to this issue.
>>
>> Denis, following your advice, I removed the "noexcept" statement from 
>> the move assignment operator definition in a bunch of the .hpp files 
>> in src/libdar/, and then the errors went away.
>>
>> Rolf, I think that you have made a good observation here. In the 
>> definition of the smart_pointer move assignment operator in 
>> smart_pointer.hpp, it seems that it can throw exceptions because 
>> /del_ref() /can throw exceptions. cat_nomme.hpp and cat_detruit.hpp 
>> are subclasses of cat_entree.hpp, and hence it seems that they would 
>> both inherit the smart_pointer variable pdesc of cat_entree.hpp. 
>> Since the move assignment operator of smart_pointer can throw 
>> exception/s/, I would like to suggest that maybe the right solution 
>> to this is to just remove the "noexcept" keyword for any move 
>> assignment operator of a subclass of cat_entree.hpp?
>
> Yes, I agree with your analysis, smart_pointer/smart_node may throw 
> exception thus the absence of noexcept keyword here is normal.
>
> I will go that way. first I have to reproduce the problem (which I 
> failed under Devuan using clang, will try other distro/clang version) 
> to be sure the fix to be exhaustive.
>
>>
>> Anyways, thank you both for helping me with this,
>> James Pedersen
>>
>
> Regards,
> Denis
>>
>
>>
>>
>>
>>
>>
>

Re: [Dar-support] Error Compiling on macOS

[Denis C. <dar...@fr...>]

On 25/04/2024 01:39, James Pedersen wrote:
> Hi Denis and Rolf,

Hi James,

> 
> Thank you for getting back to me about this as well as for suggesting 
> some solutions to this issue.
> 
> Denis, following your advice, I removed the "noexcept" statement from 
> the move assignment operator definition in a bunch of the .hpp files in 
> src/libdar/, and then the errors went away.
> 
> Rolf, I think that you have made a good observation here. In the 
> definition of the smart_pointer move assignment operator in 
> smart_pointer.hpp, it seems that it can throw exceptions because 
> /del_ref() /can throw exceptions. cat_nomme.hpp and cat_detruit.hpp are 
> subclasses of cat_entree.hpp, and hence it seems that they would both 
> inherit the smart_pointer variable pdesc of cat_entree.hpp. Since the 
> move assignment operator of smart_pointer can throw exception/s/, I 
> would like to suggest that maybe the right solution to this is to just 
> remove the "noexcept" keyword for any move assignment operator of a 
> subclass of cat_entree.hpp?

Yes, I agree with your analysis, smart_pointer/smart_node may throw 
exception thus the absence of noexcept keyword here is normal.

I will go that way. first I have to reproduce the problem (which I 
failed under Devuan using clang, will try other distro/clang version) to 
be sure the fix to be exhaustive.

> 
> Anyways, thank you both for helping me with this,
> James Pedersen
> 

Regards,
Denis
> 

> 
> 
> 
> 
> 

Re: [Dar-support] Error Compiling on macOS

[James P. <jam...@gm...>]

Hi Denis and Rolf,

Thank you for getting back to me about this as well as for suggesting some
solutions to this issue.

Denis, following your advice, I removed the "noexcept" statement from the
move assignment operator definition in a bunch of the .hpp files in
src/libdar/, and then the errors went away.

Rolf, I think that you have made a good observation here. In the definition
of the smart_pointer move assignment operator in smart_pointer.hpp, it
seems that it can throw exceptions because *del_ref() *can throw
exceptions. cat_nomme.hpp and cat_detruit.hpp are subclasses of
cat_entree.hpp, and hence it seems that they would both inherit the
smart_pointer variable pdesc of cat_entree.hpp. Since the move assignment
operator of smart_pointer can throw exception*s*, I would like to suggest
that maybe the right solution to this is to just remove the "noexcept"
keyword for any move assignment operator of a subclass of cat_entree.hpp?

Anyways, thank you both for helping me with this,
James Pedersen

Re: [Dar-support] Error Compiling on macOS

[Denis C. <dar...@fr...>]

On 24/04/2024 17:50, Rolf Gebhardt wrote:
> Hi James, hi Denis,

Hi Rolf,

> 
> because I have some macs under my desk, i tried to reproduce the error. 
> (Usually I don't compile Dar on MacOS and use the homebrew version 
> instead.)
> 
> On my new mac mini (MacOS Sonoma) the error does not occur.
> 
> So I booted Mojave on my old Mac Pro (normally I run Ubuntu on it) and 
> got the error reported by you.
> 
> It seems to have to do with one member of cat_entree: 
> smart_pointer<pile_descriptor> pdesc;
> 
> The class template "smart_pointer" itself  has a non default move 
> assignment operator that is not declared "noexcept". If you edit 
> smart_pointer.hpp and add the "noexcept"-specification to it, dar 
> compiles fine on my machine.
> 
>      /// move assignment operator
>      smart_pointer & operator = (smart_pointer && ref) noexcept
>      {
> 
> Perhaps this helps.

definitively yes, it should help James now and will reduce my 
investigation time later, thanks for that!

> 
> Don't know, whether stating the error is a compiler bug or just a 
> behaviour allowed by the language specification. I am not very familiar 
> with exception specifications, neither with the old and deprecated 
> "throws", nor with the newer "noexcept".

Yes, at the beginning of this century, dar/libdar made use of 
throws(<list of exception>) declarations, it was painful because you 
could forget a type of exception thrown indirectly by some method of 
function called from another method calling another one and so on.

the noexcept is simpler and easier to maintain (you just put it when you 
have made the necessary for no exception to be thrown, this leads to 
some possible optimization in the standard library). But the point here 
is the specifications of the default move, copy constructor, move and 
copy assignment operators. If not specified with the "= default" at the 
end I guess the compiler will generate the expected thing without 
warning, but for some reason I wanted to systematically have them 
mentionned in all classes, either defined with specific implementation 
or deleted ( = delete;) or use the default implementation ( = default;).

However C++ is changing over time (and that's good sign of life for this 
wondeful, poweful, lovely language ;) ) and compilers do not implement 
at the same pace the updated expectations/specifications of the 
language. some like clang are stricter than others like cpp... even if 
the code complied with both at a time... this changes a few years 
later... for one or the other or both.

so in short, this is probably not a compiler error, dar/libdar code 
needs to be updated from time to time to compile with modern 
compilers... and it will be.

> 
> Regards,
> 
> Rolf
> 
> 
[...]

Regards,
Denis

Re: [Dar-support] Error Compiling on macOS

[Rolf G. <rg...@no...>]

Hi James, hi Denis,

because I have some macs under my desk, i tried to reproduce the error. 
(Usually I don't compile Dar on MacOS and use the homebrew version instead.)

On my new mac mini (MacOS Sonoma) the error does not occur.

So I booted Mojave on my old Mac Pro (normally I run Ubuntu on it) and 
got the error reported by you.

It seems to have to do with one member of cat_entree: 
smart_pointer<pile_descriptor> pdesc;

The class template "smart_pointer" itself  has a non default move 
assignment operator that is not declared "noexcept". If you edit 
smart_pointer.hpp and add the "noexcept"-specification to it, dar 
compiles fine on my machine.

     /// move assignment operator
     smart_pointer & operator = (smart_pointer && ref) noexcept
     {

Perhaps this helps.

Don't know, whether stating the error is a compiler bug or just a 
behaviour allowed by the language specification. I am not very familiar 
with exception specifications, neither with the old and deprecated 
"throws", nor with the newer "noexcept".

Regards,

Rolf


Am 24.04.24 um 15:54 schrieb Denis Corbin:
> On 24/04/2024 04:45, James Pedersen wrote:
>> Hi,
>
> Hi James,
>
>>
>> When I try to compile dar 2.7.14 on MacOS Mojave, I get the following 
>> error:
>>
>> /In file included from archive.cpp:31/
>> /In file included from ./i_archive.hpp:46:/
>> /In file included from ./catalogue.hpp:46:/
>> /./cat_entree.hpp:104:15: error: exception specification of 
>> explicitly defaulted move assignment operator does not match the 
>> calculated one/
>> /    cat_entree & operator = (cat_entree && ref) noexcept = default;/
>> /
>> /
>> /In file included from archive.cpp:31:/
>> /In file included from ./i_archive.hpp:46:/
>> /In file included from ./catalogue.hpp:47:
>> /
>> /./cat_nomme.hpp:52:14: error: exception specification of explicitly 
>> defaulted move assignment operator does not match the calculated one/
>> /    cat_nomme & operator = (cat_nomme && ref) noexcept = default;/
>> /
>> /
>> /2 errors generated./
>> /make[3]: *** [archive.lo] Error 1/
>> /make[2]: *** [all-recursive] Error 1/
>> /make[1]: *** [all-recursive] Error 1/
>> /Make:  *** [all] Error 2
>> /
>> /
>> /
>> How can I fix this error?
>
> cat_entree and car_nomme classes have not changed since several 
> releases, it seems related to the compiler you are using (probably 
> clang?).
>
> There has been changed in the past on whether a default copy and move 
> assignment operator could have "noexcept" flag or not. g++ and clang 
> are not always in sync on that, some years ago the noexcept flag 
> needed to be removed:
>
> https://stackoverflow.com/questions/41620922/default-noexcept-move-semantics 
>
>
> now it seems clang wants it to be added...
>
> I will figure out what the new status of things for next release.
>
> I'll probably have to enhance the configure script to check the 
> behavior of the compiler used in regard of that type of hysterical 
> complain... and have the code adapting on it.
>
> In the meanwhile several path can be followed:
> - using gcc instead of clang
> - finding a flag in clang to disable this error (not sure it exists 
> see below)
> - edit libdar code adding (or rather removing) the "noexcept" keyworld 
> in libdar code where clang is coughing... cat_entree.hpp line 104, 
> cat_nomme.hpp line 52 and so on.
>
> how to play with compiler flags (second point above):
>
> export CXXFLAGS=-std=c++14
> ./configure ....
> make ...
>
> replace/add to -std=c++14 other flag if you find some more pertinent 
> or interesting
>
>
>>
>> Thank you,
>> James Pedersen
>>
>>
>
> Sorry to not provide a better solution for today
>
> Regards,
> Denis
>

Re: [Dar-support] Error Compiling on macOS

[Denis C. <dar...@fr...>]

On 24/04/2024 04:45, James Pedersen wrote:
> Hi,

Hi James,

> 
> When I try to compile dar 2.7.14 on MacOS Mojave, I get the following 
> error:
> 
> /In file included from archive.cpp:31/
> /In file included from ./i_archive.hpp:46:/
> /In file included from ./catalogue.hpp:46:/
> /./cat_entree.hpp:104:15: error: exception specification of explicitly 
> defaulted move assignment operator does not match the calculated one/
> /    cat_entree & operator = (cat_entree && ref) noexcept = default;/
> /
> /
> /In file included from archive.cpp:31:/
> /In file included from ./i_archive.hpp:46:/
> /In file included from ./catalogue.hpp:47:
> /
> /./cat_nomme.hpp:52:14: error: exception specification of explicitly 
> defaulted move assignment operator does not match the calculated one/
> /    cat_nomme & operator = (cat_nomme && ref) noexcept = default;/
> /
> /
> /2 errors generated./
> /make[3]: *** [archive.lo] Error 1/
> /make[2]: *** [all-recursive] Error 1/
> /make[1]: *** [all-recursive] Error 1/
> /Make:  *** [all] Error 2
> /
> /
> /
> How can I fix this error?

cat_entree and car_nomme classes have not changed since several 
releases, it seems related to the compiler you are using (probably clang?).

There has been changed in the past on whether a default copy and move 
assignment operator could have "noexcept" flag or not. g++ and clang are 
not always in sync on that, some years ago the noexcept flag needed to 
be removed:

https://stackoverflow.com/questions/41620922/default-noexcept-move-semantics

now it seems clang wants it to be added...

I will figure out what the new status of things for next release.

I'll probably have to enhance the configure script to check the behavior 
of the compiler used in regard of that type of hysterical complain... 
and have the code adapting on it.

In the meanwhile several path can be followed:
- using gcc instead of clang
- finding a flag in clang to disable this error (not sure it exists see 
below)
- edit libdar code adding (or rather removing) the "noexcept" keyworld 
in libdar code where clang is coughing... cat_entree.hpp line 104, 
cat_nomme.hpp line 52 and so on.

how to play with compiler flags (second point above):

export CXXFLAGS=-std=c++14
./configure ....
make ...

replace/add to -std=c++14 other flag if you find some more pertinent or 
interesting


> 
> Thank you,
> James Pedersen
> 
> 

Sorry to not provide a better solution for today

Regards,
Denis

[Dar-support] Error Compiling on macOS

[James P. <jam...@gm...>]

Hi,

When I try to compile dar 2.7.14 on MacOS Mojave, I get the following
error:

*In file included from archive.cpp:31*
*In file included from ./i_archive.hpp:46:*
*In file included from ./catalogue.hpp:46:*
*./cat_entree.hpp:104:15: error: exception specification of explicitly
defaulted move assignment operator does not match the calculated one*
*    cat_entree & operator = (cat_entree && ref) noexcept = default;*

*In file included from archive.cpp:31:*
*In file included from ./i_archive.hpp:46:*

*In file included from ./catalogue.hpp:47:*
*./cat_nomme.hpp:52:14: error: exception specification of explicitly
defaulted move assignment operator does not match the calculated one*
*    cat_nomme & operator = (cat_nomme && ref) noexcept = default;*

*2 errors generated.*
*make[3]: *** [archive.lo] Error 1*
*make[2]: *** [all-recursive] Error 1*
*make[1]: *** [all-recursive] Error 1*

*Make:  *** [all] Error 2 *

How can I fix this error?

Thank you,
James Pedersen

Re: [Dar-support] dar_manager - incremental backups and removed file

[Denis C. <dar...@fr...>]

On 22/04/2024 08:34, J. Roeleveld via Dar-support wrote:
> On Sunday, 21 April 2024 21:28:48 CEST Denis Corbin wrote:
>> On 20/04/2024 15:23, Joost Roeleveld via Dar-support wrote:
>>
> [...]

[...]

>>
>> So my question is now about the snapshot process and the way you expose
>> it to dar.
> 
> commands run (simplistically, nothing special is actually performed):
> # zfs snapshot ....
> # mount -oro /dev/zvol/.....  /mnt/darroot
> # dar .... /mnt/darroot
> 
>> Could you issue the following command for example in such snapshot:
>>
>> stat /usr/share/fonts/hack
> 
> # stat /usr/share/fonts/hack
>    File: /usr/share/fonts/hack
>    Size: 4096            Blocks: 8          IO Block: 4096   directory
> Device: 202,1   Inode: 532613      Links: 2
> Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
> Access: 2022-08-02 08:56:46.000000000 +0000
> Modify: 2021-06-03 13:32:10.000000000 +0000
> Change: 2024-04-08 06:13:39.937335727 +0000
>   Birth: 2017-10-26 11:41:16.541269703 +0000

OK, good to know the dates are rounded up outside dar and dar_manager

> 
> [...]
> 
>> In conclusion so far, the explanation of the warning issued by
>> dar_manager is caused by a loss of time precision and is not a bug in
>> dar_manager, maybe still possible a bug in dar but that is not the most
>> probable explanation IMHO.
> 
> I can't fully check the actual timeline and what happened as not all logs are
> kept this long, which currently is quite annoying. I wish I knew how to
> reproduce this. Going forward, I have increased log-retention and if it occurs
> again, I should have a better idea on how to reproduce this issue.
> 
> Looking at the output of the "stat" command, I wonder if "tar", by default,
> only stores second-level precision, leading to the ".000000000 +0000" I see
> for "access" and "modify"

by default tar stores at one second accuracy. I did a benchmark some 
years ago, comparing features of dar, rsync and tar and getting back to 
it at http://dar.linux.free.fr/doc/benchmark.html I read that you can 
enhance tar time precision using the --xattrs argument, but it also 
depends on the tar version... according to the test logs at
http://dar.linux.free.fr/doc/benchmark_logs.html I was using tar GNU tar 
1.30

> 
> The filesystem has been the same for years and the OS is simply being kept
> uptodate over time.
> 
> What does "dar" do when it sees ".000000000 +0000" at the end? Does it reduce
> the precision to match the lowest non-zero field?

Very good question! Looking at the code, dar reduces to the largest time 
unit (microsecond or even seconds), so the time precision reduction is 
probably due to dar hitting the .0000000 fractions of a second.

according the the integer type used to store the date, take the largest 
possible time unit when possible reduces:
- slightly the time needed to compare dates
- less probably the memory footprint (due to block size allocation)
- mostly the size of the resulting dar backup/archive

> 
> If the above doesn't yield anything pointing to a specific issue in dar/
> dar_manager, I would suggest we leave this as-is and wait for it to happen
> again.

OK, let's do that way.

> 
> --
> Joost
> 
> 
> 

Denis

Re: [Dar-support] dar_manager - incremental backups and removed file

[J. R. <jo...@an...>]

On Sunday, 21 April 2024 21:28:48 CEST Denis Corbin wrote:
> On 20/04/2024 15:23, Joost Roeleveld via Dar-support wrote:
> 
[...]
> 
> >   Timestamp read accuracy : 1 nanosecond
> >   Timestamp write accuracy : 1 nanosecond
> 
> this is above the info I wanted to check and this is not what I 
> expected, but OK. The problem seen is not caused by the way dar has been 
> compiled and seems to read dates from the filesystem the same manner
> over the time, I mean it has always read date from the fields of the 
> system "struct stat" returned by lstat() system call:
> - stat::st_atim.tv_sec
> - stat::st_mtim.tv_sec
> - stat::st_ctim.tv_sec
> 
> as implemented in src/libdar/filesystem_hard_link_read.cpp
> 
> [...]
> 
> > 
> >> Have you recently recompiled or upgraded the dar binary used here to 
> >> make backups on that host?
> 
> Here is why I ask, I'll take just the usr/share/fonts/hack/.uuid file to 
> illustrate:
> 
> [Sorry for the ugly formated output... too wide to fit in email]
> 
> # dar -l MNGR_20240301T000001_san1__zdata_os_services_binhost_root -g
> usr/share/fonts/hack -afdd
> Warning: using insecure memory!
> [Data ][D][ EA  ][FSA][Compr][S]| Permission | User  | Group | Size    |
>           Date                 |    filename
> --------------------------------+------------+-------+-------+---------+----
> ---------------------------+------------
 [InRef][-]       [-L-][  70%][ ] 
> drwxr-xr-x   root     root    5 Gio Mon Jul  1 11:34:26 2019 + 370501096 ns
> usr
> [InRef][-]       [-L-][  65%][ ]  drwxr-xr-x   root     root    872 Mio
> Mon Feb 19 17:56:59 2024 + 297538137 ns usr/share
> [InRef][-]       [-L-][  48%][ ]  drwxr-xr-x   root     root    182 Mio
> Thu Jun  3 15:32:10 2021 + 211378 us    usr/share/fonts
> [InRef][-]       [-L-][   0%][ ]  drwxr-xr-x   root     root    36 o
> Thu Jun  3 15:32:10 2021 + 301378 us    usr/share/fonts/hack
> [InRef][ ]       [-L-][-----][ ]  -rw-r--r--   root     root    36 o
> Thu Jun  3 15:32:10 2021 + 301378785 ns usr/share/fonts/hack/.uuid
> 
>  From the above output of the oldest archive, all dates have nanosecond 
> precision [the -afdd option has been added to display *F*ully *D*etailed 
> *D*ates. This option will be available with release 2.8.0 for dar and 
> dar_manager]
> 
> 
> # dar -l MNGR_20240401T000002_san1__zdata_os_services_binhost_root -g
> usr/share/fonts/hack -afdd
> Warning: using insecure memory!
> [Data ][D][ EA  ][FSA][Compr][S]| Permission | User  | Group | Size    |
>           Date                 |    filename
> --------------------------------+------------+-------+-------+---------+----
> ---------------------------+------------
 [InRef][-]       [-L-][  71%][ ] 
> drwxr-xr-x   root     root    4 Gio Thu Mar 28 10:57:11 2024 + 379415752 ns
> usr
> [InRef][-]       [-L-][  66%][ ]  drwxr-xr-x   root     root    832 Mio
> Thu Mar 28 07:53:02 2024 + 534986022 ns usr/share
> [     ][-]       [-l-][  48%][ ]  drwxr-xr-x   root     root    182 Mio
> Thu Jun  3 15:32:10 2021 + 211378 us    usr/share/fonts
> [     ][-]       [-l-][     ][ ]  drwxr-xr-x   root     root    0
> Thu Jun  3 15:32:10 2021 + 301378 us    usr/share/fonts/hack
> [     ][ ]       [-l-][-----][ ]  -rw-r--r--   root     root    36 o
> Thu Jun  3 15:32:10 2021 + 301378785 ns usr/share/fonts/hack/.uuid
> 
> But for the second backup above, the precision changed to microsecond 
> (displayed "us" above) for usr/share/fonts and usr/share/fonts/hack but 
> is kept as nonaseconds for usr/share/fonts/.hack and usr.
> 
> Reading code on how differential backup behave about dates, 
> (src/libdar/filtre.cpp line 790), a cat_inode object is created reading 
> information from the filesystem, which mtime is compared with the one of 
> the cat_inode taken from the archive of reference. The comparison takes 
> into account the possible date precision difference (datetime::loose_diff).
> If the date is equal inode created from the filesystem just drops
> attributes to its data but keeps the date and its accuracy from the
> filesystem.
> 
> Why microsecnfs for usr/share/fonts  but not for 
> usr/share/fonts/hack/.uuid ? For libdar there is not code difference for 
> directory and other inodes concerning date handling (same cat_inode is 
> the ancestor class that handles all that is common to inodes, in 
> particular dates).
> 
> Same for the third backup below:
> 
> # dar -l MNGR_20240407T000001_san1__zdata_os_services_binhost_root -g
> usr/share/fonts/hack -afdd
> Warning: using insecure memory!
> [Data ][D][ EA  ][FSA][Compr][S]| Permission | User  | Group | Size    |
>           Date                 |    filename
> --------------------------------+------------+-------+-------+---------+----
> ---------------------------+------------
 [     ][-]       [---][     ][ ] 
> drwxr-xr-x   root     root    0 Thu Mar 28 10:57:11 2024 + 379415752 ns usr
> [     ][-]       [---][     ][ ]  drwxr-xr-x   root     root    0
> Thu Mar 28 07:53:02 2024 + 534986022 ns usr/share
> [     ][-]       [---][     ][ ]  drwxr-xr-x   root     root    0
> Thu Jun  3 15:32:10 2021 + 211378 us    usr/share/fonts
> [     ][-]       [---][     ][ ]  drwxr-xr-x   root     root    0
> Thu Jun  3 15:32:10 2021 + 301378 us    usr/share/fonts/hack
> [     ][ ]       [---][-----][ ]  -rw-r--r--   root     root    36 o
> Thu Jun  3 15:32:10 2021 + 301378785 ns usr/share/fonts/hack/.uuid
> 
> Finally for the fourth backup, the precision has even dropped one step 
> further, to second for usr/share, usr/share/fonts and the deleted entry 
> holding place of usr/share/fonts/hack/.uuid which date is read at the 
> time of the backup from the mtime value of the parent directory:
> 
> # dar -l MNGR_20240414T000001_san1__zdata_os_services_binhost_root -g
> usr/share/fonts/hack -afdd
> Warning: using insecure memory!
> [Data ][D][ EA  ][FSA][Compr][S]| Permission | User  | Group | Size    |
>           Date                 |    filename
> --------------------------------+------------+-------+-------+---------+----
> ---------------------------+------------
 [     ][-]       [---][  74%][ ] 
> drwxr-xr-x   root     root    450 Mio Thu Mar 28 10:57:11 2024 + 379415752
> ns usr
> [InRef][-]       [-L-][  91%][ ]  drwxr-xr-x   root     root    233 Mio
> Mon Apr  8 13:19:02 2024 + 244801698 ns usr/share
> [     ][-]       [-L-][     ][ ]  drwxr-xr-x   root     root    0
> Thu Jun  3 15:32:10 2021        usr/share/fonts
> [     ][-]       [-L-][     ][ ]  drwxr-xr-x   root     root    0
> Thu Jun  3 15:32:10 2021        usr/share/fonts/hack
> [--- REMOVED ENTRY ----] (Thu Jun  3 15:32:10 2021) [-]
> usr/share/fonts/hack/.uuid
> 
> I tried reproducing this with backup, differential backup, backup 
> isolation on an filesystem with mounted with relatime option without 
> success... all works as expected on my side.

Yes, and it's strange this only happened to 1 of several VMs/systems.

> > It was recompiled recently (on the 10th), with me noticing the issue on 
> > the 13th or 14th.
> > But the other hosts are backed up using the exact same Dar-version. (Dar 
> > runs on the storage host which contains all the VMs. Some VMs are 
> > practically identical)
> > 
> > A backup is basically (in reality more complex due to automation) done 
> > as follows:
> > - filesystem snapshot is created
> > - snapshot is mounted
> > - snapshot is backed up using dar
> > - snapshot is umounted + removed
> 
> So my question is now about the snapshot process and the way you expose 
> it to dar.

commands run (simplistically, nothing special is actually performed):
# zfs snapshot ....
# mount -oro /dev/zvol/.....  /mnt/darroot
# dar .... /mnt/darroot

> Could you issue the following command for example in such snapshot:
> 
> stat /usr/share/fonts/hack

# stat /usr/share/fonts/hack
  File: /usr/share/fonts/hack
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 202,1   Inode: 532613      Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-08-02 08:56:46.000000000 +0000
Modify: 2021-06-03 13:32:10.000000000 +0000
Change: 2024-04-08 06:13:39.937335727 +0000
 Birth: 2017-10-26 11:41:16.541269703 +0000

[...]

> In conclusion so far, the explanation of the warning issued by 
> dar_manager is caused by a loss of time precision and is not a bug in 
> dar_manager, maybe still possible a bug in dar but that is not the most 
> probable explanation IMHO.

I can't fully check the actual timeline and what happened as not all logs are 
kept this long, which currently is quite annoying. I wish I knew how to 
reproduce this. Going forward, I have increased log-retention and if it occurs 
again, I should have a better idea on how to reproduce this issue.

Looking at the output of the "stat" command, I wonder if "tar", by default, 
only stores second-level precision, leading to the ".000000000 +0000" I see 
for "access" and "modify"

The filesystem has been the same for years and the OS is simply being kept 
uptodate over time.

What does "dar" do when it sees ".000000000 +0000" at the end? Does it reduce 
the precision to match the lowest non-zero field?

If the above doesn't yield anything pointing to a specific issue in dar/
dar_manager, I would suggest we leave this as-is and wait for it to happen 
again.

--
Joost

Re: [Dar-support] dar_manager - incremental backups and removed file

[Denis C. <dar...@fr...>]

On 20/04/2024 15:23, Joost Roeleveld via Dar-support wrote:
> 
> 
> ----------------------- Original message -----------------------
> From: Denis Corbin <dar...@fr...>
> To: dar...@li...
> Date: Sat, 20 Apr 2024 13:25:59 +0200
> ----------------------------------------------------------------
> 
>> Hi Joost,
>>
>> could you provide the output of 'dar -V'?
> 
> # dar -V
> 
>   dar version 2.7.13, Copyright (C) 2002-2023 Denis Corbin
>   Long options support : YES
> 

[...]

>   Timestamp read accuracy : 1 nanosecond
>   Timestamp write accuracy : 1 nanosecond

this is above the info I wanted to check and this is not what I 
expected, but OK. The problem seen is not caused by the way dar has been 
compiled and seems to read dates from the filesystem the same manner
over the time, I mean it has always read date from the fields of the 
system "struct stat" returned by lstat() system call:
- stat::st_atim.tv_sec
- stat::st_mtim.tv_sec
- stat::st_ctim.tv_sec

as implemented in src/libdar/filesystem_hard_link_read.cpp

[...]
> 
>> Have you recently recompiled or upgraded the dar binary used here to 
>> make backups on that host?

Here is why I ask, I'll take just the usr/share/fonts/hack/.uuid file to 
illustrate:

[Sorry for the ugly formated output... too wide to fit in email]

# dar -l MNGR_20240301T000001_san1__zdata_os_services_binhost_root -g
usr/share/fonts/hack -afdd
Warning: using insecure memory!
[Data ][D][ EA  ][FSA][Compr][S]| Permission | User  | Group | Size    |
          Date                 |    filename
--------------------------------+------------+-------+-------+---------+-------------------------------+------------
[InRef][-]       [-L-][  70%][ ]  drwxr-xr-x   root     root    5 Gio
Mon Jul  1 11:34:26 2019 + 370501096 ns usr
[InRef][-]       [-L-][  65%][ ]  drwxr-xr-x   root     root    872 Mio
Mon Feb 19 17:56:59 2024 + 297538137 ns usr/share
[InRef][-]       [-L-][  48%][ ]  drwxr-xr-x   root     root    182 Mio
Thu Jun  3 15:32:10 2021 + 211378 us    usr/share/fonts
[InRef][-]       [-L-][   0%][ ]  drwxr-xr-x   root     root    36 o
Thu Jun  3 15:32:10 2021 + 301378 us    usr/share/fonts/hack
[InRef][ ]       [-L-][-----][ ]  -rw-r--r--   root     root    36 o
Thu Jun  3 15:32:10 2021 + 301378785 ns usr/share/fonts/hack/.uuid

 From the above output of the oldest archive, all dates have nanosecond 
precision [the -afdd option has been added to display *F*ully *D*etailed 
*D*ates. This option will be available with release 2.8.0 for dar and 
dar_manager]


# dar -l MNGR_20240401T000002_san1__zdata_os_services_binhost_root -g
usr/share/fonts/hack -afdd
Warning: using insecure memory!
[Data ][D][ EA  ][FSA][Compr][S]| Permission | User  | Group | Size    |
          Date                 |    filename
--------------------------------+------------+-------+-------+---------+-------------------------------+------------
[InRef][-]       [-L-][  71%][ ]  drwxr-xr-x   root     root    4 Gio
Thu Mar 28 10:57:11 2024 + 379415752 ns usr
[InRef][-]       [-L-][  66%][ ]  drwxr-xr-x   root     root    832 Mio
Thu Mar 28 07:53:02 2024 + 534986022 ns usr/share
[     ][-]       [-l-][  48%][ ]  drwxr-xr-x   root     root    182 Mio
Thu Jun  3 15:32:10 2021 + 211378 us    usr/share/fonts
[     ][-]       [-l-][     ][ ]  drwxr-xr-x   root     root    0
Thu Jun  3 15:32:10 2021 + 301378 us    usr/share/fonts/hack
[     ][ ]       [-l-][-----][ ]  -rw-r--r--   root     root    36 o
Thu Jun  3 15:32:10 2021 + 301378785 ns usr/share/fonts/hack/.uuid

But for the second backup above, the precision changed to microsecond 
(displayed "us" above) for usr/share/fonts and usr/share/fonts/hack but 
is kept as nonaseconds for usr/share/fonts/.hack and usr.

Reading code on how differential backup behave about dates, 
(src/libdar/filtre.cpp line 790), a cat_inode object is created reading 
information from the filesystem, which mtime is compared with the one of 
the cat_inode taken from the archive of reference. The comparison takes 
into account the possible date precision difference (datetime::loose_diff).
If the date is equal inode created from the filesystem just drops 
attributes to its data but keeps the date and its accuracy from the 
filesystem.

Why microsecnfs for usr/share/fonts  but not for 
usr/share/fonts/hack/.uuid ? For libdar there is not code difference for 
directory and other inodes concerning date handling (same cat_inode is 
the ancestor class that handles all that is common to inodes, in 
particular dates).

Same for the third backup below:

# dar -l MNGR_20240407T000001_san1__zdata_os_services_binhost_root -g
usr/share/fonts/hack -afdd
Warning: using insecure memory!
[Data ][D][ EA  ][FSA][Compr][S]| Permission | User  | Group | Size    |
          Date                 |    filename
--------------------------------+------------+-------+-------+---------+-------------------------------+------------
[     ][-]       [---][     ][ ]  drwxr-xr-x   root     root    0
Thu Mar 28 10:57:11 2024 + 379415752 ns usr
[     ][-]       [---][     ][ ]  drwxr-xr-x   root     root    0
Thu Mar 28 07:53:02 2024 + 534986022 ns usr/share
[     ][-]       [---][     ][ ]  drwxr-xr-x   root     root    0
Thu Jun  3 15:32:10 2021 + 211378 us    usr/share/fonts
[     ][-]       [---][     ][ ]  drwxr-xr-x   root     root    0
Thu Jun  3 15:32:10 2021 + 301378 us    usr/share/fonts/hack
[     ][ ]       [---][-----][ ]  -rw-r--r--   root     root    36 o
Thu Jun  3 15:32:10 2021 + 301378785 ns usr/share/fonts/hack/.uuid

Finally for the fourth backup, the precision has even dropped one step 
further, to second for usr/share, usr/share/fonts and the deleted entry 
holding place of usr/share/fonts/hack/.uuid which date is read at the 
time of the backup from the mtime value of the parent directory:

# dar -l MNGR_20240414T000001_san1__zdata_os_services_binhost_root -g
usr/share/fonts/hack -afdd
Warning: using insecure memory!
[Data ][D][ EA  ][FSA][Compr][S]| Permission | User  | Group | Size    |
          Date                 |    filename
--------------------------------+------------+-------+-------+---------+-------------------------------+------------
[     ][-]       [---][  74%][ ]  drwxr-xr-x   root     root    450 Mio
Thu Mar 28 10:57:11 2024 + 379415752 ns usr
[InRef][-]       [-L-][  91%][ ]  drwxr-xr-x   root     root    233 Mio
Mon Apr  8 13:19:02 2024 + 244801698 ns usr/share
[     ][-]       [-L-][     ][ ]  drwxr-xr-x   root     root    0
Thu Jun  3 15:32:10 2021        usr/share/fonts
[     ][-]       [-L-][     ][ ]  drwxr-xr-x   root     root    0
Thu Jun  3 15:32:10 2021        usr/share/fonts/hack
[--- REMOVED ENTRY ----] (Thu Jun  3 15:32:10 2021) [-]
usr/share/fonts/hack/.uuid

I tried reproducing this with backup, differential backup, backup 
isolation on an filesystem with mounted with relatime option without 
success... all works as expected on my side.

> 
> It was recompiled recently (on the 10th), with me noticing the issue on 
> the 13th or 14th.
> But the other hosts are backed up using the exact same Dar-version. (Dar 
> runs on the storage host which contains all the VMs. Some VMs are 
> practically identical)
> 
> A backup is basically (in reality more complex due to automation) done 
> as follows:
> - filesystem snapshot is created
> - snapshot is mounted
> - snapshot is backed up using dar
> - snapshot is umounted + removed

So my question is now about the snapshot process and the way you expose 
it to dar.

Could you issue the following command for example in such snapshot:

stat /usr/share/fonts/hack

you should see something like this:

# stat /usr/share/fonts
   File: /usr/share/fonts
   Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 802h/2050d      Inode: 193473      Links: 11
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-04-21 21:08:01.923873113 +0200
Modify: 2020-06-12 20:15:09.546370763 +0200
Change: 2020-06-12 20:15:09.546370763 +0200
  Birth: 2019-04-23 20:32:20.976000000 +0200

In conclusion so far, the explanation of the warning issued by 
dar_manager is caused by a loss of time precision and is not a bug in 
dar_manager, maybe still possible a bug in dar but that is not the most 
probable explanation IMHO.

[...]

> 
> -- 
> Joost
> 
> 

Cheers,
Denis

Re: [Dar-support] dar_manager - incremental backups and removed file

[Joost R. <jo...@an...>]

----------------------- Original message -----------------------
From: Denis Corbin <dar...@fr...>
To: dar...@li...
Date: Sat, 20 Apr 2024 13:25:59 +0200
----------------------------------------------------------------

> Hi Joost,
>
> could you provide the output of 'dar -V'?

# dar -V

  dar version 2.7.13, Copyright (C) 2002-2023 Denis Corbin
  Long options support : YES

  Using libdar 6.7.1 built with compilation time options:
  gzip compression (libz) : YES
  bzip2 compression (libbzip2) : YES
  lzo compression (liblzo2) : NO
  xz compression (liblzma) : YES
  zstd compression (libzstd) : YES
  lz4 compression (liblz4) : NO
  Strong encryption (libgcrypt): YES
  Public key ciphers (gpgme) : YES
  Extended Attributes support : YES
  Large files support (>2GB) : YES
  ext2fs NODUMP flag support : YES
  Integer size used : 64 bits
  Thread safe support : YES
  Furtive read mode support : YES
  Linux ext2/3/4 FSA support : YES
  Mac OS X HFS+ FSA support : NO
  Linux statx() support : YES
  Detected system/CPU endian : little
  Posix fadvise support : YES
  Large dir. speed optimi. : YES
  Timestamp read accuracy : 1 nanosecond
  Timestamp write accuracy : 1 nanosecond
  Restores dates of symlinks : YES
  Multiple threads (libthreads): YES (1.4.0 - barrier using pthread_barrier_t)
  Delta compression (librsync) : NO
  Remote repository (libcurl) : NO
  argon2 hashing (libargon2) : YES

  compiled the Apr 10 2024 with GNUC version 13.2.1 20240210
  dar is part of the Disk ARchive suite (Release 2.7.13)
  dar comes with ABSOLUTELY NO WARRANTY; for details
  type `dar -W'. This is free software, and you are welcome
  to redistribute it under certain conditions; type `dar -L | more'
  for details.


> Have you recently recompiled or upgraded the dar binary used here to  
> make backups on that host?

It was recompiled recently (on the 10th), with me noticing the issue  
on the 13th or 14th.
But the other hosts are backed up using the exact same Dar-version.  
(Dar runs on the storage host which contains all the VMs. Some VMs are  
practically identical)

A backup is basically (in reality more complex due to automation) done  
as follows:
- filesystem snapshot is created
- snapshot is mounted
- snapshot is backed up using dar
- snapshot is umounted + removed

I actually wrote a backup system that automates all this using the  
storage layer commands (currently ZFS, I used LVM till about 6 years  
ago) and uses dar_manager statistics to determine if a potential  
"parent" for a backup is actually containing relevant files for a full  
restore. If not (both EA and Files are "0"), it is ignored and an  
older backup is selected as "parent"

Dar-files are encrypted and stored in the cloud. With monthly sets  
being written to tape. All this requires minimal monitoring by myself  
and generally runs fully automatic. If an error occurs, it's generally  
solvable by a simple restart, only rarely it requires further  
investigation. (Like this dar_manager issue).

--
Joost

Re: [Dar-support] dar_manager - incremental backups and removed file

[Denis C. <dar...@fr...>]

Hi Joost,

could you provide the output of 'dar -V'?

Have you recently recompiled or upgraded the dar binary used here to 
make backups on that host?


On 17/04/2024 06:55, J. Roeleveld via Dar-support wrote:
> On Tuesday, 16 April 2024 22:29:42 CEST Denis Corbin wrote:
>> On 16/04/2024 09:44, J. Roeleveld via Dar-support wrote:> On Monday, 15
>> April 2024 22:56:58 CEST Denis Corbin wrote:
>>   >> On 15/04/2024 07:04, J. Roeleveld via Dar-support wrote:
> 
> Hi Denis,
> 
>> OK, the first problem you reported about dar_manager warning is a bug.
>> Here is what produces the output with all detailed date information (new
>> -x option for dar_manager):
> 
> Ok, I'll ignore it for now. :)
> 
>> # dar_manager -B
>> database_delete_order_warning/zdata_os_services_binhost_root.dmd -f
>> usr/share/fonts/liberation-fonts/.uuid -x
>> Warning: using insecure memory!
>>    1       Thu Jun  3 15:32:10 2021 + 341379045 ns  saved     absent
>>    2       Thu Jun  3 15:32:10 2021 + 341379045 ns  present   absent
>>    3       Thu Jun  3 15:32:10 2021 + 341379045 ns  present   absent
>>    4       Thu Jun  3 15:32:10 2021  removed                  absent
>>

I observe that the backup you kindly provided, do contain different time 
precision for different files, some file have mtime in unit of second, 
some other have nanosecond precision, in the same backup, which is very 
weird.

I tried to reproduce this on a simple case, traces the code and could 
not see any place where time precision would be lost.

I still have some process to explore other backup transformation process 
like catalogue isolation, so investigation are still under process...

>>
>> The date at #4 is just lacking the sub-second information, which leads
>> it  to be considered older than the dates at #3 #2 and #1 by
>> approximativement 0.341 s. This is the cause of the warning. I will look
>> at the way the removal date is obtained and why it is lacking the
>> sub-second part, but that's almost harmless as far as I see the
>> consequences. This may be a bug in dar or dar_manager I have to
>> investigate that.
>>
>> The second problem is the fact the date of removal (over its lack of
>> precision), is not the last modification date of the parent directory. I
>> also have to investigate that. Same thing here, maybe the problem is in
>> dar or dar_manager but this is also a bug.
>>
>> # dar_manager -B
>> database_delete_order_warning/zdata_os_services_binhost_root.dmd -f
>> usr/share/fonts/liberation-fonts -x
>> Warning: using insecure memory!
>>    1       Thu Jun  3 15:32:10 2021 + 341379045 ns  saved     absent
>>    2       Thu Jun  3 15:32:10 2021 + 341379045 ns  present   absent
>>    3       Thu Jun  3 15:32:10 2021 + 341379045 ns  present   absent
>>    4       Mon Apr  8 08:13:39 2024 + 957335766 ns  present   absent

Here the mtime of the parent directory at #4 is different between what 
the dar backup stores and what the dar_manager database has recorded. I 
have found where it is done in dar_manager, but still have to figure out 
why. I guess this is related to the way dar_manager handles extended 
attributes and FSA.

[...]
> 
> --
> Joost
> 
> 
> 

Denis

Re: [Dar-support] dar_manager - incremental backups and removed file

[J. R. <jo...@an...>]

On Tuesday, 16 April 2024 22:29:42 CEST Denis Corbin wrote:
> On 16/04/2024 09:44, J. Roeleveld via Dar-support wrote:> On Monday, 15 
> April 2024 22:56:58 CEST Denis Corbin wrote:
>  >> On 15/04/2024 07:04, J. Roeleveld via Dar-support wrote:

Hi Denis,

> OK, the first problem you reported about dar_manager warning is a bug. 
> Here is what produces the output with all detailed date information (new 
> -x option for dar_manager):

Ok, I'll ignore it for now. :)

> # dar_manager -B 
> database_delete_order_warning/zdata_os_services_binhost_root.dmd -f 
> usr/share/fonts/liberation-fonts/.uuid -x
> Warning: using insecure memory!
>   1       Thu Jun  3 15:32:10 2021 + 341379045 ns  saved     absent
>   2       Thu Jun  3 15:32:10 2021 + 341379045 ns  present   absent
>   3       Thu Jun  3 15:32:10 2021 + 341379045 ns  present   absent
>   4       Thu Jun  3 15:32:10 2021  removed                  absent
> 
> 
> The date at #4 is just lacking the sub-second information, which leads 
> it  to be considered older than the dates at #3 #2 and #1 by 
> approximativement 0.341 s. This is the cause of the warning. I will look 
> at the way the removal date is obtained and why it is lacking the 
> sub-second part, but that's almost harmless as far as I see the 
> consequences. This may be a bug in dar or dar_manager I have to 
> investigate that.
> 
> The second problem is the fact the date of removal (over its lack of 
> precision), is not the last modification date of the parent directory. I 
> also have to investigate that. Same thing here, maybe the problem is in 
> dar or dar_manager but this is also a bug.
> 
> # dar_manager -B 
> database_delete_order_warning/zdata_os_services_binhost_root.dmd -f 
> usr/share/fonts/liberation-fonts -x
> Warning: using insecure memory!
>   1       Thu Jun  3 15:32:10 2021 + 341379045 ns  saved     absent
>   2       Thu Jun  3 15:32:10 2021 + 341379045 ns  present   absent
>   3       Thu Jun  3 15:32:10 2021 + 341379045 ns  present   absent
>   4       Mon Apr  8 08:13:39 2024 + 957335766 ns  present   absent
> #
> > 
> >> Which dar and dar_manager version are used?
> > 
> > 
> > I use version 2.7.13.
> > Am planning on upgrading to 2.7.14 later this month.
> 
> 
> 2.7.14 does not bring any fix related to this problem so
> 2.7.13 is perfectly fine, no worries. You can wait for 2.7.15 with these 
> two bug fixes.

I'll check the changelog and see if there is anything I need.

> >>> I have several systems using the same distribution, but only this
> >>> particular one is showing the problem.
> >>
> >> are they all using the same dar and dar_manager versions?
> > 
> > Yes, they all use the same dar and dar_manager versions.
> 
> thus, maybe the file removal activity is not the same on the other 
> systems? this would explain why dar_manager does not warning the same(?)

It's a file inside the font directories which might not even have existed on 
other systems. (From what I found, it's generated by certain tools which I 
might have actually used on that system at some point)

I don't generally look in the "/usr" tree for random files, so I can't be 
certain on the actual timeline. But that file must have been removed 
automatically.

--
Joost

Re: [Dar-support] dar_manager - incremental backups and removed file

[Denis C. <dar...@fr...>]

On 16/04/2024 09:44, J. Roeleveld via Dar-support wrote:> On Monday, 15 
April 2024 22:56:58 CEST Denis Corbin wrote:
 >> On 15/04/2024 07:04, J. Roeleveld via Dar-support wrote:
 >>

Hi Joost,

OK, the first problem you reported about dar_manager warning is a bug. 
Here is what produces the output with all detailed date information (new 
-x option for dar_manager):

# dar_manager -B 
database_delete_order_warning/zdata_os_services_binhost_root.dmd -f 
usr/share/fonts/liberation-fonts/.uuid -x
Warning: using insecure memory!
  1       Thu Jun  3 15:32:10 2021 + 341379045 ns  saved     absent
  2       Thu Jun  3 15:32:10 2021 + 341379045 ns  present   absent
  3       Thu Jun  3 15:32:10 2021 + 341379045 ns  present   absent
  4       Thu Jun  3 15:32:10 2021  removed                  absent


The date at #4 is just lacking the sub-second information, which leads 
it  to be considered older than the dates at #3 #2 and #1 by 
approximativement 0.341 s. This is the cause of the warning. I will look 
at the way the removal date is obtained and why it is lacking the 
sub-second part, but that's almost harmless as far as I see the 
consequences. This may be a bug in dar or dar_manager I have to 
investigate that.

The second problem is the fact the date of removal (over its lack of 
precision), is not the last modification date of the parent directory. I 
also have to investigate that. Same thing here, maybe the problem is in 
dar or dar_manager but this is also a bug.

# dar_manager -B 
database_delete_order_warning/zdata_os_services_binhost_root.dmd -f 
usr/share/fonts/liberation-fonts -x
Warning: using insecure memory!
  1       Thu Jun  3 15:32:10 2021 + 341379045 ns  saved     absent
  2       Thu Jun  3 15:32:10 2021 + 341379045 ns  present   absent
  3       Thu Jun  3 15:32:10 2021 + 341379045 ns  present   absent
  4       Mon Apr  8 08:13:39 2024 + 957335766 ns  present   absent
#


> 
>> Which dar and dar_manager version are used?
> 
> I use version 2.7.13.
> Am planning on upgrading to 2.7.14 later this month.

2.7.14 does not bring any fix related to this problem so
2.7.13 is perfectly fine, no worries. You can wait for 2.7.15 with these 
two bug fixes.

[...]

> 
>>> I have several systems using the same distribution, but only this
>>> particular one is showing the problem.
>>
>> are they all using the same dar and dar_manager versions?
> 
> Yes, they all use the same dar and dar_manager versions.

thus, maybe the file removal activity is not the same on the other 
systems? this would explain why dar_manager does not warning the same(?)

> 
> --
> Joost
> 
> 
> 

Cheers,
Denis

Re: [Dar-support] dar_manager - incremental backups and removed file

[J. R. <jo...@an...>]

On Monday, 15 April 2024 22:56:58 CEST Denis Corbin wrote:
> On 15/04/2024 07:04, J. Roeleveld via Dar-support wrote:
> 
> >> No warning showed. I just realize that dates are evaluated a the
> >> nanosecond accuracy, but the display has only the second as precision,
> >> thus this may be the reason of the warning while you see no difference.
> >> Unfortunately there is no way to get this information except restoring
> >> files.
> > 
> > 
> > The backups I register into the database are taken at a daily interval and
> > the files are unlikely to be touched.
> 
> thus, the removal date should not be the same as the creation date as 3 
> days passed in between...
> 
> [...]
> 
> > As you mention the directory:
> > $ dar_manager -B zdata_os_services_binhost_root.dmd -f usr/share/fonts/
> > liberation-fonts
> > 
> >          1       Thu Jun  3 15:32:10 2021  saved
> > 
> > absent
> > 
> >          2       Thu Jun  3 15:32:10 2021  present
> > 
> > absent
> > 
> >          3       Thu Jun  3 15:32:10 2021  present
> > 
> > absent
> > 
> >          4       Mon Apr  8 08:13:39 2024  present
> > 
> > absent
> > 
> > $ dar_manager -B zdata_os_services_binhost_root.dmd -f usr/share/fonts/
> > liberation-fonts/.uuid
> > 
> >          1       Thu Jun  3 15:32:10 2021  saved
> > 
> > absent
> > 
> >          2       Thu Jun  3 15:32:10 2021  present
> > 
> > absent
> > 
> >          3       Thu Jun  3 15:32:10 2021  present
> > 
> > absent
> > 
> >          4       Thu Jun  3 15:32:10 2021  removed
> > 
> > absent
> > 
> > The directory did get a new date, but the file has not been marked as
> > changed (if I read the output correctly)
> 
> yes, you do.
> 
> And this sounds really really weird! Though it does not explain the 
> original problem with dar_manager, it looks like an unexpected behavior 
> of dar, here, unless you make use of  -< -> -= options?

No, I don't, the commands I use are:

* full:
 /usr/bin/sudo /usr/bin/dar -m 256 -zbzip2:5 -s <slizesize> -D -R <backuproot> 
-c <dartarget> --user-comment <usercomment>
   --min-digits <zeropadding>,0,<zeropadding> -K <passphrase>
   -Z *.zip -Z *.gz -Z *.bz2 -Z *.cz -Z *.jar -Z *.rar -Z *.tgz -Z *.war -Z 
*.ear -Z *.rar -Z *.7z -Z *.tgz -Z *.tbz2 -Z *.avi -Z *.mkv -Z *.mp3
   -E "/opt/backups/etc/dar_par_create.duc %p %b %n %e %c 2"

* incremental:
/usr/bin/sudo /usr/bin/dar -m 256 -zbzip2:5 -s <slizesize> -D -R <backuproot> 
-c <dartarget> --user-comment <usercomment>
    --min-digits <zeropadding>,0,<zeropadding> -A <parent_catalogue> -K 
<passphrase>
    -Z *.zip -Z *.gz -Z *.bz2 -Z *.cz -Z *.jar -Z *.rar -Z *.tgz -Z *.war -Z 
*.ear -Z *.rar -Z *.7z -Z *.tgz -Z *.tbz2 -Z *.avi -Z *.mkv -Z *.mp3
    -E "/opt/backups/etc/dar_par_create.duc %p %b %n %e %c 2"

* create catalogue:
/usr/bin/sudo /usr/bin/dar --min-digits <zeropadding>,0,<zeropadding> -C 
<new_catalogue> -A <dartarget>

> Which dar and dar_manager version are used?

I use version 2.7.13.
Am planning on upgrading to 2.7.14 later this month.

> > As this particular partition and system is not too critical, I can share
> > the catalogues and database with you directly?
> 
> yes, you can, I'll have a look with the -x option just added to 
> dar_manager (git / master branch)
>
> >> what filesystem is used? Which option have been used to mount it?
> > 
> > 
> > The filesystem:
> > # mount
> > /dev/xvda1 on / type ext4 (rw,noatime,discard,stripe=2)
> 
> OK, the noatime does not explain the directory behavior it should not 
> concern mtime (which dar/dar_manager look at) but only atime, and ext4 
> is not known to have any limitation regarding mtime behavior AFAIK.

I haven't heard of any limitation there either.

> > I have several systems using the same distribution, but only this
> > particular one is showing the problem.
> 
> are they all using the same dar and dar_manager versions?

Yes, they all use the same dar and dar_manager versions.

--
Joost

Re: [Dar-support] dar_manager - incremental backups and removed file

[Denis C. <dar...@fr...>]

On 15/04/2024 07:04, J. Roeleveld via Dar-support wrote:
> Hi Denis,

Hi Joost,

> 
> I am replying to both in this single email.
> 

sure

[...]

>> No warning showed. I just realize that dates are evaluated a the
>> nanosecond accuracy, but the display has only the second as precision,
>> thus this may be the reason of the warning while you see no difference.
>> Unfortunately there is no way to get this information except restoring
>> files.
> 
> The backups I register into the database are taken at a daily interval and the
> files are unlikely to be touched.

thus, the removal date should not be the same as the creation date as 3 
days passed in between...

> 

[...]

> 
> As you mention the directory:
> $ dar_manager -B zdata_os_services_binhost_root.dmd -f usr/share/fonts/
> liberation-fonts
>          1       Thu Jun  3 15:32:10 2021  saved
> absent
>          2       Thu Jun  3 15:32:10 2021  present
> absent
>          3       Thu Jun  3 15:32:10 2021  present
> absent
>          4       Mon Apr  8 08:13:39 2024  present
> absent
> 
> $ dar_manager -B zdata_os_services_binhost_root.dmd -f usr/share/fonts/
> liberation-fonts/.uuid
>          1       Thu Jun  3 15:32:10 2021  saved
> absent
>          2       Thu Jun  3 15:32:10 2021  present
> absent
>          3       Thu Jun  3 15:32:10 2021  present
> absent
>          4       Thu Jun  3 15:32:10 2021  removed
> absent
> 
> The directory did get a new date, but the file has not been marked as changed
> (if I read the output correctly)

yes, you do.

And this sounds really really weird! Though it does not explain the 
original problem with dar_manager, it looks like an unexpected behavior 
of dar, here, unless you make use of  -< -> -= options?

Which dar and dar_manager version are used?

> 
> As this particular partition and system is not too critical, I can share the
> catalogues and database with you directly?

yes, you can, I'll have a look with the -x option just added to 
dar_manager (git / master branch)

[...]

>> what filesystem is used? Which option have been used to mount it?
> 
> The filesystem:
> # mount
> /dev/xvda1 on / type ext4 (rw,noatime,discard,stripe=2)

OK, the noatime does not explain the directory behavior it should not 
concern mtime (which dar/dar_manager look at) but only atime, and ext4 
is not known to have any limitation regarding mtime behavior AFAIK.

> 
> I have several systems using the same distribution, but only this particular
> one is showing the problem.

are they all using the same dar and dar_manager versions?

> 
> --
> Joost
> 
> 
Cheers,
Denis