Re: [Bacula-devel] Pruning and recycling volumes

[Ondrej P. <ond...@ig...>]

Hi,

> Hello,
>
> As you may remember, one of the things I am working on for version 2.1.x is 
> performance.  My most recent work on performance has been to review and 
> re-write the pruning and purging code.  Over the years, this code has been 
> added to and patched, and there are a number of issues that I want to 
> address:
>
> 1. Review the pruning algorithms to improve the SQL.
> 2. Pruning during "status dir"
> 3. User complaints that it requires two passes to recycle Volumes
>
> The basic algorithm for recycling volumes is documented in the manual, and it 
> will remain unchanged.  However, I have now re-written the prune/purge code 
> to do the following things:
>
> 1. Reduce unnecessary SQL calls to a minimum
> 2. When pruning, submit SQL statements that prune up to 1000 jobs
>     in a single SQL statement rather than 1000 separate SQL statements.
> 3. Prune only a single volume if the SD requests pruning rather than
>     pruning *all* the volumes in the Pool.
> 4. After pruning a Volume, explicitly check if it can be marked as Purged.
> 5. Ensure that there are not multiple copies of the same code (pruning and
>     and purging share a lot code, some of which was duplicated).
>
> The net result is not a huge gain in performance, except in a few exceptional 
> cases, but the code base is much cleaner, and I believe I have corrected at 
> least one case where pruning would be performed but Bacula did not detect and 
> hence mark the Volume purged.
>
> There are two issues where I would appreciate a bit of user feedback:
>
> 1. Previous when Bacula needed a new Volume, it would prune *all* volumes
>     in the current pool.  Now it prunes only one at a time, until it finds one 
>     that has been Purged.  This means that less pruning will be done, and
>     database records will tend to remain longer (possibly much longer).
>     Although individual volumes can be prunned by command, there is no
>     command to prune a whole pool.
>
> Question: does anyone feel there is a need for a new command that will
>                  pemit manually pruning a whole pool?  An appropriate place
>                  *may* be dbcheck.
>
> 2. Currently, the "status dir" command will do pruning while attempting to
>     find the Volume name to list if no current Volume is available.  Many
>     many users have complained about this because it can be *very* time
>     consuming.  The new code will be slightly faster because it will stop once
>     a volume is Purged rather than pruning the whole Pool.  However I am
>    considering disabling pruning by default from the "status dir" command,
>    but allow the user to optionally specify "status dir prune" to turn it
>    back on.  Note, there is a "list nextvolume" command which will continue
>    to prune by default.
>
> Question: does anyone object to turning off pruning in the "status dir"
>                  command or have a better idea?
>
>   

IMHO:I vote for turning off pruning during command 'status dir', because 
I hate very long waiting for prunning volumes :((

Ondra


-- 
Bye, bye ...

Ondrej PLANKA