Re: [Bacula-devel] Catalog maintenance and Volume recycling

[Kern S. <ke...@si...>]

Hello Phil,

Thanks for the comments. I see that you really
took the time to read it and think about it.

On Tue, 2002-04-30 at 20:37, Phil Stracchino wrote:
> On Tue, Apr 30, 2002 at 10:17:36AM +0200, Kern Sibbald wrote:
> > Hello,
> > 
> > Below you will find a brief description of what I propose
> > to implement (much already in the CVS tree) for Bacula
> > concerning automatic pruning of the database catalog
> > and recycling of volumes.
> > 
> > I would appreciate your comments.
> 
> 
> About the one comment I have after a first reading of this is that seconds
> are a very poor unit of time for this purpose.  Making people calculate,
> say, 3 months in seconds will be unmanageable.  I would suggest using days
> as the basic unit instead.  It doesn't matter if the code converts the
> retention time into seconds internally, but the *interface* needs to allow
> it to be specified in human-manageable units.  If people want to retain a
> job for 98 days (one financial quarter plus 7 days, for example), let them
> enter 98, not have to calculate that 98 days is 8467200 seconds.  If you
> feel finer resolution is required, allow retention time to be specified as
> "98d 0h", or even "98d 0h 0m".

You are absolutely right. I guess I was in implementor's mode rather 
than user's mode when I wrote the RFC. In fact, I had the same
reflection as you and should have specified that the user will enter
a time value, which is what is currently implemented.  A time is
a real number in scientific notation possibly followed by a modifier.
A modifier is on of the following:

   modifier       meaning
      s           seconds
      m           minutes
      h           hours
      d           days
      w           week
      o           month
      q           quarter
      y           year

So to enter to enter 98 days one would say:

   Volume Retention = 98d

Currently no arithmetic is allowed so you cannot say "1q + 7d" but
maybe we can implement that later. Although this is implemented,
it isn't yet documented.  I've done a similar thing for entering
bytes (sizes such as tape capacities, etc.). By the way 31.5 days
could be entered as 3.1e1d (a bit weird) or 31.5d. 

For sizes you can enter:

  modifier    meaning
     k         1024 bytes
     m      1048576 bytes
     g   1073741824 bytes

If you want 1000 bytes, you can also enter 1e3, ...

  
> 
> There should probably also be an explicit "NEVER" keyword for retention
> time, meaning that a job will never be automatically purged or a tape will
> never be automatically recycled.  This would be primarily useful for
> archival backups that will be kept for an indeterminate period of time
> until manually recycled.

Actually, I probably left out a new variable associated with each Pool,
which becomes the default for each Volume, and that is Recycle = yes/no.
If you set Recycle to NO, the Pool, or Volume will never be recycled,
unless you specifically set it to YES.  There probably should be some
concept of "precious volumes" where it would be virtually impossible
to set Recycle to YES, but I think it can be added later.

> 
> 
> I would also suggest that in order for retention times to work as people
> expect them to, the day the job is *STARTED* on should count as the first
> day of the retention period.  Otherwise, if I start a job that runs at,
> say, 10pm on Sunday night and doesn't complete until Monday morning, and I
> want the job to re-use the same tape every week and so I set the retention
> period to 7 days (to use an overly simplistic, non-real-world example),
> Bacula will try to run the job again next Sunday but the tape won't
> actually become re-usable until next Monday.  If the day the job starts is
> counted as the first day of the retention period, then the 7-day retention
> period for the tape will be [Sunday Monday Tuesday Wednesday Thursday
> Friday Saturday], and by the time the job runs again on Sunday the tape
> will be reusable.
> 

Hmmm. This is something that I had thought about for some time, but 
still am not 100% sure. My final conclusion was that especially for 
Volumes, the VolumeRetention period should be the minimum guaranteed
time after the last write to the volume before it would be considered
for recycling. Your suggestion makes a lot of sense, but is different
from what I planned to implement.  Although you can force Bacula to
write a different tape every day (i.e. write only once on a tape), 
the normal operation would be to continue filling the same tape until
it is full, which means that you wouldn't be guaranteed to use
the same tape each day of the week. 

I can see the usefulness or simplicity of what you want to do, so
I need to figure out some way to do it. Most sites get around the
problem you mention by simply setting the retention period to 6 
days, but Bacula should be able to do better.

Concerning your email: I'd make that "Recycle the oldest volume 
marked AutoRecycle".  Just for clarity.

Yes, I agree. Actually, a volume starts out as "Append". When it
is full, it is marked "Full".  If no Append volumes are available,
and if Recycle=yes and AutoRecyle=yes,
Bacula will look for the oldest volume marked "Purged", which
could happen if you ran a manual Prune on the volume or explicitly
changed the Volume status. If there are no Purged volumes, Bacula
will attempt to Purge volumes starting with the oldest one (as you
suggest). If it is successful, it will recycle the volume. 

Thanks again for the comments. I'll get back to you about the 
start date for calculating Volume Retention periods.

Best regards,

Kern