Re: [Bacula-users] Retention after migration

[Dermot B. <der...@dp...>]

On 24 December 2010 04:23, Dan Langille <da...@la...> wrote:
> On 12/23/2010 5:45 PM, Dermot Beirne wrote:
>>
>> Hi Dan,
>>
>>
>>>
>>> According to the above, Media ID 896 has a retention period of 2,851,200
>>> seconds (or 33 days).
>>>
>>
>> That's correct, and that's what I would expect.
>>
>>>
>>> Aren't Job and File retention specific to a client, not a Volume?
>>>
>>
>> I know they can be specified on the client side, but I though pool
>> definitions (where I have defined them) over rides this.  I only
>> specify a job retention there because it's mandatory, but ignored if
>> defined in the pool.  Am I wrong?
>>
>>> What I do, or have tired, and have not tested is putting Job and File
>>> retention to a value equal to the maximum Pool retention value.
>>>
>>> Thus, in your case, your maximum Volume retention is 10 years.  I suggest
>>> you want Job and File retention set at 10 years.
>>>
>>> Would that help you?
>>>
>>
>> I'm not sure why it would help, as it appears to be ignoring my
>> current retention times at the moment.
>> Do you know if the retention periods of the new pool (post migration)
>> are applied to the jobs that are migrated.  It looks like the disk
>> pool retentions (just a few days) are being used instead.
>>
>> If I specify a 10 year retention on the clients definition for jobs
>> and files, will this not mean an endlessly growing database that never
>> gets pruned.
>
> Yes, for 10 years.  But retain perspective.  There will be one entry in the
> database for every file name and every path name.  And one entry for every
> file in every backup.  How many files are in each backup, more or less?
>
> I don't prune until after three years.  If I need to restore, the last thing
> I want to do is deal with bscan.  Overall, I believe the obsession with
> pruning is easily overridden by the hassles of restoring when File/Job
> records do not exist.
>
> To give you an idea of my situation (commas added by me):
>
> [dan@ngaio:~] $ psql bacula
> psql (8.4.5)
> Type "help" for help.
>
> bacula=# select count(*) from file;
>  count
> ---------
>  8,274,704
> (1 row)
>
> bacula=# select count(*) from filename;
>  count
> ---------
>  2,307,279
> (1 row)
>
> bacula=# select count(*) from job;
>  count
> -------
>  9,056
> (1 row)
>
> bacula=# select count(*) from jobmedia;
>  count
> -------
>  13,765
> (1 row)
>
> bacula=# select count(*) from path;
>  count
> --------
>  181,568
> (1 row)
>
> bacula=#
>
>
> How big is my database on disk?
>
> bacula=# select pg_size_pretty(pg_database_size('bacula'));
>  pg_size_pretty
> ----------------
>  3031 MB
> (1 row)
>
>
> 3GB.  That includes indexes etc.  When dumped to disk, that becomes 1GB, or
> about 375MB, compressed.
>
> The oldest full backup in that database is dated 2005-01-31 21:14:25, so
> that is 6 years of backups.  A straight guess, for my situation, 10 years of
> backups would occupy 10GB of disk space.
>
> I think disk space is cheap compared to the time and bscan frustration.
>
> Just something to consider.
>
>
> --
> Dan Langille - http://langille.org/
>
>

Dan,

My database is currently 23Gb.  My counts are as follows (although I
have a lot more clients to add yet):

mysql> select count(*) from File;
+----------+
| count(*) |
+----------+
| 11707301 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from Filename;
+----------+
| count(*) |
+----------+
| 19296343 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from Job;
+----------+
| count(*) |
+----------+
|       82 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from JobMedia;
+----------+
| count(*) |
+----------+
|     4970 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from Path;
+----------+
| count(*) |
+----------+
|  2024290 |
+----------+
1 row in set (0.00 sec)

Martins link tells a lot, and I think calling this a bug is a gross
understatement.  It seems to me to severely alter the expected
functionality (based on the documentation) of the program to a
dangerous level.

I am entirely dependent on bscan for any restore older than about 1
week as far as I can tell.  At least it does work, as I've used it a
few times.

I have removed the File and Job retention from both the disk and tape
pools.  I have left the volume retention there, at the same lengths as
described previously for each pool.

I have put the Job and File retention into each client, setting them
to the max volume retention (10 years) as you suggested Dan.  It was
only have reading the thread Martin linked to, that this made sense to
me.

I expect the volume retention specified in the pools will enforce  the
desired effect of (by overriding) the File/Job retention I was trying
to use for each Pool.  I have updated all volumes from pool.

There are 2 things I don't full understand yet:
1. Will migrated jobs assume the volume retention of the tape pools
(destination), or the disk pools (source)?

2. Will the Volume Retention force the pruning of the Jobs/Files after
the volume is purged/recycled, or will the Job/File records remain in
the db for the full 10 years?  I presume the Job/Files records will be
pruned at the same time as the volume gets purged, as otherwise the
jobs/file records would point to a volume that no longer contained
that data.  However, Dan you say you only prune every 3 years.  Does
that mean you have enough volumes that you don't need to recycle for 3
years?  If I wanted to achieve something like that, I would need a
serious amount of extra tapes.

Dermot.