Re: [Bobs-devel] incremental backup cleanup

[Rene R. <re...@gr...>]

On Sun, 2004-04-25 at 07:18, Joe Zacky wrote:
> Rene,
> 
> I was going to add a routine to cleanup incremental backups but now I'm 
> not sure. I was planning to remove incremental files that were backed up 
> over X days ago, but now I see the field text is "How many incremental 
> backups to keep (0 = infinite)", which seems to mean how many copies of 
> an incremental file to keep. So if incremental is set to 9, then there 
> could be up to 9 copies of any incremental file. This means the 
> incremental backup directory could be up to 9 times as large as the 
> current backup.
> 
> Here's some points to ponder:
> 
> If we keep X copies of incrementals:
> o    The amount of storage required "could" be much more than if we kept 
> X days.
> o    It's going to be programmatically difficult and time consuming to 
> read through all the file names in incremental and count how many there 
> are of each.
> 
> If we keep X days worth of incrementals:
> o   The only backup copy of a file would be deleted after X days if the 
> file hasn't been changed. That's probably not a good thing.
> o    Removing the incremental files is easily done with a command like this:
>       find /path/to/bobsdata/incremental/<server>/<share>/ -type f 
> -ctime +<days> -ctime -999 -exec rm -v {} ';'
> 
> So I'm wondering 1) what your intention is for this field, 2) what are 
> you doing on your bobs system to cleanup incremental files, and 3) how 
> do you suggest I proceed?
> 
> The question is what do we want our selection criteria to be: "how many" 
> or "how long?"
> 

Hi Joe
Here is my view on it. 

My idea was to have X days worth of files in the incremental store. That
would allow to say that I want 90 days of incremental backups. In my
situation, if a file hasn't changed for 90 days it probably done.

I have no problem with not having a backup of a file in incrementals. If
that is a problem I need to increase the timespan I keep the files and
possibly increase my disk capacity if that is needed. That is not a BOBS
problem, but a backup administrators problem.

Thew way I delete files is to set some limits. Say files older than 6
months AND larger than 500 MB ( and sometimes by type, like .avi, .mov,
mpg and so on. Depends on the work being done.)
Generally my tolerance towards large files is biased so they are deleted
first.
A 500MB file is seldom a "from scratch work" but something made from
another file. Like a movie rendering. Exceptions are files like
photoshop (.psd) files which can be large and original works.

That is just my situation others probably have different scenarios.

Another feature I like is to have bobs decide when incrementals are
deleted. Say, when only 50 GB of diskspace is left, delete until there
is 100 GB free, oldest files first. (I guess this will be a problem when
having different time spans for the various backups.)

Anyway. A simple "delete after X incremental backups" would be a good
start. X not being days, since we can be sure the backup is run every
day, requires a little date manipulation or some other system of telling
when and what to delete.

Please used the database(s) to search for files to delete. The time on
disk is not accurate. I think it reflects the time the file was last
changed (on the originating server).
The database has the correct information by using the date tagging
system. 
I used the find command when deleting but I know it isn't the correct
way to do it. 


Cheers
Rene