Menu
▾
▴
bobs-devel
|
From: Joe Z. <jz...@co...> - 2004-04-25 05:24:31
|
Rene,
I was going to add a routine to cleanup incremental backups but now I'm
not sure. I was planning to remove incremental files that were backed up
over X days ago, but now I see the field text is "How many incremental
backups to keep (0 = infinite)", which seems to mean how many copies of
an incremental file to keep. So if incremental is set to 9, then there
could be up to 9 copies of any incremental file. This means the
incremental backup directory could be up to 9 times as large as the
current backup.
Here's some points to ponder:
If we keep X copies of incrementals:
o The amount of storage required "could" be much more than if we kept
X days.
o It's going to be programmatically difficult and time consuming to
read through all the file names in incremental and count how many there
are of each.
If we keep X days worth of incrementals:
o The only backup copy of a file would be deleted after X days if the
file hasn't been changed. That's probably not a good thing.
o Removing the incremental files is easily done with a command like this:
find /path/to/bobsdata/incremental/<server>/<share>/ -type f
-ctime +<days> -ctime -999 -exec rm -v {} ';'
So I'm wondering 1) what your intention is for this field, 2) what are
you doing on your bobs system to cleanup incremental files, and 3) how
do you suggest I proceed?
The question is what do we want our selection criteria to be: "how many"
or "how long?"
Cheers,
Joe
|
|
From: Rene R. <re...@gr...> - 2004-04-25 15:38:53
|
On Sun, 2004-04-25 at 07:18, Joe Zacky wrote:
> Rene,
>
> I was going to add a routine to cleanup incremental backups but now I'm
> not sure. I was planning to remove incremental files that were backed up
> over X days ago, but now I see the field text is "How many incremental
> backups to keep (0 = infinite)", which seems to mean how many copies of
> an incremental file to keep. So if incremental is set to 9, then there
> could be up to 9 copies of any incremental file. This means the
> incremental backup directory could be up to 9 times as large as the
> current backup.
>
> Here's some points to ponder:
>
> If we keep X copies of incrementals:
> o The amount of storage required "could" be much more than if we kept
> X days.
> o It's going to be programmatically difficult and time consuming to
> read through all the file names in incremental and count how many there
> are of each.
>
> If we keep X days worth of incrementals:
> o The only backup copy of a file would be deleted after X days if the
> file hasn't been changed. That's probably not a good thing.
> o Removing the incremental files is easily done with a command like this:
> find /path/to/bobsdata/incremental/<server>/<share>/ -type f
> -ctime +<days> -ctime -999 -exec rm -v {} ';'
>
> So I'm wondering 1) what your intention is for this field, 2) what are
> you doing on your bobs system to cleanup incremental files, and 3) how
> do you suggest I proceed?
>
> The question is what do we want our selection criteria to be: "how many"
> or "how long?"
>
Hi Joe
Here is my view on it.
My idea was to have X days worth of files in the incremental store. That
would allow to say that I want 90 days of incremental backups. In my
situation, if a file hasn't changed for 90 days it probably done.
I have no problem with not having a backup of a file in incrementals. If
that is a problem I need to increase the timespan I keep the files and
possibly increase my disk capacity if that is needed. That is not a BOBS
problem, but a backup administrators problem.
Thew way I delete files is to set some limits. Say files older than 6
months AND larger than 500 MB ( and sometimes by type, like .avi, .mov,
mpg and so on. Depends on the work being done.)
Generally my tolerance towards large files is biased so they are deleted
first.
A 500MB file is seldom a "from scratch work" but something made from
another file. Like a movie rendering. Exceptions are files like
photoshop (.psd) files which can be large and original works.
That is just my situation others probably have different scenarios.
Another feature I like is to have bobs decide when incrementals are
deleted. Say, when only 50 GB of diskspace is left, delete until there
is 100 GB free, oldest files first. (I guess this will be a problem when
having different time spans for the various backups.)
Anyway. A simple "delete after X incremental backups" would be a good
start. X not being days, since we can be sure the backup is run every
day, requires a little date manipulation or some other system of telling
when and what to delete.
Please used the database(s) to search for files to delete. The time on
disk is not accurate. I think it reflects the time the file was last
changed (on the originating server).
The database has the correct information by using the date tagging
system.
I used the find command when deleting but I know it isn't the correct
way to do it.
Cheers
Rene
|
|
From: Joe Z. <jz...@co...> - 2004-04-25 16:20:45
|
Rene Rask wrote:
>On Sun, 2004-04-25 at 07:18, Joe Zacky wrote:
>
>
>>Rene,
>>
>>I was going to add a routine to cleanup incremental backups but now I'm
>>not sure. I was planning to remove incremental files that were backed up
>>over X days ago, but now I see the field text is "How many incremental
>>backups to keep (0 = infinite)", which seems to mean how many copies of
>>an incremental file to keep. So if incremental is set to 9, then there
>>could be up to 9 copies of any incremental file. This means the
>>incremental backup directory could be up to 9 times as large as the
>>current backup.
>>
>>Here's some points to ponder:
>>
>>If we keep X copies of incrementals:
>>o The amount of storage required "could" be much more than if we kept
>>X days.
>>o It's going to be programmatically difficult and time consuming to
>>read through all the file names in incremental and count how many there
>>are of each.
>>
>>If we keep X days worth of incrementals:
>>o The only backup copy of a file would be deleted after X days if the
>>file hasn't been changed. That's probably not a good thing.
>>o Removing the incremental files is easily done with a command like this:
>> find /path/to/bobsdata/incremental/<server>/<share>/ -type f
>>-ctime +<days> -ctime -999 -exec rm -v {} ';'
>>
>>So I'm wondering 1) what your intention is for this field, 2) what are
>>you doing on your bobs system to cleanup incremental files, and 3) how
>>do you suggest I proceed?
>>
>>The question is what do we want our selection criteria to be: "how many"
>>or "how long?"
>>
>>
>>
>
>Hi Joe
>Here is my view on it.
>
>My idea was to have X days worth of files in the incremental store. That
>would allow to say that I want 90 days of incremental backups. In my
>situation, if a file hasn't changed for 90 days it probably done.
>
>
Here you're saying that X is "how long".
>I have no problem with not having a backup of a file in incrementals. If
>that is a problem I need to increase the timespan I keep the files and
>possibly increase my disk capacity if that is needed. That is not a BOBS
>problem, but a backup administrators problem.
>
>Thew way I delete files is to set some limits. Say files older than 6
>months AND larger than 500 MB ( and sometimes by type, like .avi, .mov,
>mpg and so on. Depends on the work being done.)
>Generally my tolerance towards large files is biased so they are deleted
>first.
>A 500MB file is seldom a "from scratch work" but something made from
>another file. Like a movie rendering. Exceptions are files like
>photoshop (.psd) files which can be large and original works.
>
>That is just my situation others probably have different scenarios.
>
>Another feature I like is to have bobs decide when incrementals are
>deleted. Say, when only 50 GB of diskspace is left, delete until there
>is 100 GB free, oldest files first. (I guess this will be a problem when
>having different time spans for the various backups.)
>
>Anyway. A simple "delete after X incremental backups" would be a good
>start. X not being days, since we can be sure the backup is run every
>day, requires a little date manipulation or some other system of telling
>when and what to delete.
>
>
Here you're saying X is "how many." I'm confused, should X be number of
days or number of files?
>Please used the database(s) to search for files to delete. The time on
>disk is not accurate. I think it reflects the time the file was last
>changed (on the originating server).
>The database has the correct information by using the date tagging
>system.
>I used the find command when deleting but I know it isn't the correct
>way to do it.
>
>
The tests I ran on 2 of my redhat systems showed that using find with
-ctime picked the dates the file was backed up. That is, not the date it
was last modified (-mtime), but the date it was rsync'd to bobs.
I didn't realize that information was in the database. I never did
anything with the database so I wasn't thinking about it. I'll have to
play with that - I agree that sounds like the right way to find the files.
>
>Cheers
>Rene
>
>
>
|
|
From: Rene R. <re...@gr...> - 2004-04-25 18:13:12
|
On Sun, 2004-04-25 at 18:16, Joe Zacky wrote: > >Anyway. A simple "delete after X incremental backups" would be a good > >start. X not being days, since we can be sure the backup is run every > >day, requires a little date manipulation or some other system of telling > >when and what to delete. > > > > > Here you're saying X is "how many." I'm confused, should X be number of > days or number of files? > X should be the number of backup runs. Example: Backups run on monday,tuesday,and wednesday. I have chosen to keep 20 incremental backups. 20/3 is 6.6 times one week (7 days) which gives 46.6 days to keep backups and we round the number up to 47 to be safe. 20/3 is the number of weeks times 7 is the day count. I hope this makes sense. > > > The tests I ran on 2 of my redhat systems showed that using find with > -ctime picked the dates the file was backed up. That is, not the date it > was last modified (-mtime), but the date it was rsync'd to bobs. > > I didn't realize that information was in the database. I never did > anything with the database so I wasn't thinking about it. I'll have to > play with that - I agree that sounds like the right way to find the files. It seems like I'm wrong here. Guess we could use the find method. But I'd still prefer the db method. That is better if we want to do more advanced things in the future. Cheers Rene |
|
From: Joe Z. <jz...@co...> - 2004-04-25 22:03:02
|
Rene Rask wrote: > >X should be the number of backup runs. >Example: Backups run on monday,tuesday,and wednesday. >I have chosen to keep 20 incremental backups. 20/3 is 6.6 times one week >(7 days) which gives 46.6 days to keep backups and we round the number >up to 47 to be safe. > >20/3 is the number of weeks >times 7 is the day count. > >I hope this makes sense. > > It makes mathematical sense, but it's confusing and not intuitive. I would never guess that by looking at the configuration screen. Additionally, if someone changes the days of their backups, say from 3 to 5, that shortens the retention period even though backups have only been running 3 times per week instead of 5. If we use this method, we should calculate the retention date and show that on the screen so people would know what it is. >It seems like I'm wrong here. Guess we could use the find method. But >I'd still prefer the db method. That is better if we want to do more >advanced things in the future. > > The database method is safer and more accurate because the modification date in the incremental archive could be manually changed, say if someone moved the directory or restored it from a tape backup. I'll play with the database. The 'find' method was just so easy... Cheers, Joe |
|
From: Rene R. <re...@gr...> - 2004-04-25 22:41:27
|
On Sun, 2004-04-25 at 20:59, Joe Zacky wrote: > Rene Rask wrote: > > > > >X should be the number of backup runs. > >Example: Backups run on monday,tuesday,and wednesday. > >I have chosen to keep 20 incremental backups. 20/3 is 6.6 times one week > >(7 days) which gives 46.6 days to keep backups and we round the number > >up to 47 to be safe. > > > >20/3 is the number of weeks > >times 7 is the day count. > > > >I hope this makes sense. > > > > > It makes mathematical sense, but it's confusing and not intuitive. I > would never guess that by looking at the configuration screen. > Additionally, if someone changes the days of their backups, say from 3 > to 5, that shortens the retention period even though backups have only > been running 3 times per week instead of 5. If we use this method, we > should calculate the retention date and show that on the screen so > people would know what it is. A message like: "Backups will be retained for 45 days with the current setting" should do the trick. I would expect incremental to behave as I described. I guess my way of thinking about the backup is a bit different from yours, so it is good to explain this to people in an easy to understand way. > >It seems like I'm wrong here. Guess we could use the find method. But > >I'd still prefer the db method. That is better if we want to do more > >advanced things in the future. > > > > > The database method is safer and more accurate because the modification > date in the incremental archive could be manually changed, say if > someone moved the directory or restored it from a tape backup. I'll play > with the database. The 'find' method was just so easy... I suggest you take a look at the code where I use the db for searching. You basically just make a search for files older than X. Just like you can do with the current setup. It shouldn't be hard since its basically done already :) Cheers |
|
From: Arnaud C. <arn...@ar...> - 2004-05-11 08:51:05
|
---------------------------------------------------------------------------= ---------------- Ce message et toutes les pi=E8ces jointes sont confidentiels et =E9tablis = =E0 l'attention exclusive de ses destinataires. Toute utilisation de ce message non conforme =E0 sa destination, toute diff= usion ou toute publication, totale ou partielle, est interdite, sauf autori= sation expresse. Si vous recevez ce message par erreur, merci de le d=E9truire et d'en avert= ir imm=E9diatement l'exp=E9diteur. Le Groupe C3D, ses filiales et ses partenaires d=E9clinent toute responsabi= lit=E9 au titre de ce message s'il a =E9t=E9 alt=E9r=E9, d=E9form=E9 ou fal= sifi=E9. This message and all attachments are confidential and intended solely for t= he addressees. Any use not in accord with its purpose, any dissemination or disclosure, ei= ther whole or partial, is prohibited except formal approval. If you receive this message in error, please delete it and immediately noti= fy the sender. Neither C3D Group nor any of its subsidiaries or affiliates shall be liable= for the message if altered, changed or falsified. ---------------------------------------------------------------------------= ---------------- |
|
From: Joe Z. <joe...@za...> - 2004-05-11 16:30:08
|
Arnaud CONCHON wrote:
> any news on this part ? i'm very very interested in an automatic
> cleanup feature :)
> i just checked this morning .. my bobs backuped 11GB of incremental
> data ->
> [root@srv-backup bobsdata]# du -hs *
> 4.0K admin.pwd
> 163G current
> 4.0K incoming
> 47G incremental
>
> incremental data is growing very fast ....every day !
> cheers.
>
I've been working on it slowly for a couple weeks. It should be in cvs
within another week or two.
In the meantime, this is what I do to remove old incremental files. This
will remove files older than 14 days:
find /var/bobsdata/incremental/<server>/<share>/ -ctime +14 -ctime -999
-exec /bin/rm -v {} ';'
Cheers,
Joe
|
|
From: Arnaud C. <arn...@ar...> - 2004-05-11 16:38:21
|
personnaly i would prefer to be able to keep a maximum of X copies of
anyfile at anytime, rather than "older than X days" ....
better... if i can define something like: keep 2 copies of all files at
all times AND fresher than 30 days => it would be perfect :)
thanks for all the good work!
Arnaud.
Joe Zacky a écrit :
>
>
> Arnaud CONCHON wrote:
>
>> any news on this part ? i'm very very interested in an automatic
>> cleanup feature :)
>> i just checked this morning .. my bobs backuped 11GB of incremental
>> data ->
>> [root@srv-backup bobsdata]# du -hs *
>> 4.0K admin.pwd
>> 163G current
>> 4.0K incoming
>> 47G incremental
>>
>> incremental data is growing very fast ....every day !
>> cheers.
>>
> I've been working on it slowly for a couple weeks. It should be in cvs
> within another week or two.
>
> In the meantime, this is what I do to remove old incremental files.
> This will remove files older than 14 days:
>
> find /var/bobsdata/incremental/<server>/<share>/ -ctime +14 -ctime
> -999 -exec /bin/rm -v {} ';'
>
> Cheers,
> Joe
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by Sleepycat Software
> Learn developer strategies Cisco, Motorola, Ericsson & Lucent use to
> deliver
> higher performing products faster, at low TCO.
> http://www.sleepycat.com/telcomwpreg.php?From=osdnemail3
> _______________________________________________
> Bobs-devel mailing list
> Bob...@li...
> https://lists.sourceforge.net/lists/listinfo/bobs-devel
|
Thanks for helping keep SourceForge clean.
X