Re: [Bacula-users] Advice for disk and tape backup plan

[Don M. <mac...@cm...>]

Thanks Arno, you always seem to have very helpful insights.  These
features would definitely represent a step up in complexity, and
functionality.  

Honestly, I'd love the restore-from-one-of-several-copies feature even
if I had to choose which volume to use for the restore from a list of
candidate presented on a menu, and wait for the more complex
prioritization code to come out as a separate module/code
block/whatever...

I'm guessing there are shops that fit the scenario of copying between
SDs, but I'm guessing the majority would be pleased to be able to do it
within a single SD and, also, wait for the multi-SD functionality as a
separate project.  

Again, thanks for your many contributions to the list; I've benefited
from your knowledge several times.

If I had the skills or the $$ I'd be part of the solution in a New York
minute (instantly).  

On Wed, 2007-02-07 at 10:46 +0100, Arno Lehmann wrote:
> Hi,
> 
> I'm following your discussion for a while now...
> 
> On 2/7/2007 3:54 AM, Don MacArthur wrote:
> > I think some of the code contains what might be a good solution.  
> 
> What you want seems to be he ability to copy volumes,possibly to another 
> SD, and keep complete catalog inform- Deciding which volumes / jobs to restore from: A concept of a 'volume 
> cost' representing the efort needed to access volumes wil be helpful. 
> For example, define the cost for disk based volumes to be lower than 
> that of tapes stored locally, and the cost of off-site tapes would be 
> highes.ation about both copies.
> 
> That part is probably rather simple to code (says another non-programmer 
> :-) because a big part of what is needed already exists with the 
> migration jobs.
> 
> > Right now, when a scheduled backup issues a mount request for a tape, it
> > doesn't matter what tape I load as long as it meets the requirements to
> > be used.  If it asks me for volume123 from pool xyz, and I load
> > volume456 from pool xyz the job will run (again, assuming the status of
> > the tape is correct and there aren't any other problems like file number
> > mismatch).  
> > 
> > I'm haven't coded for ... years, but I wonder if the same logic that
> > allows the director or storage daemon to determine that it didn't get
> > what it asked for (volume123), but did get what it wanted (a "qualified"
> > volume), could be applied to this situation.  That is, read the label,
> > query for the file and version, and if the mounted volume has the file,
> > rock and roll.  If not, explain it to the operator and iterate the mount
> > request.  
> > 
> > Yes?
> 
> As far as I understand, the restore part is what will be most difficult 
> to solve; currently, Bacula is not able to choose from different copies 
> of a job. This would have to be thought through first.
> 
> The user interface to such a function would be quite straightforward as 
> you outline, but the key problem lies behind the scenes.
> 
> I assume that job copies will be implemented some day, but we'd need a 
> capable programmer for that... There is a feature request dealing with 
> copies already, by the way.
> 
> Many details need to be worked out before implementation can begin, I 
> assume. Three main areas might be:
> - Deciding which volumes / jobs to restore from: A concept of a 'volume 
> cost' representing the efort needed to access volumes wil be helpful. 
> For example, define the cost for disk based volumes to be lower than 
> that of tapes stored locally, and the cost of off-site tapes would be 
> highes.
> First, the DIR has to learn that there can be multiple copies of one 
> job, though - this is not something it understands today, I think.
> - Moving data without expiring the catalog information about the source 
> job. This should be rather simple.
> - Allowing the transfer of data from one SD to another one. Today, IIRC, 
> migration only works inside one SD. That is ok for migration from disk 
> to tape, but fore real copies one will want the ability to create 
> off-site volumes directly, i.e. copy from a local disk volume to a 
> remote tape.
> 
> I imagine that the first and the last item will require most of the 
> implemetation time, and I'm qute sure that I overlooked many interesting 
> things ;-)
> 
> Anyway, that discussion probably belongs to the -devel list.
> 
> Today, if you want copies of jobs or volumes, you have only two choices: 
> Run the jobs twice, to different storage devices, and be prepared for 
> differences between jobs and manual decision which one to use for 
> restores. Or copy the volumes outside of Baculas scope, using simple 
> file copies for disk volumes, tape-to-tape dumps for tape volumes, or 
> use bcopy. In any case, you'll have to manage the copies manually, too, 
> but you will have identical copies.
> 
> Anyway, this mainly repeats what has been written in this thread before. 
> What I find most interesting, though: Is one you intersted enough in the 
> job copy feature to sponsor it or to implement it himself? I'm quite 
> sure you'll get support from Kern and the other developers, and I'm also 
> quite sure that many users of Bacula will be very grateful...
> 
> Arno
> 
> > 
> > On Tue, 2007-02-06 at 18:31 -0800, Tauren Mills wrote:
> > 
> >>This sounds like a reasonable approach.  If I request to restore a
> >>particular object and the exact object version exists on multiple
> >>volumes, it really doesn't matter which volume it is restored from.
> >>As long as that object gets restored, I will be happy.  Of course, it
> >>could make my life easier if bacula would automatically choose a
> >>volume that was mounted, such as a disk volume over a tape volume.  Or
> >>if I'm doing a manual restore, it could ask me interactively which
> >>volume to restore from, suggesting one that is mounted if available.
> >>
> >>The bottom line is that it would have information about both copies of
> >>the same object, and then logic could be added to choose which one to
> >>restore.  That logic could be very simple, such as "restore from the
> >>first volume in the list that contains the object" or more
> >>sophisticated such as "restore from a volume that contains the object,
> >>give preference to disk volumes over tape volumes".
> >>
> >>Of course, I haven't looked at any of the bacula code, so these are
> >>just perceptions from the outside as a user.  Those with more
> >>experience with the internals may see issues that I do not at this
> >>time.
> >>
> >>On 2/6/07, Don MacArthur <mac...@cm...> wrote:
> >>
> >>>I was one of the more recent posters on this topic.
> >>>
> >>>The tools I have used in the past used whichever volume was loaded as
> >>>long as it was one of the ones it asked for.  So, if the original backup
> >>>was on volume123, and copied to volume456, a list of *all* the volumes
> >>>that contain the version of the object would be displayed, the label
> >>>read, and any volume (assuming it is on the list displayed) used.
> >>>
> >>>FWIW.
> >>>
> >>>On Tue, 2007-02-06 at 20:01 -0500, Ryan Novosielski wrote:
> >>>
> >>>>-----BEGIN PGP SIGNED MESSAGE-----
> >>>>Hash: SHA1
> >>>>
> >>>>Tauren Mills wrote:
> >>>>
> >>>>>>I agree.  It would be nice if there was a 'clone volume' command that
> >>>>>>would create a copy of the tape (or backup file) and create new records
> >>>>>>for the associated catalog data with pointers to the second volume.  I
> >>>>>>used to work with a Legato system and I'm pretty sure that it had that
> >>>>>>feature.  Or maybe Bacula does have that and I just haven't found it
> >>>>>>yet.
> >>>>>
> >>>>>That's exactly why I was posting!  I would think that a copy or clone
> >>>>>command would be a built in feature and that I just hadn't found it.
> >>>>>I'll have to figure out where to post feature requests I guess.
> >>>>
> >>>>Searched the list archives -- this has been discussed before. The
> >>>>condensed version of the problem is exactly what has been described
> >>>>prior: how would Bacula know from which volume you wish to restore? I
> >>>>don't have a good answer for that. I suspect Kern has thought about it a
> >>>>lot, but last I checked he wasn't sure how this would be done either. A
> >>>>lot of the work is done as a result of migration being possible now, but
> >>>>there are still pieces left.
> >>>>
> >>>>Also, check the current projects list; I believe this is on it
> >>>>(therefore, no reason for a feature request).
> >>>>
> > 
> > 
> > -------------------------------------------------------------------------
> > Using Tomcat but need to do more? Need to support web services, security?
> > Get stuff done quickly with pre-integrated technology to make your job easier.
> > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> > _______________________________________________
> > Bacula-users mailing list
> > Bac...@li...
> > https://lists.sourceforge.net/lists/listinfo/bacula-users
>