Re: [Bacula-devel] [Bacula-users] Storage definitions

[David B. <db...@si...>]

> Thanks for your thoughts.  I think there are several points that you
have
> minimized or overlooked in your response:
>=20
> 1. Bacula currently permits specifying multiple Media Types in a Pool.
> 2. Bacula currently permits Storage devices to be specified in the Job
> resource
> 3. Bacula currently permits both Pool and Storage overrides in Run
> resources.
>=20
> None of the above can be removed or changed without causing total
disaster
> to
> a very large Bacula community. =20

All understood. I'm still thinking about how to get to that future state
where these things are controlled by Bacula, and the user doesn't have
to concern themselves with it. Since migration is a significant new
feature, it strikes me as a very good opportunity to start heading in
that direction.=20

> For version 1.39.x, you can run Bacula exactly as you want using
Storage
> resources in Pools and in the Next Pool.  Both methodologies cooexist
in
> Bacula (though they probably don't function very well together).

Yes. It's a tradeoff of one-time setup complexity for pain-free normal
operation. The bigger problem (as you pointed out) is how to get there
from here.=20


> > > Migration job:
> > >  read storage: job storage resource
> >
> > Shouldn't the migration job always be using the Pool resource being
> > migrated
>=20
> One can do that, but it is not the only way of doing things.  The read
> part
> remains compatible with the existing code.

I guess I'm not seeing why it's useful to override device selection for
reads from pooled volumes -- I'm sure there's a reason, but I can't
figure it out. The volumes (and thus the jobs you are migrating) are in
a pool and are in a changer -- otherwise they wouldn't be eligible for
migration -- thus they are associated implicitly with a SD managing that
pool. Other than academic completeness, why complicate operations by
allowing/encouraging the user to mix into Bacula's device selection and
management process outside of real total disaster? That seems to be the
core of the problem -- Bacula having to cope with things outside it's
control or users doing things that are dumb. Should we use the
opportunity to try to remedy the possibility of that? Migration is a
*major* change in capability -- I guess I'm thinking that it's also the
beginning of the kinds of changes I suggested in order to scale up.=20

I know we can't break the world as it is now. It might be worth breaking
migration to work this way and start things in the direction we want to
go in the future.=20

> > For migration, we're looking at
> > volumes in pools, not individual volumes (other than the case where
we
> > have deliberately limited the selection criteria to a specific
volume
> > within a pool via the selection keyword),
>=20
> No, that is not really correct.  For migration, as it is currently
> implemented, we are looking at Jobs.  Everything breaks down to a Job.
> This
> has certain constraints, but I could see no other way of implementing
> Migration in the current Bacula otherwise.

I think this is one of those which-perspective-do-you-start-from issues.
At the lowest level, that's true - jobs are the most granular thing in
Bacula, and that's the units that the migration code deals with. From
the larger-scale storage manager's perspective, jobs are the *contents*
of a volume, and what they're interested is managing the availability of
volumes (empty, partially-full, full). The migration code identifies and
moves the jobs contained on volumes or the volumes contained within a
pool, but what I care about is if I have enough available space on
volumes to meet the requirements.=20

> > so there should be no reason
> > to specify individual devices for migration jobs.
>=20
> You have to be able to properly get to the original data, and I don't
> think it
> would be wise to have one syntax for restores and a different one to
find
> the
> read side of a Migration.

I probably didn't explain this point well. I intend the syntax to be the
same in both cases, but in the case of a total disaster (where the
bacula configuration and/or database is lost or otherwise unavailable),
you would have to ADD parameters identifying the SD and device to the
normal syntax. In a normal case, the job would get that data from the
Pool definitions (and their associated Storage definitions), and the
user should let Bacula do it's job and select appropriate devices.


> > >  read storage: pool storage resource
> >
> > Correct.
> >
> > >  read storage: run override resource
> >
> > See above. Migration is about pools, not devices.
>=20
> No, it is about Jobs, with the caveat that Next Pool does define what
set
> of
> devices it can be migrated to.

See above discussion on perspective. The mechanics in the code are about
jobs, but that's more detail than the average storage manager cares
about.=20

>=20
> > If you need to migrate
> > to volumes that are not in a pool then you set up a temporary pool
and
> > draw it's volumes from the scratch pool.
>=20
> All Volumes by definition are in one and only one pool, and as I said,
we
> are
> migrating jobs not volumes and not pools.

OK, my bad explanatory skills again. I was thinking of the case where I
need to dump a bunch of data to tapes that have not been previously used
with Bacula. Since I typically label and assign volumes to specific
pools for specific reasons, a bunch of "generally available" volumes
aren't part of that scheme. If I define a pool for that purpose (say
"Hurricane-Coming-Mass-Export" and allow that pool to draw from the
scratch pool (rather than specifically assigning tapes to it), then
that's what I would use as the NextPool for the migration job. I would
be clearing all jobs on my normal volumes onto the new tapes in the
Hurricane-Coming-Mass-Export pool so I could put the tapes in the
station wagon and flee. In that case *I* don't care about the individual
jobs; I care about whether I have copies of all the data on the normal
volumes -- it's the job of the Migration code (in my view) to worry
about those details while I worry about the physical things I can see
and manage.=20

> > >  write storage: pool next pool storage resource
> > >    (yea, try to figure out the above in C it is
> > >      job->pool->next_pool->storage
> >
> > Correct as I understand the implementation. See comments above.
>=20
> Yes, but it is terribly complicated, and IMO the "average" user
doesn't
> understand Pools so is going to break his brain on this.

On the other hand, it boils down to:=20

Initial setup of Bacula:=20

1) Define your devices in a Storage resource.=20
2) Define a Pool resource to hold your normal backup media and specify
which Storage resource will contain the media for this pool.
3) Define your normal backup media in the pool.=20
4) Define a backup job that references the pool of media by Pool name
5) run backup job.=20

Implementing migration (or replacing the current spooling code):=20

1) If necessary, define a new Storage resource (if your migration pool
uses different hardware than the normal stuff)
2) Define a new Pool to hold the media you want to use for
migration/consolidation
3) Define new media volumes in the new Pool.=20
4) Update the original Pool with a Next Pool =3D resource specifying the
new Pool.=20
5) Define a migration job for the original Pool with appropriate
selection criteria.=20
6) run your migration job periodically.=20

I think that's pretty clear (or could be made to be that way). Basic job
description specifies the Pool name and Bacula takes it from there. If
you need to go directly to tape for some reason, you override the Pool
resource in the job with the name of the Pool that is associated with
the Storage resource that has the tapes. User doesn't get involved with
devices at all

It might be interesting to think about making a change to the default
configuration that gets initially installed with the .debs or .rpms to
configure a disk changer and disk pool by default, and then use the
enabling-migration process above to add tape capability. That would make
the Bacula installation immediately useable on install (and with
disk-based backup becoming commonplace, this would be a very handy
thing, particularly when the Windows daemons become mainstream), and
propagate the changes forward w/o breaking the world for the existing
users. One could easily write a tool to do the definition steps for the
tape enablement.=20

Maybe 1.4-ish timeframe to change the initial config?=20


> > > Backup Job:
> > >  write storage: job storage resource
> >
> > See above for discussion of pools. If pools drive the selection of
> > volumes, the pool definition will clearly define what SD should be
used,
> > unambiguously and consistently across all actions in Bacula.
>=20
> Unfortunately the above, however good it may be, is not compatible
with
> the
> existing Bacula (if you force it on the user), and probably more than
50%
> of
> beginning users will not understand how pools work.   Any big
enterprise
> backup expert will, but not the majority of Bacula users, IMO.

I understand that there are migration issues. Consider this a discussion
of future direction -- there will need to be some explaining done.=20

> > > Restore Job:
> > >   read storage: who knows, probably the same as
> > >      Migration.  To be checked.
> >
> > In a normal case, the pool definition of the pool containing the
volume
> > should determine the SD to use.
> This is now possible, but it is not *enforced* so you just need to use
a
> little discipline in writing your .conf file.

OK. I'll see if I can come up with a cookbook install doc that reflects
this approach and we can kick it around here. As outlined above, it's
(IMHO) pretty straightforward, and would be more amenable to
autogeneration and long term manageability.=20

> > See comments above wrt to Migration. The same cases exist; migration
> > just makes this problem more visible.
>=20
> Yes, it is all a bit too much complicate.  Perhaps over time, I can
> deprecate
> features that add to the complexity such as Run overrides, but that is
not
> something that will happen any time soon.

Understood. I don't think I expected that you'd do it right now; I tend
to think about the "release after next", so I get ahead of the gory
details.=20
I do think it's worth the discussion, though, especially if we end up
with a much simpler "default" configuration, or a way to autogenerate
same.=20

-- db