Re: [Bacula-devel] Bacula migration development

[Kern S. <ke...@si...>]

On Saturday 19 November 2005 17:47, Georg Altmann wrote:
> --On Friday, November 18, 2005 18:59:44 +0100 Kern Sibbald
>
> <ke...@si...> wrote:
> > I'd suggest you start as follows:
> >
> > Read a bit of the first chapter of the developers manual -- especially
> > the  copyright assignment for important contributors who are employed --
> > I need  the signoff from their company.
>
> I will send the copyright statement to you as soon as necessary.
> I've skimmed through the developer manual and a bit of the sd code. I think
> I'm fine with coding style and practices for what I've seen. :-)
> For now I would first like to have a real bacula installation running to
> get a feeling for how everything works. This will probably take two weeks.
> After that I think I can start writing some actual code. I will try track
> cvs changes in the meantime.

OK. This is fine.  I assume that you know that there is a "bacula-commits" 
mailing list that you can subscribe to where you will get diffs of all the 
commits ...

>
> > Then for the project itself, there are three major aspects:
> > 1. Work in the Storage daemon to be able to read/write multiple devices.
> > Probably 60-80% of the work is already done.  It basically involves
> > having  multiple device control records (DCR).  Each DCR allows a job to
> > read/write a  device (possibly with other jobs writing at the same time).
> > Currently there  is only one pointer to one DCR.  This needs to be
> > expanded to be a pointer to  a DCR for reading and a list of DCRs for
> > writing. The read DCR could very  well be a list too, but I think for
> > the moment that is overkill.
> >
> > 2. There needs to be a mechanism in the Director to send multiple Storage
> > devices to the SD: one for reading and more than one for writing.  The
> > basic  structures exist, it is just a matter of implementing it.
> >
> > 3. There needs to be some logic in the Director that decides what is
> > going to  be Merged -- i.e. what SD + data is going to be read and where
> > it is going to  be written.
> >
> > That is a really rough outline of the project ...
> >
> > What do you think?
>
> Hmm, my knowledge of bacula is still very weak, but it don't understand why
> you would use more than one dcr at a time in a sd. In the developer docs
> for the sd you mention, that "an individual storage deamon is associated
> with a physical permanent storage device."
> So to my understanding you would rather have two sd processes A and B,
> where A reads a backup stream and sends it to B for storage. So you would
> define some copying protocol between two sds and change the sd<->director
> protocol to make the director issue the necessary control commands.
> I guess this is to some degree similar to what happens between the sd and
> the fd during a restore.

In the Storage daemon, there is a Device resource (i.e. from conf file) that 
describes each physical device.  When the physical device is used it is 
controled by the DEVICE structure (defined in dev.h), and typically refered 
to as dev in the C++ code.  Anyone writing or reading a physical device must 
ultimately get a lock on the DEVICE structure -- this controls the device.  
However, multiple Jobs (defined by a JCR structure src/jcr.h) can be writing 
a physical DEVICE at the same time (of course they are sequenced by locking 
the DEVICE structure).  There are a lot of job dependent "device" variables 
that may be different for each Job such as spooling (one job may spool and 
another may not, and when a job is spooling, it must have an i/o packet open, 
each job has its own record and block structures, ...), so there is a device 
control record or DCR that is the primary way of interfacing to the physical 
device.  The DCR contains all the job specific data as well as a pointer to 
the Device resource (DEVRES structure) and the physical DEVICE structure. 

Now if a job is writing to two devices (it could be writing two separate 
streams to the same device), it must have two DCRs.  Today, the code only 
permits one.  This won't be hard to change, but it is new code.

Today three jobs (threads), two physical devices each job
   writes to only one device:

  Job1 -> DCR1 -> DEVICE1
  Job2 -> DCR2 -> DEVICE1
  Job3 -> DCR3 -> DEVICE2

To be implemented three jobs, three physical devices, but
    job1 is writing simultaneously to three devices:

  Job1 -> DCR1 -> DEVICE1
          -> DCR4 -> DEVICE2
          -> DCR5 -> DEVICE3
  Job2 -> DCR2 -> DEVICE1
  Job3 -> DCR3 -> DEVICE2

  Job = job control record
  DCR = Job contorl data for a specific device
  DEVICE = Device only control data

>
> I also don't understand what you mean by merging in 3.

Merging is called in other products: "consolidation" or "virtual full backup".  
It simply means that Bacula takes a Full, the last Differential, and all the 
Incrementals after the last Differential, and reads all the data (like a 
restore), then writes it to a new Volume(s), possibly in a different pool. 
This is not yet implemented.

>
> Hopefully I didn't get it all wrong... Please let me know if I did.
> I think I really need to get into bacula a bit more to talk about such
> fundamental design decisions...
>
> Regards,
> Georg
>
>
> --
>
> Georg Altmann <gal...@la...>
> LAS-CAD GmbH, Munich, Germany
> http://www.las-cad.com

-- 
Best regards,

Kern

  (">
  /\
  V_V