[Bacula-devel] Re: DVD writing

[Nicolas B. <ni...@bo...>]

Hello Kern,

On Thu, 2004-12-30 at 15:40 +0100, Kern Sibbald wrote:
> Hello Nicolas,
> 
> On Thu, 2004-12-30 at 15:05 +0100, Nicolas Boichat wrote:
> > Hello,
> > 
> > On Thu, 2004-12-30 at 11:26 +0100, Kern Sibbald wrote:
> > > On Thu, 2004-12-30 at 02:30 +0100, Nicolas Boichat wrote:
> > > > Hello,
> > > > 
> > > > > > > No, I have no objection. You will need to think carefully about this
> > > > > > > though. I don't think Bacula will be able to do a restore from a
> > > > > > > partially written DVD and from files. You will need to know that
> > > > > > > JobMedia records are written by the existing file-writing code. You will
> > > > > > > probably want to suppress this by setting a flag in the dcr record when
> > > > > > > writing to File. When you are writing the JobMedia records (they are
> > > > > > > used to seek to the right place on tape), you will probably want to use
> > > > > > > the VolIndex field to be the VolPart number. The VolIndex is not
> > > > > > > currently used for File Volumes.
> > > > > > 
> > > > > > I'm not sure to understand everything... I think I'll start to do some
> > > > > > tests, and code a little to see what happens .-)
> > > > > 
> > > > > Yes, that is the best approach. Please follow how
> > > > > dir_create_jobmedia_record() works in src/stored/block.c then you will
> > > > > understand better. 
> > > > 
> > > > ... I still don't understand... I've coded something that works, without
> > > > modifying how JobMedia are handled...
> > > 
> > > Yes, it will work, but you will have left over JobMedia records when you
> > > remove the disk parts that will "polute" the database. You need to think
> > > about how Bacula is going to know what parts are on disk and which parts
> > > are on DVD. This will be important if you want to do a restore. The
> > > other aspect is how to make the restore know what parts are on what
> > > device. As I mentioned previously, one of the current weaknesses of
> > > Bacula is handling two MediaTypes in the same Job.  When you get to that
> > > point, just ask, and we will see if we can come up with a solution -- I
> > > have some ideas, but it will be a bit of work -- it basically requires
> > > two Pools, the first one a File Pool, where you initially put the parts,
> > > and the second one a DVD pool, which is chained to the first Pool. The
> > > parts are then "migrated" from the File Pool to the DVD Pool, and all
> > > will be *really* cool.  For the restore to work correctly, we will need
> > > to implement a concept of a Pool, Device or a MediaType in the bootstrap
> > > file.
> > 
> > I thought to something much simpler... Only the last part file is on the
> > hard disk (e.g. in a temporary directory), all the others are on the DVD
> > (e.g. in the mounted directory).
> 
> Yes, that is pretty simple. During the restore, how does Bacula know if
> there is still a part on disk?

There will always be a part on the disk (maybe an empty). In the worst
case, open_dev will create it when it tries to read it, so it will be
empty, and will return an EOF immediately.

Note that the VolParts field in the Media record represent the total
number of parts which are written to the DVD, so there is always one
more (the part which is on the disk).

> > The JobMedia records are used to restore the files, as they give
> > addresses in the Volume (which are translated so the right part file is
> > opened).
> 
> Hmmm. Maybe the problem is much less important than I thought. As long
> as you transfer the full file to the DVD, all the JobMedia records will
> remain valid, since they are all relative to the part in question. So,
> for the moment, forget about this -- I no longer think it is an issue.

Ok, nice to hear it... .-)

> > > > Basically, I consider a set of files as one big file.
> > > > I modified every occurence of lseek by lseek_dev, which change of part
> > > > file if necessary. I modified read_block_from_dev function so when it
> > > > comes to the end of a part file, it does not return EOF, but open the
> > > > next part file. I also modified write_block_to_dev so when MaxPartSize
> > > > is reached, it opens the next part and write to it.
> > > 
> > > There is some terrible ugliness in reading of disk files, which could
> > > bite you. I'll explain this later. I don't want to overload you too
> > > much.
> 
> This falls much into the same area as the JobMedia records. As long as
> all the data for a given part is either all on disk or all on DVD, the
> ugliness is hidden.  Basically, it has to do with the fact that blocks
> on a disk do not need to be a multiple of the basic block size -- they
> can be smaller. This means that when a block is read, if it is a short
> block, the next block will be wholly or partly in the buffer. To
> simplify the code, I shuffle the second block up to the beginning of the
> buffer when it is going to be read, and read the remainder of the block
> from disk. A bit ugly, but it works ...  I don't think it is an issue
> for you given that parts are either on disk or DVD.

Ok thank you for the explanation.

> > Could you tell me more about this? For the moment I don't see any
> > problem, but I prefer to know more than necessary...
> > 
> > > > I choosed the simpler way that I found, and I think writing to a DVD
> > > > will be really easy now... Using this we'll be able to restore from a
> > > > partially written DVD, and even from part files that have not yet been
> > > > written to the DVD.
> > > 
> > > See the above discussion of JobMedia records. These records are the
> > > *key* to doing restores. See if you can figure out what they are and
> > > what they mean. If you have *any* questions, please ask.
> > 
> > I think I understand how they works, and as far as I can see, everything
> > works fine (I can restore from a Volume which has more than one part)...
> > 
> > > PS: I have been thinking about turning a DEVICE into a real class and
> > > to make all the dev_xxx() calls methods of the class. The big problem
> > > is the number of places in the code where the dev structure is 
> > > accessed directly.  Since you are in that area of the code, I would
> > > be interested in your opinion.
> > 
> > I think it would be a good idea. The fd should be private, so
> > read/write/lseeks are done within the class. It would make the code I
> > just wrote much cleaner, by moving all the tests (check for part end)
> > outside block.c routines (read_block_from_dev and write_block_to_dev).
> > By creating lseek_dev, I already moved a part of the tests away, but
> > there should also be a read_dev and write_dev (which should be in DEVICE
> > class).
> > In the end, block.c should not be aware of what type of device it is
> > reading/writing (this is already almost the case).
> 
> I don't want to make more work for you, but if this interests you, now
> would be a good time to do it. I have wanted to do this for a long time
> now.  The easiest way would be start start by simply adding methods and
> leave most of the structure exposed, (obviously fd could be private from
> the beginning), then little by little make more and more of the
> structure variables private, and where necessary provide access methods.
> 
> In creating methods, I would like to drop the dev from read_dev() and
> simply have dev->read(), and likewise for all the others ...
> 
> This is quite a little project in itself, so please don't let this idea
> divert you from your current project, which will really add a nice
> feature. :-)

Ok, for the moment I won't work on this, because I really want to have
DVD-writing working before the end of my holidays (on Januar 9), as I
won't have much time after that. I'll work on it if I find enough time.

Best regards,

Nicolas