Re: [Bacula-devel] Re: [Bacula-users] Multiple simultaneous jobs in Bacula

[Kern S. <ke...@si...>]

Hello,

On Fri, 2003-06-20 at 15:28, Jari Fredriksson wrote:
> >Actually, when I started 3 1/2 years ago, each client was process and
> >I was *very* reluctant to use interprocess communications within
> >a single daemon for passing buffers from the client to a tape
> >writing process.  Today, this would be a piece of cake with
> >threads, but the current code works and there are other higher
> >priority needs -- at least for the moment.
>=20
>=20
> Yep, I understand that. Reality says we can't rewrite all our work from
> scratch every day, have to stick with our work and gradually develop it;)

I agree. I try hard to rewrite 10-20% of my code each year -- well,
I mean the old code that needs rewriting.

>=20
> >Although I will someday go to this scheme, it is not because it
> >will simplify the locking, but because I can have a large pool of
> >buffers with minimum bookkeeping.
>=20
> Yes it might simplify many things.
>=20
> >Even with the FIFO scheme, the same locking difficulties arise
> >concerning "unmounting" a drive. The FIFO code is using
> >the device, and the user wants take it away and
> >"unmount" it.
>=20
> I don't know... Btw, I didn't mean actual FIFO as in mkfifo, but I might
> implement that for example as a linked list. Or a combination of an in
> memory linked list (for book keeping) and the data stored in /tmp etc..
> (buffering/pooling)

Yes, I understood first in first out (not a pipe).

>=20
> Something like this:
>=20
> // Thread receiving data to store
>=20
> LOOP
>   GET data
>   LOCK
>   ADD data TO fifo
>   UNLOCK
> UNTIL last-data
>=20
> ----------------------------------------
>=20
> // Thread handling tape/storage
>=20
> LOOP
>   LOCK
>   GET data FROM fifo
>   UNLOCK
>   REPEAT
>     WRITE data TO TAPE
>     IF end-of-volume THEN
>       YELL NEW TAPE
>     END
>   UNTIL ok
> UNTIL last-data
>=20
> ----------------------------------------
>=20
> That way, the locks would be very minimal, and the "out-of-tape" conditio=
n
> would not affect the data collection threads in no way. LOCKING would be
> needed just to keep the FIFO in order - depending of the implemention of =
the
> FIFO it might not even need a lock/mutex.
>=20
> ^That is of course very simplistic model;)

Well, as always, things are a bit more complicated.  In this case, it
is the operator (i.e. another thread that wants to take the tape=20
away from the thread handling the tape/storage.  Your loop looks
more like the following (actually even more complicated):

// Thread handling tape/storage
 LOCK device
 LOOP
   LOCK fifo
   GET data FROM fifo
   UNLOCK fifo
   REPEAT
     WRITE data TO TAPE
     IF end-of-volume THEN
       YELL NEW TAPE
     END
   UNTIL ok
 UNTIL last-data
 UNLOCK device

// Read thread for restore
 LOCK device
 ...
 UNLOCK device

// User thread from console wanting to unmount the
//   drive
  TRY_TO_LOCK device
    Fail: tell user it is busy
        Exit
    Succeed: unmount device
        Exit

Now, the TRY_TO_LOCK is really MUCH more than simply trying
to get the lock.  It must get it if the tape handling=20
thread is at the "end-of-volume", but the tape handling
thread cannot UNLOCK the device at that point or a read
thread may capture it, ...

Right now, the console thread "knows" when it can steal
the lock, but I don't like the code because it relies on
him "knowing" intimate details about the tape handling
thread.=20

The new idea I had the other day (quite trivial
really) will add code that allows the tape writing thread
to continue to hold the lock, but to "cede" it to the console
thread or "put it up for grabs" all the while holding it.
The console thread then tries to "steal" the lock,
and if it has been ceded, it gets it.  The tape writing thread
must then "get_back" his lock after ceding it. With the
new code, the console doesn't have to know the internal state
of the tape handler, only that the lock is not locked or
that it has been "ceded" -- much cleaner.

Thanks for the comments. It helps me clarify my own ideas.

Best regards,

Kern