From: Stieneke, D. <Dan.Stieneke@ARS.USDA.GOV> - 2012-12-07 18:29:28
|
Reading the mail archive, it seems I'm not alone in wanting to see concurrent spooling/despooling. Could we hand off the job to a standalone spooler like mbuffer (http://www.maier-komor.de/mbuffer.html)? For a simple tar to LTO-4 job it makes a tremendous improvement. The Bacuala developers would need to modify the storage daemon to allow the datastream to be diverted out and then back in through a pair of FIFOs before it gets [properly re-blocked? and] written to tape. The user would need to tie the FIFOs together with an external spooler. It might look like this: ------/etc/bacula-sd.conf------ <snip> Device { ... PushToFIFOSpooler = /path/to/bufreceive # bufreceive is a FIFO on this machine PullFromFIFOSpooler = /path/to/bufsend # bufsend is also a FIFO ... } </snip> ------/etc/bacula-dir.conf------ <snip> Job { ... RunBeforeJob = "mbuffer -i /path/to/bufreceive -o /path/to/bufsend" # default puny 2M RAM buffer; can be set to multi-gig RAM or disk # Alternatively: RunBeforeJob = "mbuffer < /path/to/bufreceive > /path/to/bufsend" ... } </snip> This would - be a lot easier than asking the developers to reinvent the wheel (insert your own ring-buffer pun here...) - let the developers all but ignore spooling & focus on real backup stuff - give users access to many more options, such as using RAM or disk spool, high/low-watermarks, and rate-limiting - let folks cleverer than me figure out other cool things to do with the datastream Thanks, Dan Stieneke IT Specialist USDA - ARS - NWISRL 3793 N 3600 E Kimberly, ID 83341 208/423-6519 This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately. |
From: Radosław K. <rad...@ko...> - 2012-12-10 08:02:18
|
Hello, 2012/12/7 Stieneke, Dan <Dan...@ar...> > Reading the mail archive, it seems I’m not alone in wanting to see > concurrent spooling/despooling. > > > > Could we hand off the job to a standalone spooler like mbuffer ( > http://www.maier-komor.de/mbuffer.html)? For a simple tar to LTO-4 job it > makes a tremendous improvement. > > > The main problem with concurrent spooling/despolling in Bacula Storage Daemon is not a buffer/fifo (which is already implemented because we have a spooling functionality already) but single job thread. It is not easy to implement concurrency with a current code. In this case mbuffer does not simplify things. > The Bacuala developers would need to modify the storage daemon to allow > the datastream to be diverted out and then back in through a pair of FIFOs > before it gets [properly re-blocked? and] written to tape. > Simply speaking developers have to implement a multiple threads for single job which write data to tape. This is a main missing code for build in concurrent spooling/despooling. :) The user would need to tie the FIFOs together with an external spooler. It > might look like this: > > > > ------/etc/bacula-sd.conf------ > > <snip> > > Device { > > … > > PushToFIFOSpooler = /path/to/bufreceive # bufreceive is a FIFO on > this machine > First thread to read data from client and write data to the buffer - implemented already. > PullFromFIFOSpooler = /path/to/bufsend # bufsend is also a FIFO > Second thread to read data from buffer and write to bacula device - missing. best regards -- Radosław Korzeniewski rad...@ko... |