Reading the mail archive, it seems I'm not alone in wanting to see concurrent spooling/despooling.
Could we hand off the job to a standalone spooler like mbuffer (http://www.maier-komor.de/mbuffer.html)? For a simple tar to LTO-4 job it makes a tremendous improvement.
The Bacuala developers would need to modify the storage daemon to allow the datastream to be diverted out and then back in through a pair of FIFOs before it gets [properly re-blocked? and] written to tape. The user would need to tie the FIFOs together with an external spooler. It might look like this:
PushToFIFOSpooler = /path/to/bufreceive # bufreceive is a FIFO on this machine
PullFromFIFOSpooler = /path/to/bufsend # bufsend is also a FIFO
RunBeforeJob = "mbuffer -i /path/to/bufreceive -o /path/to/bufsend" # default puny 2M RAM buffer; can be set to multi-gig RAM or disk
# Alternatively: RunBeforeJob = "mbuffer < /path/to/bufreceive > /path/to/bufsend"
- be a lot easier than asking the developers to reinvent the wheel (insert your own ring-buffer pun here...)
- let the developers all but ignore spooling & focus on real backup stuff
- give users access to many more options, such as using RAM or disk spool, high/low-watermarks, and rate-limiting
- let folks cleverer than me figure out other cool things to do with the datastream
USDA - ARS - NWISRL
3793 N 3600 E
Kimberly, ID 83341
This electronic message contains information generated by the USDA solely for the intended recipients. Any unauthorized interception of this message or the use or disclosure of the information it contains may violate the law and subject the violator to civil or criminal penalties. If you believe you have received this message in error, please notify the sender and delete the email immediately.