Re: [Bacula-devel] slow copying from tape to tape

[Wolfram Schlich <lists@wo...>]

* Ulrich Leodolter <ulrich.leodolter@...> [2010-05-14 13:09]:
> Hi,

Hi Ulrich!

> On Fri, 2010-05-14 at 10:27 +0200, Wolfram Schlich wrote:
> > Hi!
> > 
> > I'm using Bacula 5.0.2 to back up to a tape library
> > which has two LTO-4 FC tape drives. Backup speed is
> > around 95-100 MB/s which is quite ok (using a DAS
> > for spooling data though).
> > 
> > Now, when running a copy job that copies backups
> > from a tape from the full or incr backup pool to
> > a tape from the offsite pool, speed reaches 50MB/s
> > maximum, so it's just half as fast as the backups.
> > 
> > What could be the reason for that?
> > 
> 
> copy is NOT done in multi-threaded buffered way.
> it is done one by one block (default is 63k)
> 
> ...
> read block N
> write block N
> read block N+1
> write block N+1
> ...
> 
> so you get about the half speed.

What the ... ?! :(

> this is why we do a second copy-disk-to-tape
> using a special sql query to make offsite copies.

Well, the problem for us is that we don't backup
to disk because the storage isn't big enough for
that, so we backup to tape using the storage as
spool device (which is able to spool data for 4
LTO4 tapes).

Guess we're lost then.
Happy happy, joy joy *sigh*

Thanks for the hint!

Regards,
Wolfram

Re: [Bacula-devel] Bacula Wipe Tape

[Paul Davis <pdd@lg...>]

You are probably correct, given that current PRML encoding techniques 
and the given density of magnetic media (both drive and tape). The NIST 
specification even suggests that a single pass of random data would 
likely be sufficient (at least for drives, nothing is said of tapes). 
However it has been indicated to me that following the DoD/NIST specs is 
the minimum procedure, and they are the experts (not I ... although I 
like your test approach).

Given the apparent vagaries of tape data placement, job migration has 
been the only acceptable method (hence the tool was not necessary) 
albeit a rather lengthy procedure. Unfortunately given human nature, one 
I'm afraid will be repeated in the future (at least as long as we 
continue to use tape as and archive/backup method).
> It's surprising that such a "necessary" tool has never been written!
> Perhaps it's not as necessary as you think.
>
> Also, you're conflating drives and tapes, so I'll do the same.
>
> My experience with drive recovery companies suggests that it takes
> about 2kUSD to recover a drive.  And it's not always successful and it
> doesn't always recover all the data.  So the data better be worth at
> least that much.
>
> If your attacker is willing to spend $2k per tape to read your data,
> then maybe you can afford to destroy the $50 tapes and buy new ones.
> Or encrypt the data.
>
> IMHO, you should attempt to overwrite a tape with zeros then send it to
> the recovery company to recover.  This test will cost you ~$2k which is
> probably less than the development cost of this utility.  I'm willing
> to bet they won't be able to recover anything.
>
> Regards,
>    


Thanks for your thinking about this,
Paul Davis

[Bacula-devel] 5.0.2 issues: --enable-client-only doesn't? Are static Linux builds still possible?

[Phil Stracchino <alaric@me...>]

I honestly haven't noticed whether earlier 5.x releases have any of
these problems.

When building Bacula with --enable-client-only, one would assume that
this would mean, well, client ONLY.  But it doesn't.

Using the configuration arguments generated by the Gentoo package with
'USE bacula-clientonly' as an example, though I had similar results last
night trying to manually build a static FD alone from clean source:


./configure --prefix=/usr --build=i686-pc-linux-gnu \
 --host=i686-pc-linux-gnu --mandir=/usr/share/man \
 --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc \
 --localstatedir=/var/lib --libdir=/usr/lib \
 --docdir=/usr/share/doc/bacula-5.0.2 --with-pid-dir=/var/run \
 --sysconfdir=/etc/bacula --with-subsys-dir=/var/lock/subsys \
 --with-working-dir=/var/lib/bacula \
 --with-scriptdir=/usr/libexec/bacula --with-dir-user=bacula \
 --with-dir-group=bacula --with-sd-user=root --with-sd-group=bacula \
 --with-fd-user=root --with-fd-group=bacula --enable-smartalloc \
 --host=i686-pc-linux-gnu --enable-client-only --enable-static-fd \
 --disable-tray-monitor --without-x --disable-bat --enable-static-cons \
 --without-python --enable-conio --disable-readline --without-readline \
 --with-openssl --enable-ipv6 --with-tcp-wrappers --disable-libtool


And, look, --enable-client-only is in there.  But what actually gets built?

   Large file support:      yes
   Bacula conio support:    yes -ltermcap
   readline support:        no
   TCP Wrappers support:    yes -lwrap
   TLS support:             yes
   Encryption support:      yes
   ZLIB support:            yes
   enable-smartalloc:       yes
   enable-lockmgr:          no
   bat support:             no
   enable-gnome:            no
   enable-bwx-console:      no
   enable-tray-monitor:     no
   client-only:             yes
   build-dird:              yes
   build-stored:            yes
   Plugin support:          no
   ACL support:             yes
   XATTR support:           yes
   Python support:          no
   Batch insert enabled:    no

Um .....   shouldn't "client-only" be mutually exclusive with build-dird
and build-stored?  In fact, shouldn't --enable-client-only automatically
imply --disable-build-dird and --disable-build-stored?

Also, should Bacula attempt to build and install the console if
configured with --enable-client-only?  The Makefile contains:


# --client-only directories
fd_subdirs = src scripts src/lib src/findlib src/filed \
           \
       src/console

# Non-client-only directores
subdirs =    src/cats   src/tools

Shouldn't src/console be in "Non-client-only directores"?  [sic]




Another thing I'm discovering, that is possibly a side effect of current
glibc versions, it that it appears it may no longer be possible to build
a FULLY static fd on Linux.  It appears some internal functions, even on
a statically linked binary, still require /lib/libc.so.* to be present:


../lib/libbac.a(plugins.o): In function `load_plugins(void*, void*, char
const*, char const*, bool (*)(Plugin*))':
plugins.c:(.text+0x620): warning: Using 'dlopen' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
../lib/libbac.a(priv.o): In function `drop(char*, char*, bool)':
priv.c:(.text+0x12e): warning: Using 'initgroups' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
/usr/lib/gcc/i686-pc-linux-gnu/4.4.3/../../../libacl.a(__acl_to_any_text.o):
In function `__acl_to_any_text':
(.text+0x64b): warning: Using 'getgrgid' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
/usr/lib/gcc/i686-pc-linux-gnu/4.4.3/../../../libacl.a(acl_from_text.o):
In function `acl_from_text':
(.text+0x67d): warning: Using 'getgrnam' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
/usr/lib/gcc/i686-pc-linux-gnu/4.4.3/../../../libwrap.a(options.o): In
function `group_option':
(.text+0x8ec): warning: Using 'endgrent' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
/usr/lib/gcc/i686-pc-linux-gnu/4.4.3/../../../libacl.a(acl_from_text.o):
In function `acl_from_text':
(.text+0x4e8): warning: Using 'getpwnam' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
/usr/lib/gcc/i686-pc-linux-gnu/4.4.3/../../../libacl.a(__acl_to_any_text.o):
In function `__acl_to_any_text':
(.text+0x4e8): warning: Using 'getpwuid' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
/usr/lib/gcc/i686-pc-linux-gnu/4.4.3/../../../libwrap.a(options.o): In
function `user_option':
(.text+0x99f): warning: Using 'endpwent' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
/usr/lib/gcc/i686-pc-linux-gnu/4.4.3/../../../libwrap.a(hosts_access.o):
In function `string_match':
(.text+0x7f1): warning: Using 'getaddrinfo' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
../lib/libbac.a(bnet.o): In function `resolv_host(int, char const*,
dlist*)':
bnet.c:(.text+0xee4): warning: Using 'gethostbyname2' in statically
linked applications requires at runtime the shared libraries from the
glibc version used for linking
../lib/libbac.a(address_conf.o): In function `add_address(dlist**,
IPADDR::i_type, unsigned short, int, char const*, char const*, char*, int)':
address_conf.c:(.text+0x9ff): warning: Using 'getservbyname' in
statically linked applications requires at runtime the shared libraries
from the glibc version used for linking
strip static-bacula-fd
==== Make of filed is good ====


This doesn't prevent building the filed, but causes it to SEGV on
starting a job if run with a different glibc than it was built on.  Say,
on a rescue CD.

(Ask me how I know this....  my laptop irretrievably corrupted its
filesystem sometime on Wednesday, and I spent most of yesterday testing
the disk and all yesterday evening restoring it from bare metal.
Because I couldn't get even a static fd to work on the rescue CD, I
ended up having to restore to a different machine and rsync everything
over.)






-- 
  Phil Stracchino, CDK#2     DoD#299792458     ICBM: 43.5607, -71.355
  alaric@...   alaric@...   phil@...
         Renaissance Man, Unix ronin, Perl hacker, Free Stater
                 It's not the years, it's the mileage.

[Bacula-devel] slow copying from tape to tape

[Wolfram Schlich <lists@wo...>]

Hi!

I'm using Bacula 5.0.2 to back up to a tape library
which has two LTO-4 FC tape drives. Backup speed is
around 95-100 MB/s which is quite ok (using a DAS
for spooling data though).

Now, when running a copy job that copies backups
from a tape from the full or incr backup pool to
a tape from the offsite pool, speed reaches 50MB/s
maximum, so it's just half as fast as the backups.

What could be the reason for that?

TIA!
Regards,
Wolfram