[Bacula-users] backup slowdown (mysqld) after tape autochange

[Robert W. <Rob...@df...>]

Hi,

strange problem.  Here's some hardware where Bacula has been running 
successfully for ca. 5 years.  It was release 1.38.11 under Solaris 10x86.

Last month, we had a system disk crash on the backup system.  No backup 
datas have been lost.  We just had to reinstall the backup system.
Since this was our only Solaris x86 system, we decided to migrate 
to Linux and to a newer Bacula release.  Until the repaired hardware
was present, we started with a virtualized new system, just for the
daily incremental backups to disk volumes.

Since most of our actual systems are Ubuntu Hardy server LTS, we 
choosed Bacula 2.2.8 of this distribution as our new version (well,
it's old, but 1.38.11 was running well, and 2.2.8 was the default)

We upgraded Bacula's mysql database with the corresponding script
from 1.38.11 to 2.2.8.  We imported the updated DB using mysql_dump
into the new system which has MySQL 5.1.41 and Linux Kernel 2.6.32
The virtualized system worked well all the time.

Now, the hardware version of the system is ready, and a yearly full 
backup, which goes directly to tape, is imminent.  

And now, the strange things are coming...


/* The system is a 2x2 core AMD Opteron system, 4 GB RAM, 6xLSI SCSI U320
Megaraid with seperated channels for external disks, tape readers and
autochanger.  23 TB disk storage on external RAIDs, autochanger and 
HP-readers for LTO-3 tapes.   System: see above. */


NOW BACKING UP...

Starting a bunch of full backup jobs which fit into 1 SINGLE TAPE
produces NO PROBLEMS:  the jobs start, run and write, and terminate 
within a usual span of time.  In so doing, I can backup a dozen
systems with totally 360 GB on one tape in a few hours. 


FACING THE PROBLEM...

Starting a bunch of full backup jobs that DO NOT FIT into 1 single 
tape proceeds like follows  (with a fresh tape forced by setting the
former one to readonly):

- first, the jobs run well and write their data to the first fresh tape 
  of the corresponding pool.  Speed is similar as known from the old OS.

- when the tape is full with around 600GB of data, it is marked as 
  Full, being unloaded, and the next free tape of the pool is loaded.

- from this moment on, writing to the new fresh tape becomes incredibly 
  slow (4 GB/hour) and mysqld has constantly 95%-100% CPU load. 
  No other process has an important load, and the mysql load isn't 
  represented in the system's load values:
 
Cpu(s):  3.3%us,  2.2%sy,  0.0%ni, 91.6%id,  2.1%wa,  0.1%hi,  0.7%si,  0.0%st
Mem:   3961616k total,  3850072k used,   111544k free,    17532k buffers
Swap:  3906552k total,        0k used,  3906552k free,  3579956k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1356 mysql     20   0  144m  31m 2376 S   98  0.8 163:57.79 mysqld
    1 root      20   0  2620  948  528 S    0  0.0   0:00.63 init
    2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd
 ....

The only further effect I can see is that the table "bacula.JobMedia" is
growing.   No errors in system log, no mysql errors, nor in Baculas log.

What I mainly don't understand is why this happens after a tape change.
The MaxSpoolSize is 32GB, and I'm backing up 7 systems.  Each of them
had several spool steps during the first tape.   

>From the view of Bacula and its program logic, what has changed when 
the tape has been changed?  I guess it's all the same:  spooling data,
writing them to tape and update the catalog, regardless of first, second
or later tape...?!?


Regards,

	Robert


----------------------------------------------------------------
+++    German Research Center for Artificial Intelligence    +++
----------------------------------------------------------------
Dipl.-Inform. Robert V. Wirth, Campus D3_2, D-66123 Saarbruecken
@office: +49-681-85775-5078 / -5572 +++ @fax: +49-681-85775-5020
mailto:Rob...@df... ++++++++++ http://www.dfki.de/~wirth
----------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
- Firmensitz Trippstadter Strasse 122, D-67663 Kaiserslautern

Geschaeftsfuehrung (executive board):
- Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
- Dr. Walter Olthoff

Vorsitzender des Aufsichtsrats (supervisory board chairman):
- Prof. Dr. h.c. Hans A. Aukes

Amtsgericht Kaiserslautern, HRB 2313
----------------------------------------------------------------