[Bacula-users] Performance with Mac OSX, Encryption and Gzip3

[Hugo L. <hug...@gm...>]

Hi

I am using Bacula inside an Cent OS openVZ container on a
virtualization cluster. I tried with and without compression and/or
encryption and the rates are very various.
I am backing up a KVM client on the same nodes of the cluster and an
external mac client.

I can have rates of more than 30 or 40 MBytes/sec down to some hundred
of KBytes/sec

Here are some stats :

Backing up the mac client :

09-déc 10:48 bacula-dir JobId 42: Bacula bacula-dir 5.0.3 (30Aug10):
09-déc-2010 10:48:42
  Build OS:               i686-redhat-linux-gnu redhat
  JobId:                  42
  Job:                    BackupStationMac.2010-12-08_18.04.32_07
  Backup Level:           Full
  Client:                 "StationMac" 5.0.3 (04Aug10)
i386-apple-darwin10.4.0,osx,10.4.0
  FileSet:                "Test" 2010-12-08 17:53:46
  Pool:                   "Test" (From Job resource)
  Catalog:                "MyCatalog" (From Client resource)
  Storage:                "DataTest" (From Pool resource)
  Scheduled time:         08-déc-2010 18:04:28
  Start time:             08-déc-2010 18:04:34
  End time:               09-déc-2010 10:48:42
  Elapsed time:           16 hours 44 mins 8 secs
  Priority:               10
  FD Files Written:       636,312
  SD Files Written:       636,312
  FD Bytes Written:       170,577,925,504 (170.5 GB)
  SD Bytes Written:       170,856,886,548 (170.8 GB)
  Rate:                   2831.3 KB/s
  Software Compression:   25.7 %
  VSS:                    no
  Encryption:             yes
  Accurate:               yes
  Volume name(s):       =>  About 40 4GB Volumes
  Volume Session Id:      11
  Volume Session Time:    1291809192
  Last Volume Bytes:      4,007,665,932 (4.007 GB)
  Non-fatal FD errors:    0
  SD Errors:              0
  FD termination status:  OK
  SD termination status:  OK
  Termination:            Backup OK


Backing up the KVM :

 I ran many test on the KVM client with various parameter and kind of files:
     1) GZIP3 - Encryption - 1Go file from /dev/urandom
                   => Rate about    7828.8 KB/s
     2) GZIP3 - Encryption - 1Go file from /dev/zero
                   =>   Rate:                   113.1 KB/s
                   =>   Software Compression:   99.5 %
                   =>   Most of the time was to compress the file
     3) NO GZIP3 - Encryption - 1Go file from /dev/urandom
                   => Rate about    19785.7 KB/s
     4) Lots of file of various size --> Typical base install of Cent
OS and some apps about 54 000 files
                   GZIP3 & Encryption => 901.2 KB/s
                   No GZIP3 & No Encryption => 31517.3 KB/s

I also tried some copy job from hard drive to LTO3 device :
  => I am about 7 MBytes/sec I think it's correct

On which aspects can I improve theses perfs ?
I already a list of files that are not to compress :
wildfile = ".rar"
wildfile = ".zip"
wildfile = ".gz"
.......
Or to exclude
wilddir = ".TemporaryItems"
wilddir = ".Spotlight-V100"
.......

In addition, I am looking for something to don't compress tiny files.
Is this possible ?

Also which compilation options can improve my rates that are horrible
on small files. I used the default config of my RPMs and maybe I
omitted something at this moment.

In order to continue bacula implementation into a testing - pre-prod
phase I must reach at least 2 or 3 MB/sec rates for every Jobs. I want
to validate that kind of stats

Can you compare with your stats ?

The Mac machine is a Mac Mini from the last Power PC Generation

The cluster node is a  8 x Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
Virtual machines ( OpenVZ : CentOS Bacula server fd+sd+dir+DB and a
KVM : CentOS Bacula fd ) have a 50/50 repartition of ressources and it
never been fully loaded.

What should I do, in order to have a better compromise between
Compression and the Transfert Rate.

Thanks

Hugo