[Burp-users] Millions upon millions of tiny files

[Myles Loosley-M. <my...@pr...>]

Hi Folks,

We've been using BURP for a couple of years now, backing up mixed use 
linux/windows servers (as whole servers, with some excludes sprinkled in 
for good measure), encrypting the backups with BURP running in a "nobody 
trusts anybody" setup on both the client/server sides, which has been 
one of the core 'selling features'.

We recently replaced our aging 8 drive RAID6 backup array, with a 6 
drive ZFS setup, and spent a considerable amount of time trying to tweak 
for small file performance.  Realistically however, we've been unable to 
make a significant dent in small file/random IO.

The problem I'm running into, is the sheer quantity of small files in a 
typical BURP backup, ~500k files per server minimum, and we retain ~12 
backups per server.  After a few dozen servers (& >100MM files), this is 
more than enough to murder any filesystem/storage scheme I know of.  
Basic operations take hours to complete, rsync'ing the main burp storage 
directory (IE: to the new array) takes days, etc.

What are other people doing to address this in their setups, surely I 
can't be alone in this experience?  Is there something really obvious 
I've missed?  Something tried and true to condense this down?  I realize 
many people might only backup the exact files/directories they need 
(which we used to try and do, but it never worked -- we always missed 
important stuff buried somewhere forgotten, etc.).

We tried running Protocol 2 years ago (which IIRC, addressed this by not 
mirroring the files 1:1), but during restore tests, it was not reliable, 
resulting in failed backups/restores/etc.

Any advice/recommendations/whatnot?

-- 

Myles Loosley-Millman