Re: [Burp-users] Millions upon millions of tiny files

[hnsz2002 <hns...@gm...>]

Hi Myles,

Personally I tried ZFS (on freebsd) years ago, and I don't have a good 
experience... And not only for backup. Maybe my fault or missing 
knowledge, but i dropped after few months.
My other thought is, if you need performance then use RAID10. RAID5 and 
RAID6 are less effective. Millions of files need a lot of tiny IO-s, 
which RAID10 can do better with same disks.
I have a backup server with 6 disk RAID5 and other one with 6 disk 
RAID10 (both are linux softraid with md)... Machines are not comparable. 
RAID10 is faster with less disk and cpu load.

Protocol 2 is still not completely ready as I know, so use protocol 1.

Regards,
hnsz2002

06.07.21 19:06, Myles Loosley-Millman пише:
> Hi Folks,
>
> We've been using BURP for a couple of years now, backing up mixed use 
> linux/windows servers (as whole servers, with some excludes sprinkled 
> in for good measure), encrypting the backups with BURP running in a 
> "nobody trusts anybody" setup on both the client/server sides, which 
> has been one of the core 'selling features'.
>
> We recently replaced our aging 8 drive RAID6 backup array, with a 6 
> drive ZFS setup, and spent a considerable amount of time trying to 
> tweak for small file performance.  Realistically however, we've been 
> unable to make a significant dent in small file/random IO.
>
> The problem I'm running into, is the sheer quantity of small files in 
> a typical BURP backup, ~500k files per server minimum, and we retain 
> ~12 backups per server.  After a few dozen servers (& >100MM files), 
> this is more than enough to murder any filesystem/storage scheme I 
> know of.  Basic operations take hours to complete, rsync'ing the main 
> burp storage directory (IE: to the new array) takes days, etc.
>
> What are other people doing to address this in their setups, surely I 
> can't be alone in this experience?  Is there something really obvious 
> I've missed?  Something tried and true to condense this down?  I 
> realize many people might only backup the exact files/directories they 
> need (which we used to try and do, but it never worked -- we always 
> missed important stuff buried somewhere forgotten, etc.).
>
> We tried running Protocol 2 years ago (which IIRC, addressed this by 
> not mirroring the files 1:1), but during restore tests, it was not 
> reliable, resulting in failed backups/restores/etc.
>
> Any advice/recommendations/whatnot?
>