Re: [Burp-users] Millions upon millions of tiny files

[David H. R. C. <da...@ze...>]

On Tue, Jul 06, 2021 at 01:06:07PM -0400, Myles Loosley-Millman wrote:
> Hi Folks,
> 
...
> 
> What are other people doing to address this in their setups, surely I can't
> be alone in this experience?  Is there something really obvious I've
> missed?  Something tried and true to condense this down?  I realize many
> people might only backup the exact files/directories they need (which we
> used to try and do, but it never worked -- we always missed important stuff
> buried somewhere forgotten, etc.).

We've been using burp at my workplace for quite a few years now, with
mixed windows and linux machines. We mostly just backup everything on
linux boxen, and all of C:/User on windows machines.

We've got about 2.5TB of backups in burp. I don't know exactly how
many files that represents, but we've used 48514209 inodes. This
serves 45 different computers, max_children set to 5. We follow a
7 days/4 weeks /6 months and 2 years worth of those. We do *not*
encrypt any of this in burp, and we are not using block or filesystem
level encryption.

We rsync the whole /var/spool/burp periodicaly to backup disk. That
*can* take quite a while. I occasionally do restores, but have not
found them to be exceedingly slow.

We're using ext4 with no special tuning on a 4 disk (spinning rust)
raid10 mdraid setup on an HP SmartArray P420 adapter. Note that we
aren't using the controller raid function, it's just doing JBOD. This
is for management and portability reasons. We've got a 4-core older
xeon, and 12GB of memory. We upgraded to this "new" a year or two ago.

I've not seen the sort of issues you describe, with the exception of
slow rsyncs. I expect that *some* of that difference might be your use
of encryption. I've not used zfs at all, but could you use zfs send
rather than rsync, to backup to an external device? I considered using
btrfs for this reason, but ultimately decided to play it safe, with
ext4.

-davidc