Menu
▾
▴
Re: [MooseFS-Users] Backing up MooseFS
|
From: Piotr R. K. <pio...@mo...> - 2018-03-15 16:06:46
|
Hi, >>> In my case I have to put about 150 millions files/directories on Moose >>> and I'm afraid rsync-ing will last for eterni we have a script written which allows you to get recently modified files with configurable period of time. It scans changelogs and queries Master Server for these files' paths (so the best is to run it either on Master or on Metalogger). I can share it with you if you would be interested. Best regards, Peter -- Piotr Robert Konopelko | mobile: +48 601 476 440 MooseFS Client Support Team | moosefs.com > On 15 Mar 2018, at 3:27 PM, Alexander AKHOBADZE <ba...@ya...> wrote: > > > And by the way ZFS also provides a deduplication ... I believe it will dramatically help to reduce space requirements for backup storage. > > From the other side turning deduplication on really slows down file operation on ZFS :--( > >> On 15.03.2018 16:38, Wilson, Steven M wrote: >>> Alexander, >>> >>> You're right; it can take a long time with a lot of files. Our largest MooseFS file system, in terms of number of files, has 100 million files. In this case, I use rsync with GNU parallel so that I'm running several rsync jobs simultaneously. Here's the command from my backup script that runs the rsync from parallel: >>> ls $src | parallel --jobs -1 'echo DIR: {}; rsync $args $src/{} $dst' 2>&1 >>> >>> Unless there's been a lot of activity on this file system, I can usually get it backed up in about 10 hours. >>> >>> Steve >>> >>> ________________________________________ >>> From: Alexander AKHOBADZE <ba...@ya...> >>> Sent: Thursday, March 15, 2018 9:23 AM >>> To: Wilson, Steven M >>> Subject: Re: [MooseFS-Users] Backing up MooseFS >>> >>> Hi Steve! Thank you for response! >>> >>> This solution attracts me also. >>> >>> And how many files/directories are you rsync-ing ? >>> And how mach time does in take? >>> >>> In my case I have to put about 150 millions files/directories on Moose >>> and I'm afraid rsync-ing will last for eternity... >>> >>> On 15.03.2018 16:09, Wilson, Steven M wrote: >>>> Hi Alexander, >>>> >>>> We copy data from our MooseFS file systems using rsync to backup servers that have ZFS file systems. ZFS provides snapshots and compression. Before each daily backup of a given MooseFS file system, we make a snapshot of the backup (and naming it with the date of the snapshot) and then start the backup. For our purposes, 45 days of backups are sufficient so we only keep the last 45 snapshots, deleting any older snapshots to save space. This has worked out quite well for us. >>>> >>>> Regards, >>>> Steve >>>> >>>> ________________________________________ >>>> From: Alexander AKHOBADZE <ba...@ya...> >>>> Sent: Wednesday, March 14, 2018 3:13 AM >>>> To: MooseFS-Users >>>> Subject: [MooseFS-Users] Backing up MooseFS >>>> >>>> Hi dear All! >>>> >>>> Could you share your experience in how do you do backups of content >>>> stored on MooseFS? >>>> >>>> What software do you use? >>>> >>>> What advantages and disadvantages do you have in your case? >>>> >>>> Thanks a lot! >>>> >>>> wbr >>>> Alexander >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Check out the vibrant tech community on one of the world's most >>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>>> _________________________________________ >>>> moosefs-users mailing list >>>> moo...@li... >>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users Best regards, Peter -- Piotr Robert Konopelko | mobile: +48 601 476 440 MooseFS Client Support Team | moosefs.com GitHub | Twitter | Facebook | LinkedIn |
