Re: [MooseFS-Users] Long and severe performance impact of deleting files

[Wilson, S. M <st...@pu...>]

Yes, that's a lot of files!  I've been working with the users to try to figure out ways to reduce the number of files but that's a slow process.


I have 256GB of memory in the master server.  mfsmaster has 121GB allocated from when it reached the peak but "only" 81GB in use right now.  It takes 2.5m to write out the metadata from memory but I only do that every 6 hours.  I even have /var/lib/mfs on an NVMe SSD to try to make that as fast as possible.  The master server has a Xeon E5-1630v3 running at 3.7GHz.


If anyone else has some good ideas on how to improve performance in general for file systems with 100's of millions of files, I'd like to hear them!


Steve


________________________________
From: Alexander AKHOBADZE <ba...@ya...>
Sent: Wednesday, April 17, 2019 2:23 AM
To: moo...@li...
Subject: Re: [MooseFS-Users] Long and severe performance impact of deleting files



Hi!

WOW!!!  300-400 millions!!! This is a huge workload IMHO.

Let me ask how much RAM is installed in your master server and how much of it your Master-process uses to serve so many files?

How much time Master spends to hourly save the whole metadata?


On 16.04.2019 23:04, Wilson, Steven M wrote:

Hi,


One of our MooseFS file systems has four chunkservers and just over 300 million files.  A few weeks ago, the number of files had increased to almost 400 million and then a user deleted close to 100 million files at one time.  That dramatically impacted performance on the file system and it took about four weeks for the file system to return to its normal level of performance.  Users were reporting that their I/O intensive jobs were taking about 3 times longer to complete.  And they were also complaining that their active desktop sessions were very sluggish and almost unusable at times.


Our chunkservers are running 3.0.103 except for one which is still at 3.0.97 (soon to be upgraded).  The underlying file system is XFS for most, but not all, of the disks in each chunkserver (we have a few ZFS and a few ext4).  We have a goal of 2 for every file in the file system.  The chunk servers are about 95% full.


The chunk deletions per minute graph shows it starting on March 20 with about 18k deletions per minute.  By March 25 it is steady at 12K deletions per minute.  Then around March 31 it drops to 8K.  By April 4 we are at 5K and by April 11 it dropped to 2.5K.  And finally by yesterday, May 15, we are averaging 1.5K deletions per minute and our performance has returned to almost normal.


During this time the disk utilization (as seen from iostat) on the disks in the chunkservers were between 60% and 100%.  Now we're down to a more reasonable 50% utilization or less.


Is this an inherent issue with MooseFS or are there ways to lessen the severe performance impact of deleting large numbers of files?  Has anyone else expierienced this behavior?  I assume there must be something going on in the background for coalescing free space, etc. but it sure seems to come with a steep penalty.


Thanks,

Steve




_________________________________________
moosefs-users mailing list
moo...@li...<mailto:moo...@li...>
https://lists.sourceforge.net/lists/listinfo/moosefs-users