Re: [MooseFS-Users] Long and severe performance impact of deleting files

[Wilson, S. M <st...@pu...>]

________________________________
From: Ricardo J. Barberis <ric...@do...>
Sent: Tuesday, April 16, 2019 7:10 PM
To: moo...@li...
Subject: Re: [MooseFS-Users] Long and severe performance impact of deleting files

El Martes 16/04/2019 a las 17:04, Wilson, Steven M escribió:
> Hi,
>
>
> One of our MooseFS file systems has four chunkservers and just over 300
> million files.  A few weeks ago, the number of files had increased to
> almost 400 million and then a user deleted close to 100 million files at
> one time.  That dramatically impacted performance on the file system and it
> took about four weeks for the file system to return to its normal level of
> performance.  Users were reporting that their I/O intensive jobs were
> taking about 3 times longer to complete.  And they were also complaining
> that their active desktop sessions were very sluggish and almost unusable
> at times.
>
>
> Our chunkservers are running 3.0.103 except for one which is still at
> 3.0.97 (soon to be upgraded).  The underlying file system is XFS for most,
> but not all, of the disks in each chunkserver (we have a few ZFS and a few
> ext4).  We have a goal of 2 for every file in the file system.  The chunk
> servers are about 95% full.
>
>
> The chunk deletions per minute graph shows it starting on March 20 with
> about 18k deletions per minute.  By March 25 it is steady at 12K deletions
> per minute.  Then around March 31 it drops to 8K.  By April 4 we are at 5K
> and by April 11 it dropped to 2.5K.  And finally by yesterday, May 15, we
> are averaging 1.5K deletions per minute and our performance has returned to
> almost normal.
>
>
> During this time the disk utilization (as seen from iostat) on the disks in
> the chunkservers were between 60% and 100%.  Now we're down to a more
> reasonable 50% utilization or less.
>
>
> Is this an inherent issue with MooseFS or are there ways to lessen the
> severe performance impact of deleting large numbers of files?  Has anyone
> else expierienced this behavior?  I assume there must be something going on
> in the background for coalescing free space, etc. but it sure seems to come
> with a steep penalty.
>
>
> Thanks,
>
> Steve

My case is not as severe, but in general we are affected by deletions and
replications.

We mitigate this with a script that, during the day sets on the masters these
variables:

CHUNKS_SOFT_DEL_LIMIT = 1
CHUNKS_HARD_DEL_LIMIT = 1
CHUNKS_WRITE_REP_LIMIT = 1
CHUNKS_READ_REP_LIMIT = 1


And at night it sets them so:

CHUNKS_SOFT_DEL_LIMIT = 10
CHUNKS_HARD_DEL_LIMIT = 25
CHUNKS_WRITE_REP_LIMIT = 5
CHUNKS_READ_REP_LIMIT = 10


That helps us a lot, especially drugin peak hours.

Ocasionally we tweak those values by hand, e.g. in case of deleting a lot of
files, we might increase CHUNKS_SOFT_DEL_LIMIT and CHUNKS_HARD_DEL_LIMIT and
lower the other two.

After tweaking those parameters you just need to reload mfsmater.


HTH,
--
Ricardo J. Barberis
Senior SysAdmin / IT Architect
DonWeb
La Actitud Es Todo
www.DonWeb.com<http://www.DonWeb.com>
Home WebHosting Estados Unidos Ilimitado - Registro de Dominios - Revendedores - VPS<http://www.donweb.com/>
www.donweb.com
DonWeb Web Hosting y Registro de Dominios. Alojamiento para sitios desde u$s 2,27, Registro de dominios, Hosting para revendedores, Servidores Dedicados, Planes VPS (Virtual Private Servers). Somos proveedor lider en LATAM (Fuente Netcraft)

________________________________


Hi, Ricardo,

Thanks for sharing that.  After reading your reply, I realized that I had seen that mentioned before (probably by you!) on this list.  I'll probably do something similar here.

Steve