Re: [Moosefs-users] Why is it increasing my DEL_LIMIT when I don't want it to!

[Michał B. <mic...@ge...>]

Hi!

We discussed similiar problem with Ólafur in July as far as I remember. Yes,
we know it is not very optimal - if there are many files to be deleted
system tries to increase the limit so that they get deleted, but
unfortunately with realy huge number of files to be deleted, system stucks
up... We have some ideas for improvement, eg. first doing truncate (setting
to 0 bytes) and later do the real deleting. Also setting of this limit would
be possible on the fly with mastertools.



Kind regards
Michał Borychowski 
MooseFS Support Manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Gemius S.A.
ul. Wołoska 7, 02-672 Warszawa
Budynek MARS, klatka D
Tel.: +4822 874-41-00
Fax : +4822 874-41-01


-----Original Message-----
From: WK [mailto:wk...@bn...] 
Sent: Friday, September 23, 2011 3:07 AM
To: moo...@li...
Subject: Re: [Moosefs-users] Why is it increasing my DEL_LIMIT when I don't
want it to!

Well I suppose it would be easy enough to grep the source code for 
'DEL_LIMIT temporary increase' and start commenting some things out for 
a quick fix.

However, I'd prefer if the maintainers addressed the issue with 
something more comprehensive and/or a flag for strict enforcement of the 
DEL_LIMIT setting or the current setup which is obviously some sort of 
'oh no, we have a LOT of files we need to work through and need more 
resources' logic.

There may also be some sort of reason they do this, such as some 
resource issue.

We will see what they have to say. Maybe its already addressed in the 
next version.

-bill


On 9/22/2011 2:42 AM, Ólafur Ósvaldsson wrote:
> We have the exact same problem, chunk deletions have caused problems in
the past and we have DEL_LIMIT set at 5, but mfsmaster increases it to 40-50
right away and sometimes goes to 70/s
>
> This is also the case for chunk replications, it does not seem to honor
the mfsmaster.cfg settings, although that does not get logged.
>
> /Oli
>
> On 22.9.2011, at 05:03, wk...@bn... wrote:
>
>> Ok, we deleted a couple hundred thousand files from a large Maildir
>> folder set.
>>
>> We have had problems with deletions overwhelming the cluster in the
>> past, so we have a DEL_LIMIT set to 20 (which we will probably lower)
>>
>> But when the expiretime hit, the server became lethargic. in checking
>> the logs I see this
>>
>> Sep 21 21:10:31 mfs1master mfsmaster[2373]: DEL_LIMIT temporary
>> increased to: 26/s
>> Sep 21 21:15:30 mfs1master mfsmaster[2373]: DEL_LIMIT temporary
>> increased to: 33/s
>> Sep 21 21:55:24 mfs1master mfsmaster[2373]: DEL_LIMIT decreased back to:
>> 26/s
>>
>> OK, WHY IS DOING THIS! I told it no more than 20
>>
>> I do NOT want this, it kills my server and we've learned the hard way it
>> can really screw up VM images (they go read-only) if the deletions
>> overwhelm the cluster.
>>
>>
>> -bill
>>
>>
>>
>>
----------------------------------------------------------------------------
--
>> All the data continuously generated in your IT infrastructure contains a
>> definitive record of customers, application performance, security
>> threats, fraudulent activity and more. Splunk takes this data and makes
>> sense of it. Business sense. IT sense. Common sense.
>> http://p.sf.net/sfu/splunk-d2dcopy1
>> _______________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
> --
> Ólafur Osvaldsson
> System Administrator
> Nethonnun ehf.
> e-mail: osv...@ne...
> phone: +354 517 3400
>
>
>
----------------------------------------------------------------------------
--
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense.
> http://p.sf.net/sfu/splunk-d2dcopy1
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users


----------------------------------------------------------------------------
--
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2dcopy2
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users