Re: [MooseFS-Users] growing overgoal

[Markus K. <mar...@tu...>]

On Monday, 7 September 2020 12:19:42 CEST Agata Kruszona-Zawadzka wrote:
> 
> W dniu 04.09.2020 o 15:22, Markus Köberl pisze:
> > Since some time (last few versions of MooseFS) on a few chunkservers the used space grows above the default ACCEPTABLE_PERCENTAGE_DIFFERENCE = 1.0 till I restart the affected chunkserver.
> > On the webinterface i see huge numbers for overgoal (even 4 extra copies).  After the restart of the chunkserver the overgoal goes down but starts growing again after some time.
> 
> We have an issue in MooseFS currently, where on disks with I/O errors in 
> certain circustances some chunks get locked and cannot be deleted until 
> the whole chunk server process is restarted. We introduced a fix for 
> that, it's gonna be available in version 3.0.115. The issue does not 
> affect disks without I/O errors.

Thanks good to hear that a fix might be on the way.

Could it be that instead of "some chunks get locked and cannot be deleted" that there a no deletes at all on this chunk server, or might that be a different problem?

The reason is that on Friday I did restart some of our chunk servers so that everything got in balance again.
Over the Weekend one chunk server crashed. Because the combined disk space in my chunk servers varies between 2TiB and 25TiB, during rebalancing the used disk space varies for some time. But usually always got close to zero over some Time.
I rebooted the crashed chunk server Today. But on a few chunk servers (not related with the number and size of the installed disks) there are no deletes happening at all.
On the "Server Charts" tab the diagrams for "number of chunk deletions per minute" are completely empty for 7 of my 35 chunks servers.
While all the others had heavy activity for about 2 hours.
Now the used disk space varies between avg+1.22 up to avg+12.6% on those 7 hosts while all the others are in balance. There are no deletions in any diagram for the past 45 minutes.


regards
Markus Köberl
-- 
Markus Koeberl
Graz University of Technology
Signal Processing and Speech Communication Laboratory
E-mail: mar...@tu...