Re: [Moosefs-users] mfsmaster hang

[Ricardo J. B. <ric...@da...>]

Sorry, I didn't read your email until now.

You might be hitting your mfsmaster too hard, as it tries to recreate 
nonexistent chunks for later deletion.

If that's the case, keep reading but first of all, you should first backup 
your metadata files (usually located in /var/mfs or /var/lib/mfs).

El Martes 28/02/2012, Steve Thompson escribió:
> MFS 1.6.20, CentOS 5.7 64-bit. Four chunk servers.
>
> I have an MFS setup that was working without any issues, until yesterday
> when the mfsmaster hung. I shut down the master and the chunkservers. The
> master starts properly and everything looks good until I start one of the
> chunkservers (any one), at which point the master hangs.

This happened to me once and I also took down every server of the cluster, 9 
chunkservers and one dedicated metalogger (previously, I unmounted all the 
clients, about 250).

Bad idea: when the master came on-line again and I started one chunkserver, 
the master went "crazy" triyng to recreate empty chunks for later deletion.

My "solution" was to start all the chunkservers at the same time, so the 
master saw all the chunks almost simultaneously and didn't try to create 
empty chunks.

After a while (can't remember but it was half an hour to an hour) the master 
and chunkservers calmed down and the cluster was functioning proerly again.

Fortunately for me, it was our MFS for backups, so a few hours off-line or 
data loss wasn't a real problem, but you should probably be carefull and make 
a backup first of your mfsmaster files

> I would appreciate a clue as to where to look, as I have not changed
> anything from the working configuration, and I can't find anything obviously
> wrong. Of course this is an emergency.
>
> Steve

Hope it helps,
-- 
Ricardo J. Barberis
Senior SysAdmin / ITI
Dattatec.com :: Soluciones de Web Hosting
Tu Hosting hecho Simple!

------------------------------------------