Re: [Moosefs-users] Out of disk space on master / recovery failed

[Steve <st...@bo...>]

This happened to me too, on a mfsmaster running on a small silicon drive.
Lost the lot. A powercut causing a mfschunkserver failure creating extra
logging. 

Looks like a repeatable problem that needs addressing. 

 

 

 

 

 

 

-------Original Message------- 

 

From: Allen Landsidel 

Date: 04/11/2011 14:37:09 

To: moo...@li... 

Subject: [Moosefs-users] Out of disk space on master / recovery failed 

 

So I didn't plan ahead well and ended up with /var filling up on my 

master over night, causing the master to crash. 

 

mfsmetarestore refused to recover the system, I think because it didn't 

get a chance to write out the metadata file. It seems there's something 

wrong with the way it's doing writes. After the crash both the 

metadata.mfs and metadata.mfs.back were 0 bytes, and mfsmetarestore 

(obviously) refused to read from them. 

 

Some but not all of the changelog files were 0 bytes as well. Same 

story on the backup (metalogger) server. 

 

Just a heads up, I think a little more checking would be in order here 

to make sure there is space available for the metadata, and at least to 

prevent the master from crashing when/if it can't write the metadata. 

If it had stayed up with all the metadata in memory I could've seen the 

disk issue and brought up another metalogger with more disk space to 

catch up and take over. 

 

 

 

-----------------------------------------------------------------------------
 

RSA(R) Conference 2012 

Save $700 by Nov 18 

Register now 

http://p.sf.net/sfu/rsa-sfdev2dev1 

_______________________________________________ 

moosefs-users mailing list 

moo...@li... 

https://lists.sourceforge.net/lists/listinfo/moosefs-users