[Moosefs-users] Out of disk space on master / recovery failed

[Allen L. <lan...@gm...>]

So I didn't plan ahead well and ended up with /var filling up on my 
master over night, causing the master to crash.

mfsmetarestore refused to recover the system, I think because it didn't 
get a chance to write out the metadata file.  It seems there's something 
wrong with the way it's doing writes.  After the crash both the 
metadata.mfs and metadata.mfs.back were 0 bytes, and mfsmetarestore 
(obviously) refused to read from them.

Some but not all of the changelog files were 0 bytes as well.  Same 
story on the backup (metalogger) server.

Just a heads up, I think a little more checking would be in order here 
to make sure there is space available for the metadata, and at least to 
prevent the master from crashing when/if it can't write the metadata.  
If it had stayed up with all the metadata in memory I could've seen the 
disk issue and brought up another metalogger with more disk space to 
catch up and take over.