[Moosefs-users] MFSMaster crash

[Robert S. <rsa...@ne...>]

I have been seeing crashes of mfsmaster every few days. Generally it happens within a few minutes of the hour. For example 8:02 or 7:02. It has happened on June the 9th, 16th and 26th either around 7 AM or 8 AM.

I am using mfs 1.6.20 on Centos 5.6 64-bit. Currently there are only 2 chunk servers and 2 loggers. The one chunk server is also the master. Both chunk servers also mount the volume locally and serve the content through Apache and DNS based load balancing. The volumes are also mounted by other machines which may re-export it using samba or use it locally.

Any ideas of what I can do to troubleshoot/prevent this?

Some of the logs before the crash:

Jun 26 08:02:37 master mfsmaster[12670]: connection with client(ip:127.0.0.1) has been closed by peer
Jun 26 08:02:39 master mfsmaster[12670]: connection with client(ip:xxx.xxx.x.14) has been closed by peer
Jun 26 08:02:43 master mfsmaster[12670]: connection with client(ip:xxx.xxx.x.139) has been closed by peer
Jun 26 08:02:56 master mfsmaster[12670]: connection with client(ip:xxx.xxx.x.15) has been closed by peer
Jun 26 08:02:56 master mfsmaster[12670]: got chunk status, but don't want it
Jun 26 08:02:56 master mfsmaster[12670]: connection with CS(xxx.xxx.x.55) has been closed by peer
Jun 26 08:02:56 master mfsmaster[12670]: chunkserver disconnected - ip: xxx.xxx.x.55, port: 9422, usedspace: 21515155550208 (20037.55 GiB), totalspace: 25999924264960 (24214.32 GiB)
Jun 26 08:03:02 master mfsmaster[12670]: connection with ML(xxx.xxx.x.139) has been closed by peer
Jun 26 08:03:02 master mfsmaster[12670]: connection with ML(xxx.xxx.x.14) has been closed by peer
Jun 26 08:03:02 master mfsmaster[12670]: connection with client(ip:xxx.xxx.x.139) has been closed by peer
Jun 26 08:03:02 master mfsmaster[12670]: connection with ML(xxx.xxx.x.139) has been closed by peer
Jun 26 08:03:02 master mfsmaster[12670]: connection with client(ip:xxx.xxx.x.15) has been closed by peer
Jun 26 08:03:02 master mfsmaster[12670]: chunkserver register begin (packet version: 5) - ip: xxx.xxx.x.55, port: 9422
Jun 26 08:03:02 master mfsmaster[12670]: connection with ML(xxx.xxx.x.14) has been closed by peer
Jun 26 08:03:02 master mfsmaster[12670]: connection with client(ip:xxx.xxx.x.14) has been closed by peer
Jun 26 08:03:02 master mfsmaster[12670]: connection with client(ip:xxx.xxx.x.139) has been closed by peer
Jun 26 08:03:37 master mfsmaster[12670]: connection with client(ip:xxx.xxx.x.14) has been closed by peer
Jun 26 08:03:37 master mfsmaster[12670]: connection with CS(xxx.xxx.x.55) has been closed by peer
Jun 26 08:03:37 master mfsmaster[12670]: chunkserver disconnected - ip: xxx.xxx.x.55, port: 9422, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB)
Jun 26 08:03:39 master mfsmaster[12670]: connection with client(ip:xxx.xxx.x.15) has been closed by peer
Jun 26 08:03:40 master mfsmaster[12670]: connection with client(ip:xxx.xxx.x.139) has been closed by peer
Jun 26 08:03:40 master mfsmaster[12670]: connection with client(ip:xxx.xxx.x.14) has been closed by peer
Jun 26 08:03:40 master mfsmaster[12670]: connection with client(ip:xxx.xxx.x.15) has been closed by peer

Robert