[MooseFS-Users] Frequent disconnections to master since upgrade to 2.0.103

[Michael T. <mic...@ho...>]

Hi,

Since I upgraded my mfs set (master, metalogger, chunkservers) to the the latest (3.0.103), I've noticed an increased frequency of the following entries in the logs:

Dec 21 10:00:21 HO-MFSMaster01 mfsmaster[848]: chunkserver disconnected - ip: 10.77.77.103 / port: 9422, usedspace: 16885627727872 (15725.97 GiB), totalspace: 19796317044736 (18436.76 GiB)
Dec 21 10:00:21 HO-MFSMaster01 mfsmaster[848]: chunkserver disconnected - ip: 10.77.77.102 / port: 9422, usedspace: 16800836857856 (15647.00 GiB), totalspace: 18827063713792 (17534.07 GiB)
Dec 21 10:00:21 HO-MFSMaster01 mfsmaster[848]: chunkserver disconnected - ip: 10.77.77.101 / port: 9422, usedspace: 16997673164800 (15830.32 GiB), totalspace: 19796319059968 (18436.76 GiB)
Dec 21 10:00:21 HO-MFSMaster01 mfsmaster[848]: connection with client(ip:10.77.77.112) has been closed by peer
Dec 21 10:00:21 HO-MFSMaster01 mfsmaster[848]: child finished

And every time this happens, the following happens on my busiest mfsmount client:

Dec 21 10:00:21 KVM02 mfsmount[2822]: master: connection lost (data)
Dec 21 10:00:21 KVM02 mfsmount[2840]: master: connection lost (data)
Dec 21 10:00:21 KVM02 mfsmount[2822]: registered to master
Dec 21 10:00:21 KVM02 mfsmount[2840]: registered to master
Dec 21 10:00:22 KVM02 mfsmount[2840]: file: 4315203, index: 20 - fs_writechunk returned status: No space left
Dec 21 10:00:22 KVM02 mfsmount[2840]: error writing file number 4315203: ENOSPC (No space left on device)
Dec 21 10:00:22 KVM02 mfsmount[2840]: file: 22817113, index: 28 - fs_writechunk returned status: No space left
Dec 21 10:00:22 KVM02 mfsmount[2840]: error writing file number 22817113: ENOSPC (No space left on device)
Dec 21 10:00:22 KVM02 mfsmount[2840]: error writing file number 4315203: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 407 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 460 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 476 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 47 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 22895109, index: 48 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 22895109, index: 269 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 48 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 341 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 22895109: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 22895109: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 9939425, index: 34 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 9939425: ENOSPC (No space left on device)

This particular client is a KVM server with about 10 VM guests whose disk images are stored on a 'mfsmount'ed folder.  The VM guests are then "paused" by the host and are not able to be manually unpaused/resumed (whether via GUI admin or cli command).

Other clients, who are still in a previous version on moosefs-client, do not exhibit this.  I was thinking on downgrading the moosefs-client package on the affected host, but they are not available for download anymore in ppa.moosefs.com.  Is there any place where I can download it?

Best Regards,


--- mike t.