Re: [MooseFS-Users] Frequent disconnections to master since upgrade to 2.0.103

[Michael T. <mic...@ho...>]

Thanks Peter.  I will try that.

Happy Holidays!!!


--- mike t.

________________________________
From: Piotr Robert Konopelko <pio...@ge...>
Sent: Saturday, 22 December 2018 11:42 AM
To: Michael Tinsay
Cc: MooseFS-Users
Subject: Re: [MooseFS-Users] Frequent disconnections to master since upgrade to 2.0.103

Hi Mike,

Other clients, who are still in a previous version on moosefs-client, do not exhibit this.  I was thinking on downgrading the moosefs-client package on the affected host, but they are not available for download anymore in ppa.moosefs.com<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fppa.moosefs.com%2F&data=02%7C01%7C%7Caee1e454defe4e75bea308d667bf7925%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636810469400123108&sdata=dF9sUR1FCozrVfldCF3RI3NGgdtnteMA6Ivjj%2BI%2B%2Bbw%3D&reserved=0>.  Is there any place where I can download it?

They are available – the instruction is here: https://moosefs.com/download/#older<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmoosefs.com%2Fdownload%2F%23older&data=02%7C01%7C%7Caee1e454defe4e75bea308d667bf7925%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636810469400123108&sdata=irgLVNdaHUk4hgnhLG9MUsClvxQA6E5fqNK7uJqpTUM%3D&reserved=0>. In two words just replace "moosefs-3" string in your repo URL with desired version number, e.g. "3.0.101"

Best regards,
Peter

Piotr Robert Konopelko | m: +48 601 476 440 | e: pio...@mo...<mailto:pio...@mo...>
Business & Technical Support Manager
MooseFS Client Support Team

WWW<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmoosefs.com%2F&data=02%7C01%7C%7Caee1e454defe4e75bea308d667bf7925%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636810469400123108&sdata=HTFucqtvfy0qArOf5aKeA%2FvTsUUu8lJGoi%2B7faUnD2o%3D&reserved=0> | GitHub<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmoosefs%2Fmoosefs&data=02%7C01%7C%7Caee1e454defe4e75bea308d667bf7925%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636810469400123108&sdata=Hi6hKrtVcDZ08hO2cdAZjRaKpecjLtrqGL%2By72NKrew%3D&reserved=0> | Twitter<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fmoosefs&data=02%7C01%7C%7Caee1e454defe4e75bea308d667bf7925%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636810469400123108&sdata=pZd16HwCW8bxYrx%2FvxxTRIlLZEUKRYC9Qj425RbOhqc%3D&reserved=0> | Facebook<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2Fmoosefs&data=02%7C01%7C%7Caee1e454defe4e75bea308d667bf7925%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636810469400279362&sdata=ihwp227KF0LOe51aPOEsh%2FnbX5iR9y41J5LNT%2F6j5fk%3D&reserved=0> | LinkedIn<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fmoosefs&data=02%7C01%7C%7Caee1e454defe4e75bea308d667bf7925%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636810469400279362&sdata=35o3by2Zz1xAOYizr9voG%2FuhgxRSa7MwtMH404W9XOY%3D&reserved=0>

On 22 Dec 2018, at 2:19 AM, Michael Tinsay <mic...@ho...<mailto:mic...@ho...>> wrote:

Hi,

Since I upgraded my mfs set (master, metalogger, chunkservers) to the the latest (3.0.103), I've noticed an increased frequency of the following entries in the logs:

Dec 21 10:00:21 HO-MFSMaster01 mfsmaster[848]: chunkserver disconnected - ip: 10.77.77.103 / port: 9422, usedspace: 16885627727872 (15725.97 GiB), totalspace: 19796317044736 (18436.76 GiB)
Dec 21 10:00:21 HO-MFSMaster01 mfsmaster[848]: chunkserver disconnected - ip: 10.77.77.102 / port: 9422, usedspace: 16800836857856 (15647.00 GiB), totalspace: 18827063713792 (17534.07 GiB)
Dec 21 10:00:21 HO-MFSMaster01 mfsmaster[848]: chunkserver disconnected - ip: 10.77.77.101 / port: 9422, usedspace: 16997673164800 (15830.32 GiB), totalspace: 19796319059968 (18436.76 GiB)
Dec 21 10:00:21 HO-MFSMaster01 mfsmaster[848]: connection with client(ip:10.77.77.112) has been closed by peer
Dec 21 10:00:21 HO-MFSMaster01 mfsmaster[848]: child finished

And every time this happens, the following happens on my busiest mfsmount client:

Dec 21 10:00:21 KVM02 mfsmount[2822]: master: connection lost (data)
Dec 21 10:00:21 KVM02 mfsmount[2840]: master: connection lost (data)
Dec 21 10:00:21 KVM02 mfsmount[2822]: registered to master
Dec 21 10:00:21 KVM02 mfsmount[2840]: registered to master
Dec 21 10:00:22 KVM02 mfsmount[2840]: file: 4315203, index: 20 - fs_writechunk returned status: No space left
Dec 21 10:00:22 KVM02 mfsmount[2840]: error writing file number 4315203: ENOSPC (No space left on device)
Dec 21 10:00:22 KVM02 mfsmount[2840]: file: 22817113, index: 28 - fs_writechunk returned status: No space left
Dec 21 10:00:22 KVM02 mfsmount[2840]: error writing file number 22817113: ENOSPC (No space left on device)
Dec 21 10:00:22 KVM02 mfsmount[2840]: error writing file number 4315203: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 407 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 460 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 476 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 47 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 22895109, index: 48 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 22895109, index: 269 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 48 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 341 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 22895109: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 22895109: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 9939425, index: 34 - fs_writechunk returned status: No space left
Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 9939425: ENOSPC (No space left on device)

This particular client is a KVM server with about 10 VM guests whose disk images are stored on a 'mfsmount'ed folder.  The VM guests are then "paused" by the host and are not able to be manually unpaused/resumed (whether via GUI admin or cli command).

Other clients, who are still in a previous version on moosefs-client, do not exhibit this.  I was thinking on downgrading the moosefs-client package on the affected host, but they are not available for download anymore in ppa.moosefs.com<https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fppa.moosefs.com%2F&data=02%7C01%7C%7Caee1e454defe4e75bea308d667bf7925%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636810469400279362&sdata=quop3tPL3TcykdeDpZ8Gr4N9ytJILnap4HVPp%2FzYXd8%3D&reserved=0>.  Is there any place where I can download it?

Best Regards,


--- mike t.

_________________________________________
moosefs-users mailing list
moo...@li...<mailto:moo...@li...>
https://lists.sourceforge.net/lists/listinfo/moosefs-users<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fmoosefs-users&data=02%7C01%7C%7Caee1e454defe4e75bea308d667bf7925%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636810469400279362&sdata=piwDsStoP%2F1IoFflxcw%2BDufIhBaokoJ861y8DqB%2F904%3D&reserved=0>