Re: [MooseFS-Users] Frequent disconnections to master since upgrade to 2.0.103

[Piotr R. K. <pio...@ge...>]

Hi Mike,

> Other clients, who are still in a previous version on moosefs-client, do not exhibit this.  I was thinking on downgrading the moosefs-client package on the affected host, but they are not available for download anymore in ppa.moosefs.com <http://ppa.moosefs.com/>.  Is there any place where I can download it?

They are available – the instruction is here: https://moosefs.com/download/#older <https://moosefs.com/download/#older>. In two words just replace "moosefs-3" string in your repo URL with desired version number, e.g. "3.0.101"

Best regards,
Peter

Piotr Robert Konopelko | m: +48 601 476 440 | e: pio...@mo... <mailto:pio...@mo...>
Business & Technical Support Manager
MooseFS Client Support Team

WWW <http://moosefs.com/> | GitHub <https://github.com/moosefs/moosefs> | Twitter <https://twitter.com/moosefs> | Facebook <https://www.facebook.com/moosefs> | LinkedIn <https://www.linkedin.com/company/moosefs>

> On 22 Dec 2018, at 2:19 AM, Michael Tinsay <mic...@ho...> wrote:
> 
> Hi,
> 
> Since I upgraded my mfs set (master, metalogger, chunkservers) to the the latest (3.0.103), I've noticed an increased frequency of the following entries in the logs:
> 
> Dec 21 10:00:21 HO-MFSMaster01 mfsmaster[848]: chunkserver disconnected - ip: 10.77.77.103 / port: 9422, usedspace: 16885627727872 (15725.97 GiB), totalspace: 19796317044736 (18436.76 GiB)
> Dec 21 10:00:21 HO-MFSMaster01 mfsmaster[848]: chunkserver disconnected - ip: 10.77.77.102 / port: 9422, usedspace: 16800836857856 (15647.00 GiB), totalspace: 18827063713792 (17534.07 GiB)
> Dec 21 10:00:21 HO-MFSMaster01 mfsmaster[848]: chunkserver disconnected - ip: 10.77.77.101 / port: 9422, usedspace: 16997673164800 (15830.32 GiB), totalspace: 19796319059968 (18436.76 GiB)
> Dec 21 10:00:21 HO-MFSMaster01 mfsmaster[848]: connection with client(ip:10.77.77.112) has been closed by peer
> Dec 21 10:00:21 HO-MFSMaster01 mfsmaster[848]: child finished
> 
> And every time this happens, the following happens on my busiest mfsmount client:
> 
> Dec 21 10:00:21 KVM02 mfsmount[2822]: master: connection lost (data)
> Dec 21 10:00:21 KVM02 mfsmount[2840]: master: connection lost (data)
> Dec 21 10:00:21 KVM02 mfsmount[2822]: registered to master
> Dec 21 10:00:21 KVM02 mfsmount[2840]: registered to master
> Dec 21 10:00:22 KVM02 mfsmount[2840]: file: 4315203, index: 20 - fs_writechunk returned status: No space left
> Dec 21 10:00:22 KVM02 mfsmount[2840]: error writing file number 4315203: ENOSPC (No space left on device)
> Dec 21 10:00:22 KVM02 mfsmount[2840]: file: 22817113, index: 28 - fs_writechunk returned status: No space left
> Dec 21 10:00:22 KVM02 mfsmount[2840]: error writing file number 22817113: ENOSPC (No space left on device)
> Dec 21 10:00:22 KVM02 mfsmount[2840]: error writing file number 4315203: ENOSPC (No space left on device)
> Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 407 - fs_writechunk returned status: No space left
> Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
> Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 460 - fs_writechunk returned status: No space left
> Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 476 - fs_writechunk returned status: No space left
> Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
> Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 47 - fs_writechunk returned status: No space left
> Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
> Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 22895109, index: 48 - fs_writechunk returned status: No space left
> Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 22895109, index: 269 - fs_writechunk returned status: No space left
> Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 48 - fs_writechunk returned status: No space left
> Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
> Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 23977380, index: 341 - fs_writechunk returned status: No space left
> Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 22895109: ENOSPC (No space left on device)
> Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
> Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 22895109: ENOSPC (No space left on device)
> Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 23977380: ENOSPC (No space left on device)
> Dec 21 10:00:27 KVM02 mfsmount[2840]: file: 9939425, index: 34 - fs_writechunk returned status: No space left
> Dec 21 10:00:27 KVM02 mfsmount[2840]: error writing file number 9939425: ENOSPC (No space left on device)
> 
> This particular client is a KVM server with about 10 VM guests whose disk images are stored on a 'mfsmount'ed folder.  The VM guests are then "paused" by the host and are not able to be manually unpaused/resumed (whether via GUI admin or cli command).
> 
> Other clients, who are still in a previous version on moosefs-client, do not exhibit this.  I was thinking on downgrading the moosefs-client package on the affected host, but they are not available for download anymore in ppa.moosefs.com <http://ppa.moosefs.com/>.  Is there any place where I can download it?
> 
> Best Regards,
> 
> 
> --- mike t.
> 
> _________________________________________
> moosefs-users mailing list
> moo...@li... <mailto:moo...@li...>
> https://lists.sourceforge.net/lists/listinfo/moosefs-users <https://lists.sourceforge.net/lists/listinfo/moosefs-users>