Menu
▾
▴
Re: [MooseFS-Users] why master can't server any request at every o'clock?
|
From: 彭智希 <pen...@16...> - 2016-03-02 08:32:54
|
Hi the log of mfschunkserver as below: Feb 29 13:32:37 localhost mfschunkserver[9920]: (read) packet too long (1330533152/100000) Feb 29 13:32:37 localhost mfschunkserver[9920]: (read) packet too long (67108864/100000) Feb 29 13:32:37 localhost mfschunkserver[9920]: (read) packet too long (115343360/100000) Feb 29 13:32:42 localhost mfschunkserver[9920]: (read) packet too long (788529152/100000) Feb 29 13:32:42 localhost mfschunkserver[9920]: (read) packet too long (3472493488/100000) Feb 29 13:32:42 localhost mfschunkserver[9920]: (read) packet too long (16777216/100000) Feb 29 13:32:42 localhost mfschunkserver[9920]: got unknown message (type:196609) Feb 29 13:32:42 localhost mfschunkserver[9920]: (read) packet too long (790644820/100000) Feb 29 13:33:29 localhost mfschunkserver[9920]: (read) packet too long (790644820/100000) Feb 29 13:33:29 localhost mfschunkserver[9920]: (read) packet too long (1330533152/100000) Feb 29 13:33:29 localhost mfschunkserver[9920]: (read) packet too long (1330533152/100000) Feb 29 13:33:29 localhost mfschunkserver[9920]: (read) packet too long (1929256211/100000) Feb 29 13:33:29 localhost mfschunkserver[9920]: (read) packet too long (16777217/100000) Feb 29 13:33:29 localhost mfschunkserver[9920]: (read) packet too long (268435456/100000) Feb 29 13:33:35 localhost mfschunkserver[9920]: (read) packet too long (1392574464/100000) Feb 29 13:33:35 localhost mfschunkserver[9920]: (read) packet too long (4283649346/100000) Feb 29 13:33:35 localhost mfschunkserver[9920]: got unknown message (type:1811942144) Feb 29 13:33:35 localhost mfschunkserver[9920]: (read) packet too long (795765091/100000) Feb 29 13:33:35 localhost mfschunkserver[9920]: (read) packet too long (1635085428/100000) Feb 29 13:33:35 localhost mfschunkserver[9920]: (read) packet too long (23070466/100000) Feb 29 13:33:35 localhost mfschunkserver[9920]: (read) packet too long (1330533152/100000) Feb 29 13:33:35 localhost mfschunkserver[9920]: (read) packet too long (67108864/100000) Feb 29 13:33:35 localhost mfschunkserver[9920]: (read) packet too long (115343360/100000) Feb 29 13:33:40 localhost mfschunkserver[9920]: (read) packet too long (788529152/100000) Feb 29 13:33:41 localhost mfschunkserver[9920]: (read) packet too long (3472493488/100000) Feb 29 13:33:42 localhost mfschunkserver[9920]: (read) packet too long (16777216/100000) Feb 29 13:33:42 localhost mfschunkserver[9920]: got unknown message (type:196609) Feb 29 13:33:42 localhost mfschunkserver[9920]: (read) packet too long (790644820/100000) Feb 29 14:00:43 localhost mfschunkserver[9920]: masterconn: connection timed out Feb 29 14:00:46 localhost mfschunkserver[9920]: connecting ... Feb 29 14:00:46 localhost mfschunkserver[9920]: connected to Master Feb 29 14:00:56 localhost mfschunkserver[9920]: masterconn: connection timed out Feb 29 14:00:56 localhost mfschunkserver[9920]: connecting ... Feb 29 14:00:56 localhost mfschunkserver[9920]: connected to Master Feb 29 15:00:34 localhost mfschunkserver[9920]: masterconn: connection timed out Feb 29 15:00:36 localhost mfschunkserver[9920]: connecting ... Feb 29 15:00:36 localhost mfschunkserver[9920]: connected to Master Feb 29 15:00:46 localhost mfschunkserver[9920]: masterconn: connection timed out Feb 29 15:00:46 localhost mfschunkserver[9920]: connecting ... Feb 29 15:00:46 localhost mfschunkserver[9920]: connected to Master Feb 29 16:00:20 localhost mfschunkserver[9920]: masterconn: connection timed out Feb 29 16:00:21 localhost mfschunkserver[9920]: connecting ... Feb 29 16:00:21 localhost mfschunkserver[9920]: connected to Master Feb 29 17:00:41 localhost mfschunkserver[9920]: masterconn: connection timed out Feb 29 17:00:46 localhost mfschunkserver[9920]: connecting ... Feb 29 17:00:46 localhost mfschunkserver[9920]: connected to Master Feb 29 18:00:45 localhost mfschunkserver[9920]: masterconn: connection timed out Feb 29 18:00:46 localhost mfschunkserver[9920]: connecting ... Feb 29 18:00:46 localhost mfschunkserver[9920]: connected to Master Feb 29 18:00:56 localhost mfschunkserver[9920]: masterconn: connection timed out Feb 29 18:00:56 localhost mfschunkserver[9920]: connecting ... Feb 29 18:00:56 localhost mfschunkserver[9920]: connected to Master Feb 29 19:00:31 localhost mfschunkserver[9920]: masterconn: connection timed out Feb 29 19:00:36 localhost mfschunkserver[9920]: connecting ... Feb 29 19:00:36 localhost mfschunkserver[9920]: connected to Master Feb 29 19:16:58 localhost mfschunkserver[9920]: (read) packet too long (790644820/100000) Feb 29 19:17:32 localhost mfschunkserver[9920]: (read) packet too long (790644820/100000) Feb 29 21:00:46 localhost mfschunkserver[9920]: masterconn: connection timed out Feb 29 21:00:51 localhost mfschunkserver[9920]: connecting ... Feb 29 21:00:51 localhost mfschunkserver[9920]: connected to Master Feb 29 22:00:31 localhost mfschunkserver[9920]: masterconn: connection timed out Feb 29 22:00:36 localhost mfschunkserver[9920]: connecting ... Feb 29 22:00:36 localhost mfschunkserver[9920]: connected to Master Feb 29 23:00:45 localhost mfschunkserver[9920]: masterconn: connection timed out Feb 29 23:00:46 localhost mfschunkserver[9920]: connecting ... Feb 29 23:00:46 localhost mfschunkserver[9920]: connected to Master Feb 29 23:00:56 localhost mfschunkserver[9920]: masterconn: connection timed out Feb 29 23:00:56 localhost mfschunkserver[9920]: connecting ... Feb 29 23:00:56 localhost mfschunkserver[9920]: connected to Master At 2016-03-02 16:09:18, "Aleksander Wieliczko" <ale...@mo...> wrote: >Hi, > >Thank you for this information. >All results look reasonable. >Also I would like to add that you don't have problems with fork >operation. Metadata dump process took: 24.561 seconds: > >Feb 29 10:00:24 mfs-CNC-GZSX-231 mfsmaster20936: store process has >finished - store time: 24.561 > >The most alarming aspect in your system log is that all chunkservers >disconnected at the same time and 34 seconds after metadata save: > >Feb 29 10:00:58 mfs-CNC-GZSX-231 mfsmaster20936: chunkserver >disconnected - ip: 115.238.*.* / port: 9422, usedspace: 11537568169984 >(10745.20 GiB), totalspace: 22425943621632 (20885.79 GiB) >Feb 29 10:00:58 mfs-CNC-GZSX-231 mfsmaster20936: chunkserver >disconnected - ip: 115.238.*.* / port: 9422, usedspace: 11535864147968 >(10743.61 GiB), totalspace: 22425943621632 (20885.79 GiB) >Feb 29 10:00:58 mfs-CNC-GZSX-231 mfsmaster20936: chunkserver >disconnected - ip: 115.238.*.* / port: 9422, usedspace: 11536655306752 >(10744.35 GiB), totalspace: 22425943621632 (20885.79 GiB) >Feb 29 10:00:58 mfs-CNC-GZSX-231 mfsmaster20936: chunkserver >disconnected - ip: 115.238.*.* / port: 9422, usedspace: 11536570953728 >(10744.27 GiB), totalspace: 22425943621632 (20885.79 GiB) >Feb 29 10:00:58 mfs-CNC-GZSX-231 mfsmaster20936: chunkserver >disconnected - ip: 115.238.*.* / port: 9422, usedspace: 11536672366592 >(10744.36 GiB), totalspace: 22425943621632 (20885.79 GiB) >Feb 29 10:00:58 mfs-CNC-GZSX-231 mfsmaster20936: chunkserver >disconnected - ip: 115.238.*.* / port: 9422, usedspace: 11537568169984 >(10745.20 GiB), totalspace: 22425943621632 (20885.79 GiB) >Feb 29 10:00:58 mfs-CNC-GZSX-231 mfsmaster20936: chunkserver >disconnected - ip: 115.238.*.* / port: 9422, usedspace: 11536639004672 >(10744.33 GiB), totalspace: 22425943621632 (20885.79 GiB) >Feb 29 10:00:58 mfs-CNC-GZSX-231 mfsmaster20936: chunkserver >disconnected - ip: 115.238.*.* / port: 9422, usedspace: 11536581537792 >(10744.28 GiB), totalspace: 22425943621632 (20885.79 GiB) >Feb 29 10:00:58 mfs-CNC-GZSX-231 mfsmaster20936: chunkserver >disconnected - ip: 115.238.*.* / port: 9422, usedspace: 11536540909568 >(10744.24 GiB), totalspace: 22425943621632 (20885.79 GiB) >Feb 29 10:00:58 mfs-CNC-GZSX-231 mfsmaster20936: chunkserver >disconnected - ip: 115.238..*.* / port: 9422, usedspace: 11536369881088 >(10744.08 GiB), totalspace: 22425943621632 (20885.79 GiB) >Feb 29 10:00:58 mfs-CNC-GZSX-231 mfsmaster20936: chunkserver >disconnected - ip: 115.238..*.* / port: 9422, usedspace: 11536581566464 >(10744.28 GiB), totalspace: 22425943621632 (20885.79 GiB) >Feb 29 10:00:58 mfs-CNC-GZSX-231 mfsmaster20936: chunkserver >disconnected - ip: 115.238..*.* / port: 9422, usedspace: 11536586833920 >(10744.28 GiB), totalspace: 22425943621632 (20885.79 GiB) >Feb 29 10:00:58 mfs-CNC-GZSX-231 mfsmaster20936: chunkserver >disconnected - ip: 115.238.*.* / port: 9422, usedspace: 11536620646400 >(10744.32 GiB), totalspace: 22425943621632 (20885.79 GiB) > >Would you be so kind and send us logs from one chunkserver and client >machines. > >We suspect that something is going on in your network. >It's look like your master server loosing network connection. > >Please check your network configuration. >We are waiting for your feedback. > >Best regards >Aleksander Wieliczko >Technical Support Engineer >MooseFS.com <moosefs.com> |
