Menu
▾
▴
Re: [Moosefs-users] timeout
|
From: Papp T. <to...@ma...> - 2011-05-28 13:13:10
|
On 05/27/2011 10:19 AM, Papp Tamas wrote: > hi! > > Sometimes there is an error on our mini cluster. > Still Ubuntu Natty with recompiled moosefs from ppa. > > May 27 08:15:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739308400640 (7207.79 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:15:00 backup1 mfsmaster[12869]: total: usedspace: > 7739308400640 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:15:02 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B1/chunk_0000000000005EB1_00000001.mfs > May 27 08:15:12 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B2/chunk_0000000000005EB2_00000001.mfs > May 27 08:15:23 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B3/chunk_0000000000005EB3_00000001.mfs > May 27 08:15:28 backup1 mfsmount[12917]: master: tcp recv error: > ETIMEDOUT (Operation timed out) (1) > May 27 08:15:29 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:15:32 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:15:33 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B4/chunk_0000000000005EB4_00000001.mfs > May 27 08:15:35 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:15:44 backup1 mfsmount[12917]: last message repeated 3 times > May 27 08:15:44 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B5/chunk_0000000000005EB5_00000001.mfs > May 27 08:15:47 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:15:55 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:15:55 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B6/chunk_0000000000005EB6_00000001.mfs > May 27 08:15:56 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:15:59 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:16:00 backup1 mfschunkserver[2556]: connecting ... > May 27 08:16:00 backup1 mfschunkserver[2556]: connected to Master > May 27 08:16:02 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:17:05 backup1 mfsmount[12917]: last message repeated 21 times > May 27 08:17:15 backup1 mfsmount[12917]: last message repeated 3 times > May 27 08:17:15 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:17:15 backup1 mfsmaster[12869]: connection with > CS(192.168.3.21) has been closed by peer > May 27 08:17:15 backup1 mfsmaster[12869]: chunkserver disconnected - ip: > 192.168.3.21, port: 9422, usedspace: 7739308400640 (7207.79 GiB), > totalspace: 10944744390656 (10193.09 GiB) > May 27 08:17:16 backup1 mfsmaster[12869]: connection with > ML(192.168.3.13) has been closed by peer > May 27 08:17:16 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:17:17 backup1 mfsmaster[12869]: last message repeated 7 times > May 27 08:17:17 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:17:17 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:17:19 backup1 mfsmaster[12869]: last message repeated 28 times > May 27 08:17:19 backup1 mfsmount[12917]: registered to master > May 27 08:18:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:18:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:18:16 backup1 mfsmaster[12869]: chunkserver disconnected - ip: > 192.168.3.21, port: 0, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB) > May 27 08:19:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:19:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:19:14 backup1 mfsmount[12917]: file: 7340122, index: 0 - > fs_writechunk returns status 8 > May 27 08:20:00 backup1 mfsmount[12917]: last message repeated 17 times > May 27 08:20:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:20:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:20:05 backup1 mfsmount[12917]: file: 7340122, index: 0 - > fs_writechunk returns status 8 > May 27 08:21:00 backup1 mfsmount[12917]: last message repeated 7 times > May 27 08:21:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:21:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:21:02 backup1 mfsmount[12917]: file: 7340122, index: 0 - > fs_writechunk returns status 8 > May 27 08:21:29 backup1 mfsmount[12917]: last message repeated 3 times > May 27 08:21:29 backup1 mfsmount[12917]: error writing file number > 7340122: EIO (Input/output error) > May 27 08:21:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: > 4816320, version: 1 - there are no valid copies > May 27 08:21:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't > connect to proper chunkserver (try counter: 1) > May 27 08:22:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:22:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:22:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: > 4816320, version: 1 - there are no valid copies > May 27 08:22:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't > connect to proper chunkserver (try counter: 8) > May 27 08:23:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:23:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:23:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: > 4816320, version: 1 - there are no valid copies > May 27 08:23:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't > connect to proper chunkserver (try counter: 15) > May 27 08:24:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:24:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:24:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: > 4816320, version: 1 - there are no valid copies > May 27 08:24:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't > connect to proper chunkserver (try counter: 22) > May 27 08:25:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:25:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:25:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: > 4816320, version: 1 - there are no valid copies > May 27 08:25:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't > connect to proper chunkserver (try counter: 29) > May 27 08:26:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:26:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:26:10 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B7/chunk_0000000000005EB7_00000001.mfs > May 27 08:26:10 backup1 mfschunkserver[2556]: connection reset by Master > May 27 08:26:20 backup1 mfschunkserver[2556]: connecting ... > May 27 08:26:20 backup1 mfschunkserver[2556]: connected to Master > May 27 08:26:21 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B8/chunk_0000000000005EB8_00000001.mfs > May 27 08:26:21 backup1 mfsmaster[12869]: chunkserver register begin > (packet version: 5) - ip: 192.168.3.21, port: 9422 > May 27 08:26:28 backup1 mfsmaster[12869]: chunkserver register end > (packet version: 5) - ip: 192.168.3.21, port: 9422, usedspace: > 7739308400640 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB) > May 27 08:26:31 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B9/chunk_0000000000005EB9_00000001.mfs > May 27 08:26:41 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BA/chunk_0000000000005EBA_00000001.mfs > May 27 08:26:42 backup1 mfsmount[12917]: file: 7445943, index: 0, chunk: > 4816323, version: 1 - writeworker: connection with (C0A80315:9422) was > timed out (unfinished writes: 2; try counter: 1) > May 27 08:26:51 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:26:51 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BB/chunk_0000000000005EBB_00000001.mfs > May 27 08:27:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:27:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739309965312 (7207.79 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:27:00 backup1 mfsmaster[12869]: total: usedspace: > 7739309965312 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:27:01 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BC/chunk_0000000000005EBC_00000001.mfs > May 27 08:27:11 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BD/chunk_0000000000005EBD_00000001.mfs > May 27 08:27:21 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BE/chunk_0000000000005EBE_00000001.mfs > May 27 08:27:31 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BF/chunk_0000000000005EBF_00000001.mfs > May 27 08:27:41 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C0/chunk_0000000000005EC0_00000001.mfs > May 27 08:27:51 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C1/chunk_0000000000005EC1_00000001.mfs > May 27 08:28:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:28:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739310948352 (7207.79 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:28:00 backup1 mfsmaster[12869]: total: usedspace: > 7739310948352 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:28:01 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C3/chunk_0000000000005EC3_00000001.mfs > May 27 08:28:11 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C4/chunk_0000000000005EC4_00000001.mfs > May 27 08:28:22 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C5/chunk_0000000000005EC5_00000001.mfs > May 27 08:28:32 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C7/chunk_0000000000005EC7_00000001.mfs > May 27 08:28:42 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C8/chunk_0000000000005EC8_00000001.mfs > May 27 08:28:52 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C9/chunk_0000000000005EC9_00000001.mfs > May 27 08:29:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739313045504 (7207.80 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:29:00 backup1 mfsmaster[12869]: total: usedspace: > 7739313045504 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:29:02 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/CA/chunk_0000000000005ECA_00000001.mfs > May 27 08:29:12 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/CB/chunk_0000000000005ECB_00000001.mfs > May 27 08:29:22 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/CC/chunk_0000000000005ECC_00000001.mfs > May 27 08:29:26 backup1 mfsmount[12917]: master: tcp recv error: > ETIMEDOUT (Operation timed out) (1) > May 27 08:29:27 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:29:30 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:29:32 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/CD/chunk_0000000000005ECD_00000001.mfs > May 27 08:29:32 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:29:32 backup1 mfsmaster[12869]: last message repeated 2 times > May 27 08:29:32 backup1 mfsmount[12917]: registered to master > May 27 08:29:42 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/CE/chunk_0000000000005ECE_00000001.mfs > May 27 08:29:52 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D0/chunk_0000000000005ED0_00000001.mfs > May 27 08:30:02 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D1/chunk_0000000000005ED1_00000001.mfs > May 27 08:30:05 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: > 4816324, version: 1 - writeworker: connection with (C0A80315:9422) was > timed out (unfinished writes: 1; try counter: 1) > May 27 08:30:12 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D2/chunk_0000000000005ED2_00000001.mfs > May 27 08:30:22 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D3/chunk_0000000000005ED3_00000001.mfs > May 27 08:30:33 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D4/chunk_0000000000005ED4_00000001.mfs > May 27 08:30:43 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D5/chunk_0000000000005ED5_00000001.mfs > May 27 08:30:53 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D6/chunk_0000000000005ED6_00000001.mfs > May 27 08:31:03 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D7/chunk_0000000000005ED7_00000001.mfs > May 27 08:31:13 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D8/chunk_0000000000005ED8_00000001.mfs > May 27 08:31:23 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D9/chunk_0000000000005ED9_00000001.mfs > May 27 08:31:33 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/DA/chunk_0000000000005EDA_00000001.mfs > May 27 08:31:43 backup1 mfsmount[12917]: master: tcp recv error: > ETIMEDOUT (Operation timed out) (1) > May 27 08:31:43 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/DB/chunk_0000000000005EDB_00000001.mfs > May 27 08:31:44 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:31:53 backup1 mfsmount[12917]: last message repeated 3 times > May 27 08:31:53 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/DD/chunk_0000000000005EDD_00000001.mfs > May 27 08:31:56 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:32:03 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:32:03 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/DE/chunk_0000000000005EDE_00000001.mfs > May 27 08:32:05 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:32:14 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:32:14 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/DF/chunk_0000000000005EDF_00000001.mfs > May 27 08:32:14 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:32:24 backup1 mfsmount[12917]: last message repeated 3 times > May 27 08:32:24 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E0/chunk_0000000000005EE0_00000001.mfs > May 27 08:32:25 backup1 mfschunkserver[2556]: connecting ... > May 27 08:32:25 backup1 mfschunkserver[2556]: connected to Master > May 27 08:32:26 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:33:29 backup1 mfsmount[12917]: last message repeated 21 times > May 27 08:34:25 backup1 mfsmount[12917]: last message repeated 18 times > May 27 08:34:25 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:34:25 backup1 mfsmaster[12869]: connection with > CS(192.168.3.21) has been closed by peer > May 27 08:34:25 backup1 mfsmaster[12869]: chunkserver disconnected - ip: > 192.168.3.21, port: 9422, usedspace: 7739311943680 (7207.80 GiB), > totalspace: 10944744390656 (10193.09 GiB) > May 27 08:34:26 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:34:33 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:34:33 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E1/chunk_0000000000005EE1_00000001.mfs > May 27 08:34:34 backup1 mfsmaster[12869]: connection with > ML(192.168.3.13) has been closed by peer > May 27 08:34:34 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:34:34 backup1 mfsmaster[12869]: chunkserver register begin > (packet version: 5) - ip: 192.168.3.21, port: 9422 > May 27 08:34:34 backup1 mfsmaster[12869]: connection with > ML(192.168.3.13) has been closed by peer > May 27 08:34:34 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:34:34 backup1 mfsmaster[12869]: connection with > CS(192.168.3.21) has been closed by peer > May 27 08:34:34 backup1 mfsmaster[12869]: chunkserver disconnected - ip: > 192.168.3.21, port: 9422, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB) > May 27 08:34:35 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:34:35 backup1 mfsmaster[12869]: last message repeated 2 times > May 27 08:34:35 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:34:35 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:34:37 backup1 mfsmaster[12869]: last message repeated 52 times > May 27 08:34:37 backup1 mfsmount[12917]: registered to master > May 27 08:34:39 backup1 mfsmount[12917]: file: 7445943, index: 0 - > fs_writechunk returns status 8 > May 27 08:34:40 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:34:40 backup1 mfschunkserver[2556]: connecting ... > May 27 08:34:40 backup1 mfschunkserver[2556]: connected to Master > May 27 08:34:40 backup1 mfsmount[12917]: file: 7445943, index: 0 - > fs_writechunk returns status 8 > May 27 08:34:41 backup1 mfsmount[12917]: file: 7445943, index: 0 - > fs_writechunk returns status 8 > May 27 08:34:41 backup1 mfsmaster[12869]: chunkserver register begin > (packet version: 5) - ip: 192.168.3.21, port: 9422 > May 27 08:34:42 backup1 mfsmaster[12869]: chunkserver register end > (packet version: 5) - ip: 192.168.3.21, port: 9422, usedspace: > 7739311943680 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB) > May 27 08:34:43 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E2/chunk_0000000000005EE2_00000001.mfs > May 27 08:34:53 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E3/chunk_0000000000005EE3_00000001.mfs > May 27 08:35:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:35:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739314479104 (7207.80 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:35:00 backup1 mfsmaster[12869]: total: usedspace: > 7739314479104 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:35:03 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E4/chunk_0000000000005EE4_00000001.mfs > May 27 08:35:14 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E5/chunk_0000000000005EE5_00000001.mfs > May 27 08:35:24 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E6/chunk_0000000000005EE6_00000001.mfs > May 27 08:35:34 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E7/chunk_0000000000005EE7_00000001.mfs > May 27 08:35:44 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E8/chunk_0000000000005EE8_00000001.mfs > May 27 08:35:54 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E9/chunk_0000000000005EE9_00000001.mfs > May 27 08:36:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:36:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739314479104 (7207.80 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:36:00 backup1 mfsmaster[12869]: total: usedspace: > 7739314479104 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:36:04 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/EA/chunk_0000000000005EEA_00000001.mfs > May 27 08:36:14 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/EB/chunk_0000000000005EEB_00000001.mfs > May 27 08:36:24 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/EC/chunk_0000000000005EEC_00000001.mfs > May 27 08:36:34 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/ED/chunk_0000000000005EED_00000001.mfs > May 27 08:36:45 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/EE/chunk_0000000000005EEE_00000001.mfs > May 27 08:36:55 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/EF/chunk_0000000000005EEF_00000001.mfs > May 27 08:37:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:37:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739314479104 (7207.80 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:37:00 backup1 mfsmaster[12869]: total: usedspace: > 7739314479104 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > > > Can you help me what to do with it? > Is this fuse, moosefs, kernel problem or something else? hi! I've just realized an other error. $ dirvish --vault cluster/Projects dirvish cluster/Projects:default fatal error: filesystem full dirvish cluster/Projects:default fatal error (12) -- filesystem full cluster/Projects:default post-server failed (1) log: May 28 14:58:04 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/70/chunk_0000000000008470_00000001.mfs May 28 14:58:14 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/71/chunk_0000000000008471_00000001.mfs May 28 14:58:24 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/72/chunk_0000000000008472_00000001.mfs May 28 14:58:34 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/73/chunk_0000000000008473_00000001.mfs May 28 14:58:44 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/74/chunk_0000000000008474_00000001.mfs May 28 14:58:55 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/75/chunk_0000000000008475_00000001.mfs May 28 14:59:00 backup1 mfsmaster[12869]: chunkservers status: May 28 14:59:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, port: 9422): usedspace: 7734218166272 (7203.05 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.67% May 28 14:59:00 backup1 mfsmaster[12869]: total: usedspace: 7734218166272 (7203.05 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.67% May 28 14:59:05 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/76/chunk_0000000000008476_00000001.mfs May 28 14:59:15 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/77/chunk_0000000000008477_00000001.mfs May 28 14:59:24 backup1 mfsmount[12917]: master: tcp recv error: ETIMEDOUT (Operation timed out) (1) May 28 14:59:25 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 14:59:25 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/78/chunk_0000000000008478_00000001.mfs May 28 14:59:28 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 14:59:35 backup1 mfsmount[12917]: last message repeated 2 times May 28 14:59:35 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/79/chunk_0000000000008479_00000001.mfs May 28 14:59:37 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 14:59:45 backup1 mfsmount[12917]: last message repeated 2 times May 28 14:59:45 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7A/chunk_000000000000847A_00000001.mfs May 28 14:59:46 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 14:59:55 backup1 mfsmount[12917]: last message repeated 3 times May 28 14:59:55 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7B/chunk_000000000000847B_00000001.mfs May 28 14:59:58 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 15:00:06 backup1 mfsmount[12917]: last message repeated 2 times May 28 15:00:06 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7C/chunk_000000000000847C_00000001.mfs May 28 15:00:07 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 15:00:15 backup1 mfsmount[12917]: last message repeated 2 times May 28 15:00:15 backup1 mfschunkserver[2556]: connecting ... May 28 15:00:15 backup1 mfschunkserver[2556]: connected to Master May 28 15:00:16 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 15:01:19 backup1 mfsmount[12917]: last message repeated 21 times May 28 15:02:19 backup1 mfsmount[12917]: last message repeated 19 times May 28 15:02:19 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 15:02:19 backup1 mfsmaster[12869]: connection with CS(192.168.3.21) has been closed by peer May 28 15:02:19 backup1 mfsmaster[12869]: chunkserver disconnected - ip: 192.168.3.21, port: 9422, usedspace: 7734399709184 (7203.22 GiB), totalspace: 10944744390656 (10193.09 GiB) May 28 15:02:19 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 15:02:22 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 15:02:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.20) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.13) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.13) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.20) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.13) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 15:02:24 backup1 mfsmaster[12869]: last message repeated 55 times May 28 15:02:24 backup1 mfsmount[12917]: registered to master May 28 15:02:25 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: 4816943, version: 1 - there are no valid copies May 28 15:02:25 backup1 mfsmount[12917]: file: 122301, index: 0 - can't connect to proper chunkserver (try counter: 1) May 28 15:02:26 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:26 backup1 mfsmount[12917]: last message repeated 2 times May 28 15:02:26 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:02:27 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:32 backup1 mfsmount[12917]: last message repeated 3 times May 28 15:02:32 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:02:33 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:38 backup1 mfsmount[12917]: last message repeated 2 times May 28 15:02:38 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:02:41 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:44 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:44 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:02:48 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:51 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:02:52 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:56 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:58 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:01 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:05 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:06 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:11 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:13 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:17 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:20 backup1 mfsmaster[12869]: chunkserver disconnected - ip: 192.168.3.21, port: 0, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB) May 28 15:03:22 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:23 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:25 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: 4816943, version: 1 - there are no valid copies May 28 15:03:25 backup1 mfsmount[12917]: file: 122301, index: 0 - can't connect to proper chunkserver (try counter: 8) May 28 15:03:29 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:30 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:36 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:39 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:43 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:48 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:50 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:57 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:57 backup1 mfsmount[12917]: error writing file number 107222: EIO (Input/output error) May 28 15:03:58 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:04:00 backup1 mfsmaster[12869]: chunkservers status: May 28 15:04:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 28 15:04:06 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:04:25 backup1 mfsmount[12917]: last message repeated 2 times May 28 15:04:25 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: 4816943, version: 1 - there are no valid copies May 28 15:04:25 backup1 mfsmount[12917]: file: 122301, index: 0 - can't connect to proper chunkserver (try counter: 15) May 28 15:04:32 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:04:41 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:04:41 backup1 mfsmount[12917]: error writing file number 4691391: ENOSPC (No space left on device) May 28 15:04:42 backup1 mfsmount[12917]: file: 107221, index: 0 - fs_writechunk returns status 8 May 28 15:05:00 backup1 mfsmount[12917]: last message repeated 10 times May 28 15:05:00 backup1 mfsmaster[12869]: chunkservers status: May 28 15:05:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 28 15:05:00 backup1 mfsmount[12917]: file: 107221, index: 0 - fs_writechunk returns status 8 May 28 15:05:25 backup1 mfsmount[12917]: last message repeated 5 times May 28 15:05:25 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: 4816943, version: 1 - there are no valid copies May 28 15:05:25 backup1 mfsmount[12917]: file: 122301, index: 0 - can't connect to proper chunkserver (try counter: 22) May 28 15:05:27 backup1 mfsmount[12917]: file: 107221, index: 0 - fs_writechunk returns status 8 May 28 15:06:00 backup1 mfsmount[12917]: last message repeated 4 times May 28 15:06:00 backup1 mfsmaster[12869]: chunkservers status: May 28 15:06:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 28 15:06:01 backup1 mfsmount[12917]: file: 107221, index: 0 - fs_writechunk returns status 8 May 28 15:06:25 backup1 mfsmount[12917]: last message repeated 3 times May 28 15:06:25 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: 4816943, version: 1 - there are no valid copies May 28 15:06:25 backup1 mfsmount[12917]: file: 122301, index: 0 - can't connect to proper chunkserver (try counter: 29) May 28 15:06:25 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7D/chunk_000000000000847D_00000001.mfs May 28 15:06:25 backup1 mfschunkserver[2556]: connection reset by Master May 28 15:06:32 backup1 mfsmount[12917]: file: 107221, index: 0 - fs_writechunk returns status 8 May 28 15:06:35 backup1 mfschunkserver[2556]: connecting ... May 28 15:06:35 backup1 mfschunkserver[2556]: connected to Master May 28 15:06:36 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7E/chunk_000000000000847E_00000001.mfs May 28 15:06:36 backup1 mfsmaster[12869]: chunkserver register begin (packet version: 5) - ip: 192.168.3.21, port: 9422 May 28 15:06:38 backup1 mfsmaster[12869]: chunkserver register end (packet version: 5) - ip: 192.168.3.21, port: 9422, usedspace: 7734393311232 (7203.22 GiB), totalspace: 10944744390656 (10193.09 GiB) May 28 15:06:46 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7F/chunk_000000000000847F_00000001.mfs May 28 15:06:56 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/80/chunk_0000000000008480_00000001.mfs May 28 15:07:00 backup1 mfsmaster[12869]: chunkservers status: May 28 15:07:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, port: 9422): usedspace: 7734393442304 (7203.22 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.67% May 28 15:07:00 backup1 mfsmaster[12869]: total: usedspace: 7734393442304 (7203.22 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.67% Filesystem Size Used Avail Use% Mounted on /dev/sda2 19G 1.4G 17G 8% / none 3.9G 180K 3.9G 1% /dev none 4.0G 0 4.0G 0% /dev/shm none 4.0G 716K 4.0G 1% /var/run none 4.0G 0 4.0G 0% /var/lock /dev/sda6 10T 7.1T 3.0T 71% /mnt/mfschunk1 /dev/sda3 19G 12G 6.0G 66% /var /dev/sda4 4.6G 138M 4.3G 4% /tmp mfsmaster:9421 10T 7.1T 3.0T 71% /data/backup Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 1222992 64010 1158982 6% / none 1021194 762 1020432 1% /dev none 1023079 1 1023078 1% /dev/shm none 1023079 58 1023021 1% /var/run none 1023079 1 1023078 1% /var/lock /dev/sda6 2138062720 4778154 2133284566 1% /mnt/mfschunk1 /dev/sda3 1222992 3174 1219818 1% /var /dev/sda4 305216 13 305203 1% /tmp mfsmaster:9421 1031528383 30522363 1001006020 3% /data/backup So it's definetly not full. I don't understand, what's going on:/ Does somebody have any idea? Thank you, tamas |
