Menu
▾
▴
[Moosefs-users] replication status 19 causing RO on vm images?
|
From: wkmail <wk...@bn...> - 2012-05-24 19:56:25
|
We have a test MFS cluster (1.6.20) that we are hosting some utility/research VM's on it using Oracle Virtual Box to see how well MFS handles VMs. All has gone well besides the expected disk i/o speed hit, which we are can handle. However we have TWICE now experienced this problem. We decide to replace one of the chunkservers with better gear. It had 2 x 1TB SATA drives and we usually remove them one at a time. We first marked drive #2 for removal using '*' in mfshdd.cfg and restarted the chunkserver process (.i.e. ./chunkerserver restart). The CGI showed the drive as marked for removal and replicated off the chunks as normal. Two days later it was done. No problems or errors were observed. The following day, we then excluded drive #2 the now deprecated drive by commenting it out in mfshdd.cfg and restarted the chunkerserver again. During the scan, the CGI correctly reported the 650,000+ chunks or so as undergoal and the chunkserver was missing from the server list. When the chunkserver process finished scanning the single remaining drive, the CGI program reported that all chunkers were back but now showed 700+ chunks as undergoal (2/3) AND 700+ chunks as overgoal which confused us. The MFS cluster then began to fix the undergoal chunks. During that period we began to see lot of the error 19 messages which the source code identifies as Wrong Chunk Version. During the undergoal replication process, we had a number of the busier VMs report hard drive errors and either go RO or Panic. When the undergoal chunk replication completed about an hour later, we ceased to have any problems and we are no longer seeing the error 19 messages. Does anyone have an explanation as to what is going on and what we can do to prevent the issue from re-occurring? Are Disk images problematic on MFS? Below are the relevant log entries in the MFSmaster log showing us restarting the CS and the subsequent errors. Note, on the chunkerservers themselves, each error in the master log has a corresponding error in the Chunkserver log i.e. May 24 10:58:25 mfs7chunker7 mfschunkserver[1085]: replicator: got status: 19 from (C0A80016:24CE) May 24 10:58:26 mfs7chunker7 mfschunkserver[1085]: replicator: got status: 19 from (C0A80018:24CE) from /var/log/messages on the master. May 24 10:27:45 mfs7master mfsmaster[32522]: connection with CS(192.168.0.23) has been closed by peer May 24 10:32:49 mfs7master mfsmaster[32522]: chunkserver register begin (packet version: 5) - ip: 192.168.0.23, port: 9422 May 24 10:33:37 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000BA8A31 replication status: 19 May 24 10:34:43 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000B85FD0 replication status: 19 May 24 10:35:05 mfs7master mfsmaster[32522]: (192.168.0.25:9422) chunk: 0000000000CF1D62 replication status: 19 May 24 10:35:26 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000C519DE replication status: 19 May 24 10:35:34 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000C61B66 replication status: 19 May 24 10:36:05 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000CD75B2 replication status: 19 May 24 10:36:19 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000CA45CC replication status: 19 May 24 10:36:57 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000CBA189 replication status: 19 May 24 10:37:03 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000B12E19 replication status: 19 May 24 10:37:19 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000B4ED79 replication status: 19 May 24 10:37:20 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000CBA1A6 replication status: 19 May 24 10:37:32 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000CD784C replication status: 19 May 24 10:38:14 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000ADC1A1 replication status: 19 May 24 10:38:43 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000CD7055 replication status: 19 May 24 10:39:25 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000A9BE51 replication status: 19 May 24 10:39:51 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000A9BFA2 replication status: 19 May 24 10:40:00 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000CBA184 replication status: 19 May 24 10:40:07 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000CF45E5 replication status: 19 May 24 10:40:31 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000AA4400 replication status: 19 May 24 10:40:33 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000A995E4 replication status: 19 May 24 10:40:40 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000CA4B25 replication status: 19 May 24 10:40:42 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000A9CD0E replication status: 19 May 24 10:40:48 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000CD788D replication status: 19 May 24 10:41:13 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000A9C033 replication status: 19 May 24 10:41:20 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000CA45A5 replication status: 19 May 24 10:41:41 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000CD64E0 replication status: 19 May 24 10:41:47 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000CD7605 replication status: 19 May 24 10:42:10 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000BCB066 replication status: 19 May 24 10:43:37 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000ADB285 replication status: 19 May 24 10:43:38 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000CD78B6 replication status: 19 May 24 10:43:42 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000A9BF85 replication status: 19 May 24 10:43:43 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000ADB3B0 replication status: 19 May 24 10:44:01 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000BF1674 replication status: 19 May 24 10:44:09 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000CB5FC1 replication status: 19 May 24 10:44:18 mfs7master mfsmaster[32522]: (192.168.0.25:9422) chunk: 0000000000C790A4 replication status: 19 May 24 10:44:22 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000ADC29C replication status: 19 May 24 10:44:23 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000CD7E62 replication status: 19 May 24 10:44:31 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000CD78F2 replication status: 19 May 24 10:45:02 mfs7master mfsmaster[32522]: (192.168.0.25:9422) chunk: 0000000000CA4B55 replication status: 19 May 24 10:45:13 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000C2F8BF replication status: 19 May 24 10:45:25 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000C2FF5A replication status: 19 May 24 10:45:42 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000CD6510 replication status: 19 May 24 10:45:44 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000A9C007 replication status: 19 May 24 10:46:00 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000CF1D2D replication status: 28 May 24 10:46:00 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000CA3FA7 replication status: 28 May 24 10:46:00 mfs7master mfsmaster[32522]: (192.168.0.25:9422) chunk: 0000000000A9C41B replication status: 28 May 24 10:46:06 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000C78D16 replication status: 19 May 24 10:46:15 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000A9C156 replication status: 19 May 24 10:46:18 mfs7master mfsmaster[32522]: (192.168.0.25:9422) chunk: 0000000000CA45A5 replication status: 19 May 24 10:46:32 mfs7master mfsmaster[32522]: (192.168.0.25:9422) chunk: 0000000000BA89D7 replication status: 19 May 24 10:46:33 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000CD79CF replication status: 19 May 24 10:46:40 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000A9CCB7 replication status: 19 May 24 10:47:07 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000A9BEE6 replication status: 19 May 24 10:47:36 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000CA458E replication status: 19 May 24 10:47:44 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000C7B9F1 replication status: 19 May 24 10:47:51 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000ADB2AC replication status: 19 May 24 10:48:00 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000CD645B replication status: 19 May 24 10:48:01 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000A995B9 replication status: 19 May 24 10:48:08 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000C2F955 replication status: 19 May 24 10:48:19 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000C8007A replication status: 19 May 24 10:48:21 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000C78CF4 replication status: 19 May 24 10:48:38 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000CD78B6 replication status: 28 May 24 10:48:39 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000BA8AA4 replication status: 28 May 24 10:48:40 mfs7master mfsmaster[32522]: (192.168.0.25:9422) chunk: 0000000000CBA179 replication status: 28 May 24 10:48:41 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000B590BC replication status: 19 May 24 10:48:47 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000CD7929 replication status: 19 May 24 10:48:53 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000BF1674 replication status: 19 May 24 10:48:56 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000C2F8AA replication status: 19 May 24 10:48:59 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000A9DAC9 replication status: 19 May 24 10:49:04 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000A996F3 replication status: 19 May 24 10:49:08 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000CB5FC1 replication status: 19 May 24 10:49:32 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000CBA1B2 replication status: 19 May 24 10:49:36 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000C790A4 replication status: 19 May 24 10:49:43 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000C7913D replication status: 19 May 24 10:49:51 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000B582B4 replication status: 19 May 24 10:50:35 mfs7master mfsmaster[32522]: (192.168.0.25:9422) chunk: 0000000000CF1CE1 replication status: 19 May 24 10:51:03 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000C78D16 replication status: 19 May 24 10:51:04 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000C2F366 replication status: 19 May 24 10:51:18 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000CA45A5 replication status: 19 May 24 10:51:24 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000A9BED0 replication status: 19 May 24 10:51:33 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000A91554 replication status: 19 May 24 10:51:37 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000A9154D replication status: 19 May 24 10:52:01 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000B12E19 replication status: 19 May 24 10:52:37 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000ADC2CD replication status: 19 May 24 10:53:22 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000CBA195 replication status: 19 May 24 10:53:42 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000C33575 replication status: 19 May 24 10:54:28 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000A9C82A replication status: 19 May 24 10:54:48 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000B582B4 replication status: 19 May 24 10:55:13 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000C78EDD replication status: 19 May 24 10:55:35 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000CA4B44 replication status: 19 May 24 10:55:58 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000CB5FBE replication status: 19 May 24 10:56:06 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000BF6DF2 replication status: 19 May 24 10:56:26 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000A9BE28 replication status: 19 May 24 10:56:40 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000CD789C replication status: 19 May 24 10:56:48 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000CD7837 replication status: 19 May 24 10:56:58 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000CD7800 replication status: 19 May 24 10:57:11 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000A9C09C replication status: 19 May 24 10:57:26 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000A9BE0F replication status: 19 May 24 10:57:40 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000CD784C replication status: 19 May 24 10:57:49 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000A9C00B replication status: 19 May 24 10:57:56 mfs7master mfsmaster[32522]: (192.168.0.25:9422) chunk: 0000000000CBA179 replication status: 19 May 24 10:58:08 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000A995B9 replication status: 19 May 24 10:58:14 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000C79138 replication status: 19 May 24 10:58:23 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000CD707C replication status: 19 May 24 10:58:25 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000CD710F replication status: 19 May 24 10:58:26 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000C5769C replication status: 19 May 24 10:58:55 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000C68224 replication status: 19 May 24 10:59:33 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000CBA1B2 replication status: 19 May 24 11:00:05 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000B3CB41 replication status: 19 May 24 11:00:54 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000AA4400 replication status: 19 May 24 11:01:17 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000CD7837 replication status: 19 May 24 11:01:33 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000BA89D7 replication status: 19 May 24 11:01:51 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000C8375C replication status: 19 May 24 11:02:22 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000A995AC replication status: 19 May 24 11:02:28 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000CD78B4 replication status: 19 May 24 11:02:32 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000C615E8 replication status: 19 May 24 11:02:35 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000CD64D7 replication status: 19 May 24 11:02:51 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000CBA196 replication status: 19 May 24 11:02:58 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000BEC4A1 replication status: 19 May 24 11:03:09 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000C79138 replication status: 19 May 24 11:03:18 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000A9C260 replication status: 19 May 24 11:03:21 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000ADB266 replication status: 19 May 24 11:03:23 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000CBA195 replication status: 19 May 24 11:03:27 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000C68224 replication status: 19 May 24 11:03:30 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000C5769C replication status: 19 May 24 11:03:32 mfs7master mfsmaster[32522]: (192.168.0.25:9422) chunk: 0000000000A9CCC9 replication status: 19 May 24 11:04:15 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000CD70D7 replication status: 19 May 24 11:04:25 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000CB5FB1 replication status: 19 May 24 11:04:45 mfs7master mfsmaster[32522]: (192.168.0.25:9422) chunk: 0000000000CD7DDF replication status: 19 May 24 11:05:43 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000C8375C replication status: 19 May 24 11:05:50 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000CD7625 replication status: 19 May 24 11:06:01 mfs7master mfsmaster[32522]: (192.168.0.25:9422) chunk: 0000000000BF163B replication status: 19 May 24 11:06:03 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000A91545 replication status: 19 May 24 11:06:08 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000A9BE28 replication status: 19 May 24 11:06:13 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000C7914A replication status: 19 May 24 11:07:28 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000CD784C replication status: 19 May 24 11:07:43 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000C78EE1 replication status: 19 May 24 11:08:08 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000B4EDF3 replication status: 19 May 24 11:08:21 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000CBA195 replication status: 19 May 24 11:08:28 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000C5769C replication status: 19 May 24 11:08:35 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000C7906F replication status: 19 May 24 11:08:43 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000CD75C5 replication status: 19 May 24 11:08:48 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000A9BF85 replication status: 19 May 24 11:09:12 mfs7master mfsmaster[32522]: (192.168.0.25:9422) chunk: 0000000000CD70D7 replication status: 19 May 24 11:10:01 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000CF1D62 replication status: 19 May 24 11:10:07 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000CA4588 replication status: 19 May 24 11:10:25 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000B578E9 replication status: 19 May 24 11:10:39 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000CA4B44 replication status: 19 May 24 11:11:15 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000C7914A replication status: 19 May 24 11:12:28 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000C7B551 replication status: 19 May 24 11:12:28 mfs7master mfsmaster[32522]: (192.168.0.25:9422) chunk: 0000000000CD784C replication status: 19 May 24 11:12:35 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000C78EE1 replication status: 19 May 24 11:13:25 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000ADB266 replication status: 19 May 24 11:13:25 mfs7master mfsmaster[32522]: (192.168.0.25:9422) chunk: 0000000000C5769C replication status: 19 May 24 11:13:41 mfs7master mfsmaster[32522]: (192.168.0.27:9422) chunk: 0000000000A9BF85 replication status: 19 May 24 11:15:08 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000CA4588 replication status: 19 May 24 11:15:19 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000B578E9 replication status: 19 May 24 11:15:55 mfs7master mfsmaster[32522]: (192.168.0.25:9422) chunk: 0000000000BF163B replication status: 19 May 24 11:15:56 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000A9BE28 replication status: 19 May 24 11:16:00 mfs7master mfsmaster[32522]: (192.168.0.21:9422) chunk: 0000000000CD7625 replication status: 19 May 24 11:16:30 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000C3A0E7 replication status: 19 May 24 11:16:45 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000CBA190 replication status: 19 May 24 11:17:31 mfs7master mfsmaster[32522]: (192.168.0.25:9422) chunk: 0000000000CD784C replication status: 19 May 24 11:17:36 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000C78EE1 replication status: 19 May 24 11:18:25 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000C5769C replication status: 19 May 24 11:20:08 mfs7master mfsmaster[32522]: (192.168.0.24:9422) chunk: 0000000000CA4588 replication status: 19 May 24 11:20:47 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000CD7625 replication status: 19 May 24 11:23:26 mfs7master mfsmaster[32522]: (192.168.0.22:9422) chunk: 0000000000C5769C replication status: 19 May 24 11:25:06 mfs7master mfsmaster[32522]: (192.168.0.23:9422) chunk: 0000000000CA4588 replication status: 19 May 24 11:25:48 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000CD7625 replication status: 19 May 24 11:30:04 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000CA4588 replication status: 19 May 24 11:40:04 mfs7master mfsmaster[32522]: (192.168.0.26:9422) chunk: 0000000000CA4588 replication status: 19 Note the undergoal replication process was completed about this time. |
