From: lwxian_aha <lwx...@16...> - 2010-05-10 15:17:03
|
today I have new trouble with my new MFS system; my MFS system consist of one masterserver,three chunkserver,every chunkserver with 7.2T diskspace; about 1.4T data and about 7 million files ,every file with 2 copies; today my one chunkserver 's disk is damaged; fist time ,the MFS filesystem is work ok; only problem is lots of file have only on copy; but after about 2 hours , I found that another chunkserver is disappear from the console of mfscgiserv; then, I restart the chunkserver : [root@localhost mfs]# ./sbin/mfschunkserver working directory: /usr/local/mfs/var/mfs lockfile created and locked initializing mfschunkserver modules ... scanning folder /data/mfsdisk1/ ... scanning folder /newdata4/mfs/ ... scanning folder /newdata3/mfs/ ... /data/mfsdisk1/: 277654 chunks found /newdata4/mfs/: 2209988 chunks found /newdata3/mfs/: 2211619 chunks found scanning complete main server module: listen on *:9422 stats file has been loaded mfschunkserver daemon initialized properly it seems the chunkserver is start ok,but it still disappear on the console of mfscgiserv; and then I restart all masterserver and chunkserver ,but after this ,all chunkserver disappear on the console of mfscgiserv; bellow is some message: chunk 1: [root@localhost mfs]# ./sbin/mfschunkserver working directory: /usr/local/mfs/var/mfs lockfile created and locked initializing mfschunkserver modules ... scanning folder /data/mfsdisk1/ ... scanning folder /newdata4/mfs/ ... scanning folder /newdata3/mfs/ ... /data/mfsdisk1/: 277654 chunks found /newdata4/mfs/: 2209988 chunks found /newdata3/mfs/: 2211619 chunks found scanning complete main server module: listen on *:9422 stats file has been loaded mfschunkserver daemon initialized properly [root@localhost mfs]# tail /var/log/messages May 10 20:36:00 localhost mfschunkserver[15720]: connected to Master May 10 20:36:04 localhost mfschunkserver[15720]: testing chunk: /data/mfsdisk1/09/chunk_0000000000287409_00000001.mfs May 10 20:36:10 localhost mfschunkserver[15720]: connecting ... May 10 20:36:10 localhost mfschunkserver[15720]: connected to Master May 10 20:36:14 localhost mfschunkserver[15720]: testing chunk: /newdata4/mfs/D1/chunk_000000000005EAD1_00000001.mfs May 10 20:36:20 localhost mfschunkserver[15720]: connecting ... May 10 20:36:20 localhost mfschunkserver[15720]: connected to Master May 10 20:36:24 localhost mfschunkserver[15720]: testing chunk: /newdata3/mfs/86/chunk_0000000000061086_00000001.mfs May 10 20:36:30 localhost mfschunkserver[15720]: connecting ... May 10 20:36:30 localhost mfschunkserver[15720]: connected to Master chunk 2: [root@localhost mfs]# ./sbin/mfschunkserver working directory: /usr/local/mfs/var/mfs lockfile created and locked initializing mfschunkserver modules ... scanning folder /data/mfsdisk1/ ... scanning folder /newdata2/mfs/ ... scanning folder /newdata1/mfs/ ... /data/mfsdisk1/: 312506 chunks found /newdata2/mfs/: 2182563 chunks found /newdata1/mfs/: 2182481 chunks found scanning complete main server module: listen on *:9422 stats file has been loaded mfschunkserver daemon initialized properly [root@localhost mfs]# tail /var/log/messages May 10 20:33:35 localhost mfschunkserver[25168]: connected to Master May 10 20:33:44 localhost mfschunkserver[25168]: testing chunk: /newdata1/mfs/79/chunk_000000000008A979_00000001.mfs May 10 20:33:45 localhost mfschunkserver[25168]: connecting ... May 10 20:33:45 localhost mfschunkserver[25168]: connected to Master May 10 20:33:54 localhost mfschunkserver[25168]: testing chunk: /data/mfsdisk1/58/chunk_000000000028A658_00000001.mfs May 10 20:33:55 localhost mfschunkserver[25168]: connecting ... May 10 20:33:55 localhost mfschunkserver[25168]: connected to Master May 10 20:34:05 localhost mfschunkserver[25168]: testing chunk: /newdata2/mfs/D6/chunk_00000000000D05D6_00000001.mfs May 10 20:34:05 localhost mfschunkserver[25168]: connecting ... May 10 20:34:05 localhost mfschunkserver[25168]: connected to Master master: [root@localhost mfs]# ./sbin/mfsmaster working directory: /usr/local/mfs/var/mfs lockfile created and locked initializing mfsmaster modules ... loading sessions ... ok sessions file has been loaded exports file has been loaded loading metadata ... loading objects (files,directories,etc.) ... ok loading names ... ok loading deletion timestamps ... ok checking filesystem consistency ... ok loading chunks data ... ok connecting files and chunks ... ok all inodes: 7476300 directory inodes: 19151 file inodes: 7457149 chunks: 7456858 metadata file has been loaded stats file has been loaded master <-> metaloggers module: listen on *:9419 master <-> chunkservers module: listen on *:9420 main master server module: listen on *:9421 mfsmaster daemon initialized properly [root@localhost mfs]# tail /var/log/messages # mass bellow message in the same time May 10 20:34:58 localhost mfsmaster[20168]: * currently unavailable file 158921: photos/47/74/82/r_163042.jpg May 10 20:34:58 localhost mfsmaster[20168]: currently unavailable chunk 00000000004465CA (inode: 4353226 ; index: 0) May 10 20:34:58 localhost mfsmaster[20168]: * currently unavailable file 4353226: photos/46/45/10/857020.jpg May 10 20:34:58 localhost mfsmaster[20168]: currently unavailable chunk 0000000000026B8D (inode: 158922 ; index: 0) May 10 20:34:58 localhost mfsmaster[20168]: * currently unavailable file 158922: photos/47/62/63/m_853671.jpg May 10 20:34:58 localhost mfsmaster[20168]: currently unavailable chunk 00000000004465CB (inode: 4353227 ; index: 0) May 10 20:34:58 localhost mfsmaster[20168]: * currently unavailable file 4353227: photos/46/45/10/s_848116.jpg May 10 20:34:58 localhost mfsmaster[20168]: currently unavailable chunk 0000000000026B8E (inode: 158923 ; index: 0) May 10 20:34:58 localhost mfsmaster[20168]: * currently unavailable file 158923: photos/47/62/63/739789.jpg May 10 20:34:58 localhost mfsmaster[20168]: currently unavailable chunk 00000000004465CC (inode: 4353228 ; index: 0) May 10 20:34:58 localhost mfsmaster[20168]: * currently unavailable file 4353228: photos/46/91/73/936449.jpg May 10 20:34:58 localhost mfsmaster[20168]: currently unavailable chunk 00000000000CBC48 (inode: 158924 ; index: 0) May 10 20:34:58 localhost mfsmaster[20168]: * currently unavailable file 158924: photos/47/74/82/s_171883.jpg May 10 20:34:58 localhost mfsmaster[20168]: currently unavailable chunk 00000000004465CD (inode: 4353229 ; index: 0) May 10 20:34:58 localhost mfsmaster[20168]: * currently unavailable file 4353229: photos/46/45/10/l_848116.jpg May 10 20:34:58 localhost mfsmaster[20168]: currently unavailable chunk 00000000000CBC49 (inode: 158925 ; index: 0) May 10 20:34:58 localhost mfsmaster[20168]: * currently unavailable file 158925: photos/47/74/82/m_156158.jpg May 10 20:34:58 localhost mfsmaster[20168]: currently unavailable chunk 00000000004465CE (inode: 4353230 ; index: 0) May 10 20:34:58 localhost mfsmaster[20168]: * currently unavailable file 4353230: photos/46/91/73/m_928481.jpg May 10 20:34:58 localhost mfsmaster[20168]: currently unavailable chunk 0000000000026B91 (inode: 158926 ; index: 0) May 10 20:34:58 localhost mfsmaster[20168]: * current |