From: Stéphane B. <ste...@ga...> - 2011-03-10 19:47:49
|
Hi, We have moosefs running 2 months now. And today we got few errors on it. The master has done the: structure check loop and it ended with 72 unavailable chunks and files. I don't know what this really means. The files are still accessible and mfscheckfile output is good too. We have about 2 millions files on the mooseFS currently and 2 chunkserver with a redundant setup. On this cluster we run the version: 1.6.17 we plan to upgrade it soon. Thanks, Stephane |
From: Michal B. <mic...@ge...> - 2011-03-15 12:51:09
|
Hi! These are logs from a test loop. It may happen that while the loop is running some chunkservers are unavailable. The next loop should show that everything is allright. Unless you have numbers in red in the first column in CGI monitor, everything is fine. Kind regards Michał Borychowski MooseFS Support Manager _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Gemius S.A. ul. Wołoska 7, 02-672 Warszawa Budynek MARS, klatka D Tel.: +4822 874-41-00 Fax : +4822 874-41-01 -----Original Message----- From: Stéphane Boisvert [mailto:ste...@ga...] Sent: Thursday, March 10, 2011 8:40 PM To: moosefs-users Subject: [Moosefs-users] Unavailable Chunks Hi, We have moosefs running 2 months now. And today we got few errors on it. The master has done the: structure check loop and it ended with 72 unavailable chunks and files. I don't know what this really means. The files are still accessible and mfscheckfile output is good too. We have about 2 millions files on the mooseFS currently and 2 chunkserver with a redundant setup. On this cluster we run the version: 1.6.17 we plan to upgrade it soon. Thanks, Stephane ---------------------------------------------------------------------------- -- Colocation vs. Managed Hosting A question and answer guide to determining the best fit for your organization - today and in the future. http://p.sf.net/sfu/internap-sfd2d _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
From: Stéphane B. <ste...@ga...> - 2011-03-15 14:24:31
|
Thanks for the answer, But it happens really often. Maybe every 2 or 3 loops. Maybe it is some settings wrong ? We have 2 chunkserver ... that means that both of them was unavailable. I changed the timeout settings a little. Here is my settings. Chunkservers: MASTER_TIMEOUT = 2 HDD_TEST_FREQ = 10 This is the only 2 settings I changed Masters: CHUNKS_LOOP_TIME = 60 CHUNKS_DEL_LIMIT = 5000 CHUNKS_WRITE_REP_LIMIT = 5 CHUNKS_READ_REP_LIMIT = 10 This the only settings I changed the remaining should be Default More explanation about the setup... We have 2 master running Keepalived (same as carp) the chunk servers are on the same servers as the master the machines are Dual quad core Xeons with 8G of Ram and 15k rpm disks in raid 5 On 11-03-15 08:50 AM, Michal Borychowski wrote: > Hi! > > These are logs from a test loop. It may happen that while the loop is > running some chunkservers are unavailable. The next loop should show that > everything is allright. > > Unless you have numbers in red in the first column in CGI monitor, > everything is fine. > > > Kind regards > Michał Borychowski > MooseFS Support Manager > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > Gemius S.A. > ul. Wołoska 7, 02-672 Warszawa > Budynek MARS, klatka D > Tel.: +4822 874-41-00 > Fax : +4822 874-41-01 > > > > -----Original Message----- > From: Stéphane Boisvert [mailto:ste...@ga...] > Sent: Thursday, March 10, 2011 8:40 PM > To: moosefs-users > Subject: [Moosefs-users] Unavailable Chunks > > Hi, > We have moosefs running 2 months now. And today we got few errors > on it. The master has done the: structure check loop and it ended with > 72 unavailable chunks and files. I don't know what this really means. > The files are still accessible and mfscheckfile output is good too. We > have about 2 millions files on the mooseFS currently and 2 chunkserver > with a redundant setup. On this cluster we run the version: 1.6.17 we > plan to upgrade it soon. > > > Thanks, > > Stephane > > > > > ---------------------------------------------------------------------------- > -- > Colocation vs. Managed Hosting > A question and answer guide to determining the best fit > for your organization - today and in the future. > http://p.sf.net/sfu/internap-sfd2d > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > -- *Stephane Boisvert* Unix Administrator Msn : ste...@ga... E-mail :ste...@ga... |
From: Stéphane B. <ste...@ga...> - 2011-03-17 15:40:43
|
Its starting to really concering me. Every loop I get missing chunks and under-goal files... We have production web server running on this and we cannot afford to have unavailable files. When a file is unavailable this means it's a lost sell. The numbers are getting higher. Maybe the load is too high ? We are having about 6 server with about 1 millions lookup per hours each , 500k opens per hours and 70k read per hours. We also have another mooseFS cluster with the same settings on different machines and we have no under-goal files or missing files. Thanks check loop start time check loop end time files under-goal files missing files chunks under-goal chunks missing chunks Thu Mar 17 11:23:52 2011 Thu Mar 17 15:24:58 2011 2125751 25891 2075 2119886 25891 2075 On 11-03-15 10:24 AM, Stéphane Boisvert wrote: > Thanks for the answer, > > But it happens really often. Maybe every 2 or 3 loops. Maybe it is > some settings wrong ? We have 2 chunkserver ... that means that both > of them was unavailable. I changed the timeout settings a little. > > Here is my settings. > > > Chunkservers: > > MASTER_TIMEOUT = 2 > HDD_TEST_FREQ = 10 > > This is the only 2 settings I changed > > Masters: > > CHUNKS_LOOP_TIME = 60 > CHUNKS_DEL_LIMIT = 5000 > CHUNKS_WRITE_REP_LIMIT = 5 > CHUNKS_READ_REP_LIMIT = 10 > > This the only settings I changed the remaining should be Default > > > More explanation about the setup... > > We have 2 master running Keepalived (same as carp) > the chunk servers are on the same servers as the master > the machines are Dual quad core Xeons with 8G of Ram > and 15k rpm disks in raid 5 > > > > On 11-03-15 08:50 AM, Michal Borychowski wrote: >> Hi! >> >> These are logs from a test loop. It may happen that while the loop is >> running some chunkservers are unavailable. The next loop should show that >> everything is allright. >> >> Unless you have numbers in red in the first column in CGI monitor, >> everything is fine. >> >> >> Kind regards >> Michał Borychowski >> MooseFS Support Manager >> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >> Gemius S.A. >> ul. Wołoska 7, 02-672 Warszawa >> Budynek MARS, klatka D >> Tel.: +4822 874-41-00 >> Fax : +4822 874-41-01 >> >> >> >> -----Original Message----- >> From: Stéphane Boisvert [mailto:ste...@ga...] >> Sent: Thursday, March 10, 2011 8:40 PM >> To: moosefs-users >> Subject: [Moosefs-users] Unavailable Chunks >> >> Hi, >> We have moosefs running 2 months now. And today we got few errors >> on it. The master has done the: structure check loop and it ended with >> 72 unavailable chunks and files. I don't know what this really means. >> The files are still accessible and mfscheckfile output is good too. We >> have about 2 millions files on the mooseFS currently and 2 chunkserver >> with a redundant setup. On this cluster we run the version: 1.6.17 we >> plan to upgrade it soon. >> >> >> Thanks, >> >> Stephane >> >> >> >> >> ---------------------------------------------------------------------------- >> -- >> Colocation vs. Managed Hosting >> A question and answer guide to determining the best fit >> for your organization - today and in the future. >> http://p.sf.net/sfu/internap-sfd2d >> _______________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> > > > -- > > > > *Stephane Boisvert* > Unix Administrator > > Msn : ste...@ga... > E-mail :ste...@ga... > > > > > ------------------------------------------------------------------------------ > Colocation vs. Managed Hosting > A question and answer guide to determining the best fit > for your organization - today and in the future. > http://p.sf.net/sfu/internap-sfd2d > > > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users -- *Stephane Boisvert* Unix Administrator Msn : ste...@ga... E-mail :ste...@ga... |