Menu
▾
▴
moosefs-users — Mailing list for MooseFS Users
You can subscribe to this list here.
| 2009 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(4) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2010 |
Jan
(20) |
Feb
(11) |
Mar
(11) |
Apr
(9) |
May
(22) |
Jun
(85) |
Jul
(94) |
Aug
(80) |
Sep
(72) |
Oct
(64) |
Nov
(69) |
Dec
(89) |
| 2011 |
Jan
(72) |
Feb
(109) |
Mar
(116) |
Apr
(117) |
May
(117) |
Jun
(102) |
Jul
(91) |
Aug
(72) |
Sep
(51) |
Oct
(41) |
Nov
(55) |
Dec
(74) |
| 2012 |
Jan
(45) |
Feb
(77) |
Mar
(99) |
Apr
(113) |
May
(132) |
Jun
(75) |
Jul
(70) |
Aug
(58) |
Sep
(58) |
Oct
(37) |
Nov
(51) |
Dec
(15) |
| 2013 |
Jan
(28) |
Feb
(16) |
Mar
(25) |
Apr
(38) |
May
(23) |
Jun
(39) |
Jul
(42) |
Aug
(19) |
Sep
(41) |
Oct
(31) |
Nov
(18) |
Dec
(18) |
| 2014 |
Jan
(17) |
Feb
(19) |
Mar
(39) |
Apr
(16) |
May
(10) |
Jun
(13) |
Jul
(17) |
Aug
(13) |
Sep
(8) |
Oct
(53) |
Nov
(23) |
Dec
(7) |
| 2015 |
Jan
(35) |
Feb
(13) |
Mar
(14) |
Apr
(56) |
May
(8) |
Jun
(18) |
Jul
(26) |
Aug
(33) |
Sep
(40) |
Oct
(37) |
Nov
(24) |
Dec
(20) |
| 2016 |
Jan
(38) |
Feb
(20) |
Mar
(25) |
Apr
(14) |
May
(6) |
Jun
(36) |
Jul
(27) |
Aug
(19) |
Sep
(36) |
Oct
(24) |
Nov
(15) |
Dec
(16) |
| 2017 |
Jan
(8) |
Feb
(13) |
Mar
(17) |
Apr
(20) |
May
(28) |
Jun
(10) |
Jul
(20) |
Aug
(3) |
Sep
(18) |
Oct
(8) |
Nov
|
Dec
(5) |
| 2018 |
Jan
(15) |
Feb
(9) |
Mar
(12) |
Apr
(7) |
May
(123) |
Jun
(41) |
Jul
|
Aug
(14) |
Sep
|
Oct
(15) |
Nov
|
Dec
(7) |
| 2019 |
Jan
(2) |
Feb
(9) |
Mar
(2) |
Apr
(9) |
May
|
Jun
|
Jul
(2) |
Aug
|
Sep
(6) |
Oct
(1) |
Nov
(12) |
Dec
(2) |
| 2020 |
Jan
(2) |
Feb
|
Mar
|
Apr
(3) |
May
|
Jun
(4) |
Jul
(4) |
Aug
(1) |
Sep
(18) |
Oct
(2) |
Nov
|
Dec
|
| 2021 |
Jan
|
Feb
(3) |
Mar
|
Apr
|
May
|
Jun
|
Jul
(6) |
Aug
|
Sep
(5) |
Oct
(5) |
Nov
(3) |
Dec
|
| 2022 |
Jan
|
Feb
|
Mar
(3) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Roger S. <rog...@un...> - 2011-05-30 12:43:14
|
mfsmlpromote - promote av metalogger to metaserver
mfsmqstatus - shows a metaserver and loggers as a quorum and it's
status
mfsmsreelect - reelect metaserver (between metaserver and loggers)
I know its new functionality, especially for HA around metaserver, but
it is extremely important.
On 5/30/11 10:09 , Michal Borychowski wrote:
> Hi!
>
> You often ask „how to do this” / “why can’t I do this in command line”
> and every time we answer that sometime we’ll prepare a set of tools
> called “mastertools” where you could do lots of useful tasks. And this
> moment is slowly coming up :)
>
> We have thought of some tools but for sure you can give us more suggestions:
>
> - mfschunkinfo chunkid [, chunkid ...]
>
> - mfsinodeinfo inode [, inode ...]
>
> - mfsinfo or mfsstats(?) – returning how many files, folders, missing
> chunks, etc. there are
>
> - mfscsinfo– returning list of connected chunkservers
>
> - mfsmlinfo- returning list of connected metaloggerów
>
> - mfsclinfo- returning list of connected clients
>
> - mfshdinfo- returning list of connected hard drives
>
> Do you think of any other useful tools which could be indluded in the
> “mastertools” set? Please share your thoughts with us.
>
> Thanks!
>
> Best regards
>
> Michał Borychowski
>
> MooseFS Support Manager
>
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Gemius S.A.
>
> ul. Wołoska 7, 02-672 Warszawa
>
> Budynek MARS, klatka D
>
> Tel.: +4822 874-41-00
>
> Fax : +4822 874-41-01
>
>
>
> ------------------------------------------------------------------------------
> vRanger cuts backup time in half-while increasing security.
> With the market-leading solution for virtual backup and recovery,
> you get blazing-fast, flexible, and affordable data protection.
> Download your free trial now.
> http://p.sf.net/sfu/quest-d2dcopy1
>
>
>
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
|
|
From: Papp T. <to...@ma...> - 2011-05-30 10:25:16
|
On 05/30/2011 11:28 AM, Michal Borychowski wrote:
> length: 72TiB
> > size: 73TiB
> > realsize: 73TiB
Do you mean, this should be read as:
length: 7.2TiB
size: 7.3TiB
realsize: 7.3TiB
?
Thanks,
tamas
|
|
From: Michal B. <mic...@ge...> - 2011-05-30 09:29:02
|
On 05/30/2011 11:13 AM, Michal Borychowski wrote: > Hi! > > Just start your chunkservers one by one in such a situation. But this > happens very rarely. In this case this is not the same situation. I have only one chunkserver which the same as the master server. It starts this behaviour after a day or two days uptime. [MB] We'll look into this again > I don't see much differences in the sizes - what do you mean exactly? > /data/backup: > inodes: 33Mi > directories: 4.2Mi > files: 29Mi > chunks: 29Mi > length: 72TiB > size: 73TiB > realsize: 73TiB > > > /dev/sda6 10T 7.3T 2.8T 73% /mnt/mfschunk1 > mfsmaster:9421 10T 7.3T 2.8T 73% /data/backup msdirinfo shows the volume size 73T while df shows the real one, which is 7.3T, or do I misunderstand something? [MB] Too much 7(.)3 ;) Regards Michal |
|
From: Papp T. <to...@ma...> - 2011-05-30 09:23:49
|
On 05/30/2011 11:13 AM, Michal Borychowski wrote: > Hi! > > Just start your chunkservers one by one in such a situation. But this > happens very rarely. In this case this is not the same situation. I have only one chunkserver which the same as the master server. It starts this behaviour after a day or two days uptime. > I don't see much differences in the sizes - what do you mean exactly? > /data/backup: > inodes: 33Mi > directories: 4.2Mi > files: 29Mi > chunks: 29Mi > length: 72TiB > size: 73TiB > realsize: 73TiB > > > /dev/sda6 10T 7.3T 2.8T 73% /mnt/mfschunk1 > mfsmaster:9421 10T 7.3T 2.8T 73% /data/backup msdirinfo shows the volume size 73T while df shows the real one, which is 7.3T, or do I misunderstand something? Thanks, tamas |
|
From: R.C. <mil...@gm...> - 2011-05-30 09:18:12
|
A (maybe) weird idea: mfsmlinfo --fullstate : Analize and report state, age and useability of metadata on any metalogger connected (very useful just until a redundant master won't be available) JM2C Raf ----- Original Message ----- From: Michal Borychowski To: moo...@li... Sent: Monday, May 30, 2011 10:09 AM Subject: [Moosefs-users] We need your help! Mastertools brainstorming Hi! You often ask "how to do this" / "why can't I do this in command line" and every time we answer that sometime we'll prepare a set of tools called "mastertools" where you could do lots of useful tasks. And this moment is slowly coming up :) We have thought of some tools but for sure you can give us more suggestions: - mfschunkinfo chunkid [, chunkid ...] - mfsinodeinfo inode [, inode ...] - mfsinfo or mfsstats (?) - returning how many files, folders, missing chunks, etc. there are - mfscsinfo - returning list of connected chunkservers - mfsmlinfo - returning list of connected metaloggerów - mfsclinfo - returning list of connected clients - mfshdinfo - returning list of connected hard drives Do you think of any other useful tools which could be indluded in the "mastertools" set? Please share your thoughts with us. Thanks! Best regards Michał Borychowski MooseFS Support Manager _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Gemius S.A. ul. Wołoska 7, 02-672 Warszawa Budynek MARS, klatka D Tel.: +4822 874-41-00 Fax : +4822 874-41-01 ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ vRanger cuts backup time in half-while increasing security. With the market-leading solution for virtual backup and recovery, you get blazing-fast, flexible, and affordable data protection. Download your free trial now. http://p.sf.net/sfu/quest-d2dcopy1 ------------------------------------------------------------------------------ _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Papp T. <to...@ma...> - 2011-05-30 09:08:42
|
On 05/30/2011 09:22 AM, Michal Borychowski wrote: > Hi! > > It looks like a problem we already know about - the master can be stuck when > a group of chunkservers disconnects what causes big data flow upon their > reconnection which leads to network timeouts what again causes chunkservers > disconnections... We have some ideas to improve this behaviour. hi! I'm sad to hear the reason, but good to hear you know the problem. My server swapping almost 5GB, so I decreased swappines but didn't help. FYI, I don't know, how much it matters: /data/backup: inodes: 33Mi directories: 4.2Mi files: 29Mi chunks: 29Mi length: 72TiB size: 73TiB realsize: 73TiB /dev/sda6 10T 7.3T 2.8T 73% /mnt/mfschunk1 mfsmaster:9421 10T 7.3T 2.8T 73% /data/backup So it counts wrong size and real size because of hardlinks. Thank for your help and your work, tamas |
|
From: Florent B. <fl...@co...> - 2011-05-30 09:05:23
|
Hi, Thank you for your answer. The question is not really what would be better in IPv6, but the fact is that IPv4 *will* stop one day, and maybe soon. I know MooseFS is built to run in a LAN, so we are free to use IPv4, and not concerned about the shortage of IPv4 in the world. But in my mind, all network will soon be IPv6 and no more IPv4. So it could be good to have a product like MooseFS compatible with that. (in my mind, it's not very difficult to maintain IPv4/IPv6 compatibility in the source code, but I haven't looked at it yet...). By the way, I would like to know something. MooseFS is a free and open-source project, but I do not see where can people contribute to it ? There is no SVN, GIT, ... ? What is the procedure ? Le 30/05/2011 09:25, Michal Borychowski a écrit : > > Hi Florent! > > No, IPv6 is not supported by MooseFS. We try to imagine possible > advantages of introducing IPv6. Could you give us some tips what would > be better on IPv6 over IPv4? > > Kind regards > > Michał Borychowski > > MooseFS Support Manager > > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > Gemius S.A. > > ul. Wołoska 7, 02-672 Warszawa > > Budynek MARS, klatka D > > Tel.: +4822 874-41-00 > > Fax : +4822 874-41-01 > > *From:*Florent Bautista [mailto:fl...@co...] > *Sent:* Tuesday, May 24, 2011 10:05 AM > *To:* moo...@li... > *Subject:* [Moosefs-users] IPv6 ? > > Hi everyone, > > First of all, a big thank to Gemius team for their work on MooseFS, > thank you! > > Has anyone succeeded to run MooseFS using IPv6 network ? I'm trying > but it seems not working ... > > It could be a great update if it is not yet compatible. > > By the way, could we have an idea of when 1.7 version could be > released ? (quotas management is really important :) ) > > Thank you a lot. > > -- > > > Florent Bautista > > ------------------------------------------------------------------------ > > Ce message et ses éventuelles pièces jointes sont personnels, > confidentiels et à l'usage exclusif de leur destinataire. > Si vous n'êtes pas la personne à laquelle ce message est destiné, > veuillez noter que vous avez reçu ce courriel par erreur et qu'il vous > est strictement interdit d'utiliser, de diffuser, de transférer, > d'imprimer ou de copier ce message. > > This e-mail and any attachments hereto are strictly personal, > confidential and intended solely for the addressee. > If you are not the intended recipient, be advised that you have > received this email in error and that any use, dissemination, > forwarding, printing, or copying of this message is strictly prohibited. > > ------------------------------------------------------------------------ > > 30440 Saint Laurent le Minier > France > > *Compagnie pour des Prestations Internet* > > Téléphone : +33 (0)467 73 89 48 > Télécopie : + 33 (0)9 59 48 06 27 > > Courriel : Fl...@Co... <mailto:fl...@co...> > > ------------------------------------------------------------------------ -- Florent Bautista ------------------------------------------------------------------------ Ce message et ses éventuelles pièces jointes sont personnels, confidentiels et à l'usage exclusif de leur destinataire. Si vous n'êtes pas la personne à laquelle ce message est destiné, veuillez noter que vous avez reçu ce courriel par erreur et qu'il vous est strictement interdit d'utiliser, de diffuser, de transférer, d'imprimer ou de copier ce message. This e-mail and any attachments hereto are strictly personal, confidential and intended solely for the addressee. If you are not the intended recipient, be advised that you have received this email in error and that any use, dissemination, forwarding, printing, or copying of this message is strictly prohibited. ------------------------------------------------------------------------ 30440 Saint Laurent le Minier France *Compagnie pour des Prestations Internet* Téléphone : +33 (0)467 73 89 48 Télécopie : + 33 (0)9 59 48 06 27 Courriel : Fl...@Co... <mailto:fl...@co...> ------------------------------------------------------------------------ |
|
From: Michal B. <mic...@ge...> - 2011-05-30 08:09:42
|
Hi! You often ask "how to do this" / "why can't I do this in command line" and every time we answer that sometime we'll prepare a set of tools called "mastertools" where you could do lots of useful tasks. And this moment is slowly coming up :) We have thought of some tools but for sure you can give us more suggestions: - mfschunkinfo chunkid [, chunkid ...] - mfsinodeinfo inode [, inode ...] - mfsinfo or mfsstats (?) - returning how many files, folders, missing chunks, etc. there are - mfscsinfo - returning list of connected chunkservers - mfsmlinfo - returning list of connected metaloggerów - mfsclinfo - returning list of connected clients - mfshdinfo - returning list of connected hard drives Do you think of any other useful tools which could be indluded in the "mastertools" set? Please share your thoughts with us. Thanks! Best regards Michał Borychowski MooseFS Support Manager _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Gemius S.A. ul. Wołoska 7, 02-672 Warszawa Budynek MARS, klatka D Tel.: +4822 874-41-00 Fax : +4822 874-41-01 |
|
From: Michal B. <mic...@ge...> - 2011-05-30 07:32:06
|
Hi Tom! You'll see this information (what file or files are missing chunks) in CGI monitor after several hours. Kind regards Michał Borychowski MooseFS Support Manager _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Gemius S.A. ul. Wołoska 7, 02-672 Warszawa Budynek MARS, klatka D Tel.: +4822 874-41-00 Fax : +4822 874-41-01 -----Original Message----- From: Tom Eastman [mailto:tom...@ot...] Sent: Tuesday, May 24, 2011 3:54 AM To: moo...@li... Subject: [Moosefs-users] Finding out what file is corrupt Hey guys, I've just set up my first test cluster for trying out MooseFS, and when I restarted one of my chunkservers I ended up with one chunk in the 'red' (zero copies). How do I actually find out what file or files are missing chunks? I can see where the chunks are for each file using 'mfsfileinfo', but how can I get a quick summary of what the problem files are? Is it the 'filesystem check info' at the bottom of the cgi page? Mine still just says 'no data'. Is there a way to trigger a 'file system check' or something like that? Thanks! Tom ---------------------------------------------------------------------------- -- vRanger cuts backup time in half-while increasing security. With the market-leading solution for virtual backup and recovery, you get blazing-fast, flexible, and affordable data protection. Download your free trial now. http://p.sf.net/sfu/quest-d2dcopy1 _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Michal B. <mic...@ge...> - 2011-05-30 07:22:56
|
Hi! It looks like a problem we already know about - the master can be stuck when a group of chunkservers disconnects what causes big data flow upon their reconnection which leads to network timeouts what again causes chunkservers disconnections... We have some ideas to improve this behaviour. Kind regards Michał Borychowski MooseFS Support Manager _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Gemius S.A. ul. Wołoska 7, 02-672 Warszawa Budynek MARS, klatka D Tel.: +4822 874-41-00 Fax : +4822 874-41-01 -----Original Message----- From: Papp Tamas [mailto:to...@ma...] Sent: Saturday, May 28, 2011 3:13 PM To: moo...@li... Subject: Re: [Moosefs-users] timeout On 05/27/2011 10:19 AM, Papp Tamas wrote: > hi! > > Sometimes there is an error on our mini cluster. > Still Ubuntu Natty with recompiled moosefs from ppa. > > May 27 08:15:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739308400640 (7207.79 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:15:00 backup1 mfsmaster[12869]: total: usedspace: > 7739308400640 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:15:02 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B1/chunk_0000000000005EB1_00000001.mfs > May 27 08:15:12 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B2/chunk_0000000000005EB2_00000001.mfs > May 27 08:15:23 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B3/chunk_0000000000005EB3_00000001.mfs > May 27 08:15:28 backup1 mfsmount[12917]: master: tcp recv error: > ETIMEDOUT (Operation timed out) (1) > May 27 08:15:29 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:15:32 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:15:33 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B4/chunk_0000000000005EB4_00000001.mfs > May 27 08:15:35 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:15:44 backup1 mfsmount[12917]: last message repeated 3 times > May 27 08:15:44 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B5/chunk_0000000000005EB5_00000001.mfs > May 27 08:15:47 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:15:55 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:15:55 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B6/chunk_0000000000005EB6_00000001.mfs > May 27 08:15:56 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:15:59 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:16:00 backup1 mfschunkserver[2556]: connecting ... > May 27 08:16:00 backup1 mfschunkserver[2556]: connected to Master > May 27 08:16:02 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:17:05 backup1 mfsmount[12917]: last message repeated 21 times > May 27 08:17:15 backup1 mfsmount[12917]: last message repeated 3 times > May 27 08:17:15 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:17:15 backup1 mfsmaster[12869]: connection with > CS(192.168.3.21) has been closed by peer > May 27 08:17:15 backup1 mfsmaster[12869]: chunkserver disconnected - ip: > 192.168.3.21, port: 9422, usedspace: 7739308400640 (7207.79 GiB), > totalspace: 10944744390656 (10193.09 GiB) > May 27 08:17:16 backup1 mfsmaster[12869]: connection with > ML(192.168.3.13) has been closed by peer > May 27 08:17:16 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:17:17 backup1 mfsmaster[12869]: last message repeated 7 times > May 27 08:17:17 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:17:17 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:17:19 backup1 mfsmaster[12869]: last message repeated 28 times > May 27 08:17:19 backup1 mfsmount[12917]: registered to master > May 27 08:18:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:18:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:18:16 backup1 mfsmaster[12869]: chunkserver disconnected - ip: > 192.168.3.21, port: 0, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB) > May 27 08:19:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:19:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:19:14 backup1 mfsmount[12917]: file: 7340122, index: 0 - > fs_writechunk returns status 8 > May 27 08:20:00 backup1 mfsmount[12917]: last message repeated 17 times > May 27 08:20:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:20:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:20:05 backup1 mfsmount[12917]: file: 7340122, index: 0 - > fs_writechunk returns status 8 > May 27 08:21:00 backup1 mfsmount[12917]: last message repeated 7 times > May 27 08:21:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:21:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:21:02 backup1 mfsmount[12917]: file: 7340122, index: 0 - > fs_writechunk returns status 8 > May 27 08:21:29 backup1 mfsmount[12917]: last message repeated 3 times > May 27 08:21:29 backup1 mfsmount[12917]: error writing file number > 7340122: EIO (Input/output error) > May 27 08:21:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: > 4816320, version: 1 - there are no valid copies > May 27 08:21:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't > connect to proper chunkserver (try counter: 1) > May 27 08:22:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:22:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:22:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: > 4816320, version: 1 - there are no valid copies > May 27 08:22:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't > connect to proper chunkserver (try counter: 8) > May 27 08:23:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:23:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:23:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: > 4816320, version: 1 - there are no valid copies > May 27 08:23:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't > connect to proper chunkserver (try counter: 15) > May 27 08:24:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:24:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:24:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: > 4816320, version: 1 - there are no valid copies > May 27 08:24:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't > connect to proper chunkserver (try counter: 22) > May 27 08:25:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:25:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:25:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: > 4816320, version: 1 - there are no valid copies > May 27 08:25:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't > connect to proper chunkserver (try counter: 29) > May 27 08:26:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:26:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:26:10 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B7/chunk_0000000000005EB7_00000001.mfs > May 27 08:26:10 backup1 mfschunkserver[2556]: connection reset by Master > May 27 08:26:20 backup1 mfschunkserver[2556]: connecting ... > May 27 08:26:20 backup1 mfschunkserver[2556]: connected to Master > May 27 08:26:21 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B8/chunk_0000000000005EB8_00000001.mfs > May 27 08:26:21 backup1 mfsmaster[12869]: chunkserver register begin > (packet version: 5) - ip: 192.168.3.21, port: 9422 > May 27 08:26:28 backup1 mfsmaster[12869]: chunkserver register end > (packet version: 5) - ip: 192.168.3.21, port: 9422, usedspace: > 7739308400640 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB) > May 27 08:26:31 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B9/chunk_0000000000005EB9_00000001.mfs > May 27 08:26:41 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BA/chunk_0000000000005EBA_00000001.mfs > May 27 08:26:42 backup1 mfsmount[12917]: file: 7445943, index: 0, chunk: > 4816323, version: 1 - writeworker: connection with (C0A80315:9422) was > timed out (unfinished writes: 2; try counter: 1) > May 27 08:26:51 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:26:51 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BB/chunk_0000000000005EBB_00000001.mfs > May 27 08:27:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:27:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739309965312 (7207.79 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:27:00 backup1 mfsmaster[12869]: total: usedspace: > 7739309965312 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:27:01 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BC/chunk_0000000000005EBC_00000001.mfs > May 27 08:27:11 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BD/chunk_0000000000005EBD_00000001.mfs > May 27 08:27:21 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BE/chunk_0000000000005EBE_00000001.mfs > May 27 08:27:31 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BF/chunk_0000000000005EBF_00000001.mfs > May 27 08:27:41 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C0/chunk_0000000000005EC0_00000001.mfs > May 27 08:27:51 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C1/chunk_0000000000005EC1_00000001.mfs > May 27 08:28:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:28:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739310948352 (7207.79 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:28:00 backup1 mfsmaster[12869]: total: usedspace: > 7739310948352 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:28:01 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C3/chunk_0000000000005EC3_00000001.mfs > May 27 08:28:11 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C4/chunk_0000000000005EC4_00000001.mfs > May 27 08:28:22 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C5/chunk_0000000000005EC5_00000001.mfs > May 27 08:28:32 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C7/chunk_0000000000005EC7_00000001.mfs > May 27 08:28:42 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C8/chunk_0000000000005EC8_00000001.mfs > May 27 08:28:52 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C9/chunk_0000000000005EC9_00000001.mfs > May 27 08:29:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739313045504 (7207.80 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:29:00 backup1 mfsmaster[12869]: total: usedspace: > 7739313045504 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:29:02 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/CA/chunk_0000000000005ECA_00000001.mfs > May 27 08:29:12 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/CB/chunk_0000000000005ECB_00000001.mfs > May 27 08:29:22 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/CC/chunk_0000000000005ECC_00000001.mfs > May 27 08:29:26 backup1 mfsmount[12917]: master: tcp recv error: > ETIMEDOUT (Operation timed out) (1) > May 27 08:29:27 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:29:30 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:29:32 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/CD/chunk_0000000000005ECD_00000001.mfs > May 27 08:29:32 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:29:32 backup1 mfsmaster[12869]: last message repeated 2 times > May 27 08:29:32 backup1 mfsmount[12917]: registered to master > May 27 08:29:42 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/CE/chunk_0000000000005ECE_00000001.mfs > May 27 08:29:52 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D0/chunk_0000000000005ED0_00000001.mfs > May 27 08:30:02 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D1/chunk_0000000000005ED1_00000001.mfs > May 27 08:30:05 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: > 4816324, version: 1 - writeworker: connection with (C0A80315:9422) was > timed out (unfinished writes: 1; try counter: 1) > May 27 08:30:12 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D2/chunk_0000000000005ED2_00000001.mfs > May 27 08:30:22 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D3/chunk_0000000000005ED3_00000001.mfs > May 27 08:30:33 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D4/chunk_0000000000005ED4_00000001.mfs > May 27 08:30:43 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D5/chunk_0000000000005ED5_00000001.mfs > May 27 08:30:53 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D6/chunk_0000000000005ED6_00000001.mfs > May 27 08:31:03 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D7/chunk_0000000000005ED7_00000001.mfs > May 27 08:31:13 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D8/chunk_0000000000005ED8_00000001.mfs > May 27 08:31:23 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D9/chunk_0000000000005ED9_00000001.mfs > May 27 08:31:33 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/DA/chunk_0000000000005EDA_00000001.mfs > May 27 08:31:43 backup1 mfsmount[12917]: master: tcp recv error: > ETIMEDOUT (Operation timed out) (1) > May 27 08:31:43 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/DB/chunk_0000000000005EDB_00000001.mfs > May 27 08:31:44 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:31:53 backup1 mfsmount[12917]: last message repeated 3 times > May 27 08:31:53 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/DD/chunk_0000000000005EDD_00000001.mfs > May 27 08:31:56 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:32:03 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:32:03 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/DE/chunk_0000000000005EDE_00000001.mfs > May 27 08:32:05 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:32:14 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:32:14 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/DF/chunk_0000000000005EDF_00000001.mfs > May 27 08:32:14 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:32:24 backup1 mfsmount[12917]: last message repeated 3 times > May 27 08:32:24 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E0/chunk_0000000000005EE0_00000001.mfs > May 27 08:32:25 backup1 mfschunkserver[2556]: connecting ... > May 27 08:32:25 backup1 mfschunkserver[2556]: connected to Master > May 27 08:32:26 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:33:29 backup1 mfsmount[12917]: last message repeated 21 times > May 27 08:34:25 backup1 mfsmount[12917]: last message repeated 18 times > May 27 08:34:25 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:34:25 backup1 mfsmaster[12869]: connection with > CS(192.168.3.21) has been closed by peer > May 27 08:34:25 backup1 mfsmaster[12869]: chunkserver disconnected - ip: > 192.168.3.21, port: 9422, usedspace: 7739311943680 (7207.80 GiB), > totalspace: 10944744390656 (10193.09 GiB) > May 27 08:34:26 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:34:33 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:34:33 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E1/chunk_0000000000005EE1_00000001.mfs > May 27 08:34:34 backup1 mfsmaster[12869]: connection with > ML(192.168.3.13) has been closed by peer > May 27 08:34:34 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:34:34 backup1 mfsmaster[12869]: chunkserver register begin > (packet version: 5) - ip: 192.168.3.21, port: 9422 > May 27 08:34:34 backup1 mfsmaster[12869]: connection with > ML(192.168.3.13) has been closed by peer > May 27 08:34:34 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:34:34 backup1 mfsmaster[12869]: connection with > CS(192.168.3.21) has been closed by peer > May 27 08:34:34 backup1 mfsmaster[12869]: chunkserver disconnected - ip: > 192.168.3.21, port: 9422, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB) > May 27 08:34:35 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:34:35 backup1 mfsmaster[12869]: last message repeated 2 times > May 27 08:34:35 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:34:35 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:34:37 backup1 mfsmaster[12869]: last message repeated 52 times > May 27 08:34:37 backup1 mfsmount[12917]: registered to master > May 27 08:34:39 backup1 mfsmount[12917]: file: 7445943, index: 0 - > fs_writechunk returns status 8 > May 27 08:34:40 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:34:40 backup1 mfschunkserver[2556]: connecting ... > May 27 08:34:40 backup1 mfschunkserver[2556]: connected to Master > May 27 08:34:40 backup1 mfsmount[12917]: file: 7445943, index: 0 - > fs_writechunk returns status 8 > May 27 08:34:41 backup1 mfsmount[12917]: file: 7445943, index: 0 - > fs_writechunk returns status 8 > May 27 08:34:41 backup1 mfsmaster[12869]: chunkserver register begin > (packet version: 5) - ip: 192.168.3.21, port: 9422 > May 27 08:34:42 backup1 mfsmaster[12869]: chunkserver register end > (packet version: 5) - ip: 192.168.3.21, port: 9422, usedspace: > 7739311943680 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB) > May 27 08:34:43 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E2/chunk_0000000000005EE2_00000001.mfs > May 27 08:34:53 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E3/chunk_0000000000005EE3_00000001.mfs > May 27 08:35:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:35:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739314479104 (7207.80 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:35:00 backup1 mfsmaster[12869]: total: usedspace: > 7739314479104 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:35:03 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E4/chunk_0000000000005EE4_00000001.mfs > May 27 08:35:14 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E5/chunk_0000000000005EE5_00000001.mfs > May 27 08:35:24 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E6/chunk_0000000000005EE6_00000001.mfs > May 27 08:35:34 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E7/chunk_0000000000005EE7_00000001.mfs > May 27 08:35:44 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E8/chunk_0000000000005EE8_00000001.mfs > May 27 08:35:54 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E9/chunk_0000000000005EE9_00000001.mfs > May 27 08:36:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:36:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739314479104 (7207.80 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:36:00 backup1 mfsmaster[12869]: total: usedspace: > 7739314479104 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:36:04 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/EA/chunk_0000000000005EEA_00000001.mfs > May 27 08:36:14 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/EB/chunk_0000000000005EEB_00000001.mfs > May 27 08:36:24 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/EC/chunk_0000000000005EEC_00000001.mfs > May 27 08:36:34 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/ED/chunk_0000000000005EED_00000001.mfs > May 27 08:36:45 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/EE/chunk_0000000000005EEE_00000001.mfs > May 27 08:36:55 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/EF/chunk_0000000000005EEF_00000001.mfs > May 27 08:37:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:37:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739314479104 (7207.80 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:37:00 backup1 mfsmaster[12869]: total: usedspace: > 7739314479104 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > > > Can you help me what to do with it? > Is this fuse, moosefs, kernel problem or something else? hi! I've just realized an other error. $ dirvish --vault cluster/Projects dirvish cluster/Projects:default fatal error: filesystem full dirvish cluster/Projects:default fatal error (12) -- filesystem full cluster/Projects:default post-server failed (1) log: May 28 14:58:04 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/70/chunk_0000000000008470_00000001.mfs May 28 14:58:14 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/71/chunk_0000000000008471_00000001.mfs May 28 14:58:24 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/72/chunk_0000000000008472_00000001.mfs May 28 14:58:34 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/73/chunk_0000000000008473_00000001.mfs May 28 14:58:44 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/74/chunk_0000000000008474_00000001.mfs May 28 14:58:55 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/75/chunk_0000000000008475_00000001.mfs May 28 14:59:00 backup1 mfsmaster[12869]: chunkservers status: May 28 14:59:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, port: 9422): usedspace: 7734218166272 (7203.05 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.67% May 28 14:59:00 backup1 mfsmaster[12869]: total: usedspace: 7734218166272 (7203.05 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.67% May 28 14:59:05 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/76/chunk_0000000000008476_00000001.mfs May 28 14:59:15 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/77/chunk_0000000000008477_00000001.mfs May 28 14:59:24 backup1 mfsmount[12917]: master: tcp recv error: ETIMEDOUT (Operation timed out) (1) May 28 14:59:25 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 14:59:25 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/78/chunk_0000000000008478_00000001.mfs May 28 14:59:28 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 14:59:35 backup1 mfsmount[12917]: last message repeated 2 times May 28 14:59:35 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/79/chunk_0000000000008479_00000001.mfs May 28 14:59:37 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 14:59:45 backup1 mfsmount[12917]: last message repeated 2 times May 28 14:59:45 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7A/chunk_000000000000847A_00000001.mfs May 28 14:59:46 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 14:59:55 backup1 mfsmount[12917]: last message repeated 3 times May 28 14:59:55 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7B/chunk_000000000000847B_00000001.mfs May 28 14:59:58 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 15:00:06 backup1 mfsmount[12917]: last message repeated 2 times May 28 15:00:06 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7C/chunk_000000000000847C_00000001.mfs May 28 15:00:07 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 15:00:15 backup1 mfsmount[12917]: last message repeated 2 times May 28 15:00:15 backup1 mfschunkserver[2556]: connecting ... May 28 15:00:15 backup1 mfschunkserver[2556]: connected to Master May 28 15:00:16 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 15:01:19 backup1 mfsmount[12917]: last message repeated 21 times May 28 15:02:19 backup1 mfsmount[12917]: last message repeated 19 times May 28 15:02:19 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 15:02:19 backup1 mfsmaster[12869]: connection with CS(192.168.3.21) has been closed by peer May 28 15:02:19 backup1 mfsmaster[12869]: chunkserver disconnected - ip: 192.168.3.21, port: 9422, usedspace: 7734399709184 (7203.22 GiB), totalspace: 10944744390656 (10193.09 GiB) May 28 15:02:19 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 15:02:22 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 15:02:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.20) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.13) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.13) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.20) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.13) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 15:02:24 backup1 mfsmaster[12869]: last message repeated 55 times May 28 15:02:24 backup1 mfsmount[12917]: registered to master May 28 15:02:25 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: 4816943, version: 1 - there are no valid copies May 28 15:02:25 backup1 mfsmount[12917]: file: 122301, index: 0 - can't connect to proper chunkserver (try counter: 1) May 28 15:02:26 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:26 backup1 mfsmount[12917]: last message repeated 2 times May 28 15:02:26 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:02:27 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:32 backup1 mfsmount[12917]: last message repeated 3 times May 28 15:02:32 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:02:33 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:38 backup1 mfsmount[12917]: last message repeated 2 times May 28 15:02:38 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:02:41 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:44 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:44 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:02:48 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:51 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:02:52 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:56 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:58 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:01 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:05 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:06 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:11 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:13 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:17 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:20 backup1 mfsmaster[12869]: chunkserver disconnected - ip: 192.168.3.21, port: 0, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB) May 28 15:03:22 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:23 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:25 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: 4816943, version: 1 - there are no valid copies May 28 15:03:25 backup1 mfsmount[12917]: file: 122301, index: 0 - can't connect to proper chunkserver (try counter: 8) May 28 15:03:29 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:30 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:36 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:39 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:43 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:48 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:50 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:57 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:57 backup1 mfsmount[12917]: error writing file number 107222: EIO (Input/output error) May 28 15:03:58 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:04:00 backup1 mfsmaster[12869]: chunkservers status: May 28 15:04:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 28 15:04:06 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:04:25 backup1 mfsmount[12917]: last message repeated 2 times May 28 15:04:25 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: 4816943, version: 1 - there are no valid copies May 28 15:04:25 backup1 mfsmount[12917]: file: 122301, index: 0 - can't connect to proper chunkserver (try counter: 15) May 28 15:04:32 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:04:41 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:04:41 backup1 mfsmount[12917]: error writing file number 4691391: ENOSPC (No space left on device) May 28 15:04:42 backup1 mfsmount[12917]: file: 107221, index: 0 - fs_writechunk returns status 8 May 28 15:05:00 backup1 mfsmount[12917]: last message repeated 10 times May 28 15:05:00 backup1 mfsmaster[12869]: chunkservers status: May 28 15:05:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 28 15:05:00 backup1 mfsmount[12917]: file: 107221, index: 0 - fs_writechunk returns status 8 May 28 15:05:25 backup1 mfsmount[12917]: last message repeated 5 times May 28 15:05:25 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: 4816943, version: 1 - there are no valid copies May 28 15:05:25 backup1 mfsmount[12917]: file: 122301, index: 0 - can't connect to proper chunkserver (try counter: 22) May 28 15:05:27 backup1 mfsmount[12917]: file: 107221, index: 0 - fs_writechunk returns status 8 May 28 15:06:00 backup1 mfsmount[12917]: last message repeated 4 times May 28 15:06:00 backup1 mfsmaster[12869]: chunkservers status: May 28 15:06:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 28 15:06:01 backup1 mfsmount[12917]: file: 107221, index: 0 - fs_writechunk returns status 8 May 28 15:06:25 backup1 mfsmount[12917]: last message repeated 3 times May 28 15:06:25 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: 4816943, version: 1 - there are no valid copies May 28 15:06:25 backup1 mfsmount[12917]: file: 122301, index: 0 - can't connect to proper chunkserver (try counter: 29) May 28 15:06:25 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7D/chunk_000000000000847D_00000001.mfs May 28 15:06:25 backup1 mfschunkserver[2556]: connection reset by Master May 28 15:06:32 backup1 mfsmount[12917]: file: 107221, index: 0 - fs_writechunk returns status 8 May 28 15:06:35 backup1 mfschunkserver[2556]: connecting ... May 28 15:06:35 backup1 mfschunkserver[2556]: connected to Master May 28 15:06:36 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7E/chunk_000000000000847E_00000001.mfs May 28 15:06:36 backup1 mfsmaster[12869]: chunkserver register begin (packet version: 5) - ip: 192.168.3.21, port: 9422 May 28 15:06:38 backup1 mfsmaster[12869]: chunkserver register end (packet version: 5) - ip: 192.168.3.21, port: 9422, usedspace: 7734393311232 (7203.22 GiB), totalspace: 10944744390656 (10193.09 GiB) May 28 15:06:46 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7F/chunk_000000000000847F_00000001.mfs May 28 15:06:56 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/80/chunk_0000000000008480_00000001.mfs May 28 15:07:00 backup1 mfsmaster[12869]: chunkservers status: May 28 15:07:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, port: 9422): usedspace: 7734393442304 (7203.22 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.67% May 28 15:07:00 backup1 mfsmaster[12869]: total: usedspace: 7734393442304 (7203.22 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.67% Filesystem Size Used Avail Use% Mounted on /dev/sda2 19G 1.4G 17G 8% / none 3.9G 180K 3.9G 1% /dev none 4.0G 0 4.0G 0% /dev/shm none 4.0G 716K 4.0G 1% /var/run none 4.0G 0 4.0G 0% /var/lock /dev/sda6 10T 7.1T 3.0T 71% /mnt/mfschunk1 /dev/sda3 19G 12G 6.0G 66% /var /dev/sda4 4.6G 138M 4.3G 4% /tmp mfsmaster:9421 10T 7.1T 3.0T 71% /data/backup Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 1222992 64010 1158982 6% / none 1021194 762 1020432 1% /dev none 1023079 1 1023078 1% /dev/shm none 1023079 58 1023021 1% /var/run none 1023079 1 1023078 1% /var/lock /dev/sda6 2138062720 4778154 2133284566 1% /mnt/mfschunk1 /dev/sda3 1222992 3174 1219818 1% /var /dev/sda4 305216 13 305203 1% /tmp mfsmaster:9421 1031528383 30522363 1001006020 3% /data/backup So it's definetly not full. I don't understand, what's going on:/ Does somebody have any idea? Thank you, tamas ---------------------------------------------------------------------------- -- vRanger cuts backup time in half-while increasing security. With the market-leading solution for virtual backup and recovery, you get blazing-fast, flexible, and affordable data protection. Download your free trial now. http://p.sf.net/sfu/quest-d2dcopy1 _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Michal B. <mic...@ge...> - 2011-05-30 06:45:54
|
Hi Samuel! Please also have a look at this answer: http://www.moosefs.org/moosefs-faq.html#source_code Kind regards Michał Borychowski MooseFS Support Manager _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Gemius S.A. ul. Wołoska 7, 02-672 Warszawa Budynek MARS, klatka D Tel.: +4822 874-41-00 Fax : +4822 874-41-01 -----Original Message----- From: wk...@bn... [mailto:wk...@bn...] Sent: Monday, May 30, 2011 2:08 AM To: moo...@li... Subject: Re: [Moosefs-users] Small files real size On 5/29/11 7:05 AM, Samuel Hassine, Olympe Network wrote: > Hi all, > > I know that MooseFS chunks take more disk space than the realsize of the > files. But for us, it become very critical : > > root@on-003:~# mfsdirinfo -h /dns > /dns: > inodes: 15Mi > directories: 1.9Mi > files: 13Mi > chunks: 13Mi > length: 247GiB > size: 1.0TiB > realsize: 2.1TiB > > 1TB for 247GB of real files... and with a goal of 2, we are using 2TB... > You have 13 Million files each of which has a minimum 64K fixed block size in MooseFS (plus some overhead). The devs feel that is the optimum block for their purposes and are unlikely to change that. Here is a typical mount for one our dedicated customer imap servers. inodes: 1.2Mi directories: 17Ki files: 1.2Mi chunks: 1.2Mi length: 111GiB size: 183GiB realsize: 550GiB Note: We are in the process of increasing all our mount goals from 2 to 3 So taking the storage size increase hit is part of the tradeoff using MFS. Fortunately, MFS makes it easy to increase the pool size, either by adding chunkservers and/or adding drives to a chunkserver. In my part of the world 1TB drives can be had for US$59 and 2TB are usually below US$100 (though we don't quite trust those yet after having some horrific experiences a year or so ago), so we happily trade off getting an extra drive or so for the ease of maintenance and speed of the MFS cluster vs the other high availability solutions we have tried (DRBD, GFS2/OCFS clusters, Gluster, etc). -bill ---------------------------------------------------------------------------- -- vRanger cuts backup time in half-while increasing security. With the market-leading solution for virtual backup and recovery, you get blazing-fast, flexible, and affordable data protection. Download your free trial now. http://p.sf.net/sfu/quest-d2dcopy1 _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: <wk...@bn...> - 2011-05-30 00:08:21
|
On 5/29/11 7:05 AM, Samuel Hassine, Olympe Network wrote: > Hi all, > > I know that MooseFS chunks take more disk space than the realsize of the > files. But for us, it become very critical : > > root@on-003:~# mfsdirinfo -h /dns > /dns: > inodes: 15Mi > directories: 1.9Mi > files: 13Mi > chunks: 13Mi > length: 247GiB > size: 1.0TiB > realsize: 2.1TiB > > 1TB for 247GB of real files... and with a goal of 2, we are using 2TB... > You have 13 Million files each of which has a minimum 64K fixed block size in MooseFS (plus some overhead). The devs feel that is the optimum block for their purposes and are unlikely to change that. Here is a typical mount for one our dedicated customer imap servers. inodes: 1.2Mi directories: 17Ki files: 1.2Mi chunks: 1.2Mi length: 111GiB size: 183GiB realsize: 550GiB Note: We are in the process of increasing all our mount goals from 2 to 3 So taking the storage size increase hit is part of the tradeoff using MFS. Fortunately, MFS makes it easy to increase the pool size, either by adding chunkservers and/or adding drives to a chunkserver. In my part of the world 1TB drives can be had for US$59 and 2TB are usually below US$100 (though we don't quite trust those yet after having some horrific experiences a year or so ago), so we happily trade off getting an extra drive or so for the ease of maintenance and speed of the MFS cluster vs the other high availability solutions we have tried (DRBD, GFS2/OCFS clusters, Gluster, etc). -bill |
|
From: Samuel H. O. N. <sam...@ol...> - 2011-05-29 14:22:30
|
Hi all, I know that MooseFS chunks take more disk space than the realsize of the files. But for us, it become very critical : root@on-003:~# mfsdirinfo -h /dns /dns: inodes: 15Mi directories: 1.9Mi files: 13Mi chunks: 13Mi length: 247GiB size: 1.0TiB realsize: 2.1TiB 1TB for 247GB of real files... and with a goal of 2, we are using 2TB... Please, could you give us more information about the solutions you are planning to provide in the next version? Thanks for your help. Best regards. -- Samuel HASSINE Président Olympe Network - 31 avenue Sainte Victoire, 13100 Aix-en-Pce Tel. : +33(0)6.26.81.01.87 Site : www.olympe-network.com |
|
From: Steve <st...@bo...> - 2011-05-29 09:34:42
|
Yes it does get confusing with the 'reply to all' not being the default. Its a simple change at least to the listserv's iv'e used in the past. Same as me, then a home system. We don't get power cuts often here, last two caused by me! but ive survived a few and have no ups. Your ext4 should give some resilience over older system and is same as I use. So I guess we can assume your underlying filesystem is ok and you've just been unlucky with the timing of what moose was doing at the time of power loss ? You probably need to get your metadata to the devs on Monday they may help with a fix and may want to see if they can handle recovery better if its at all possible. Dual load sharing master servers maybe desirable to help reduce risks of data loss too. Otherwise ive had no problems with moose running it since before it became well know except that by my own cause - a brief try of early btrfs! -------Original Message------- From: Tuukka Luolamo Date: 29/05/2011 09:55:14 To: Steve; moo...@li... Subject: Re: [Moosefs-users] Problems after power failure Sorry forgot to send my response tot he group too. Tuukka On Sun, May 29, 2011 at 1:41 AM, Tuukka Luolamo <tlu...@gm...> wrote: > This is just a home setup so no UPS though I thought maybe I should > have one. However I assumed the meta logger mechanism should be enough > of a backup to recover in case of a failure, if this disk goes down > and takes me an hours to get back up it is not the end of the world =) > (Right now it has been down for a week) > > They run on ext4. > > Thanks, > > Tuukka > > > On Sun, May 29, 2011 at 1:18 AM, Steve <st...@bo...> wrote: >> >> I wonder why chunkservers arent by default capable of logging the data too >> as a built in function. >> >> >> >> If its mission critical I guess you should have had at least one of these on >> UPS. I cant preach as mine are not!! >> >> What filesystems are your running on ? >> >> >> >> >> >> >> >> >> >> -------Original Message------- >> >> >> >> From: Tuukka Luolamo >> >> Date: 29/05/2011 02:36:23 >> >> To: moo...@li... >> >> Subject: [Moosefs-users] Problems after power failure >> >> >> >> I had a power failure and both my master and meta logger went down >> >> simultaneously. >> >> >> >> When I turned them back on the master process failed to start, so I >> >> ran metarestore -a but got the following error: >> >> >> >> loading objects (files,directories,etc.) ... ok >> >> loading names ... loading edge: 7527,DSC01862.JPG->7554 error: child not >> found >> >> error >> >> can't read metadata from file: metadata.mfs.back >> >> >> >> So I went to the metalogger and got the same error. >> >> >> >> Now I am not sure what to try next. >> >> >> >> Any help would be appreciated. >> >> >> >> >> >> Tuukka >> >> >> >> ----------------------------------------------------------------------------- >> >> >> vRanger cuts backup time in half-while increasing security. >> >> With the market-leading solution for virtual backup and recovery, >> >> you get blazing-fast, flexible, and affordable data protection. >> >> Download your free trial now. >> >> http://p.sf.net/sfu/quest-d2dcopy1 >> >> _______________________________________________ >> >> moosefs-users mailing list >> >> moo...@li... >> >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> >> >> > |
|
From: Tuukka L. <tlu...@gm...> - 2011-05-29 08:55:16
|
Sorry forgot to send my response tot he group too. Tuukka On Sun, May 29, 2011 at 1:41 AM, Tuukka Luolamo <tlu...@gm...> wrote: > This is just a home setup so no UPS though I thought maybe I should > have one. However I assumed the meta logger mechanism should be enough > of a backup to recover in case of a failure, if this disk goes down > and takes me an hours to get back up it is not the end of the world =) > (Right now it has been down for a week) > > They run on ext4. > > Thanks, > > Tuukka > > > On Sun, May 29, 2011 at 1:18 AM, Steve <st...@bo...> wrote: >> >> I wonder why chunkservers arent by default capable of logging the data too >> as a built in function. >> >> >> >> If its mission critical I guess you should have had at least one of these on >> UPS. I cant preach as mine are not!! >> >> What filesystems are your running on ? >> >> >> >> >> >> >> >> >> >> -------Original Message------- >> >> >> >> From: Tuukka Luolamo >> >> Date: 29/05/2011 02:36:23 >> >> To: moo...@li... >> >> Subject: [Moosefs-users] Problems after power failure >> >> >> >> I had a power failure and both my master and meta logger went down >> >> simultaneously. >> >> >> >> When I turned them back on the master process failed to start, so I >> >> ran metarestore -a but got the following error: >> >> >> >> loading objects (files,directories,etc.) ... ok >> >> loading names ... loading edge: 7527,DSC01862.JPG->7554 error: child not >> found >> >> error >> >> can't read metadata from file: metadata.mfs.back >> >> >> >> So I went to the metalogger and got the same error. >> >> >> >> Now I am not sure what to try next. >> >> >> >> Any help would be appreciated. >> >> >> >> >> >> Tuukka >> >> >> >> ----------------------------------------------------------------------------- >> >> >> vRanger cuts backup time in half-while increasing security. >> >> With the market-leading solution for virtual backup and recovery, >> >> you get blazing-fast, flexible, and affordable data protection. >> >> Download your free trial now. >> >> http://p.sf.net/sfu/quest-d2dcopy1 >> >> _______________________________________________ >> >> moosefs-users mailing list >> >> moo...@li... >> >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> >> >> > |
|
From: Steve <st...@bo...> - 2011-05-29 08:18:53
|
I wonder why chunkservers arent by default capable of logging the data too as a built in function. If its mission critical I guess you should have had at least one of these on UPS. I cant preach as mine are not!! What filesystems are your running on ? -------Original Message------- From: Tuukka Luolamo Date: 29/05/2011 02:36:23 To: moo...@li... Subject: [Moosefs-users] Problems after power failure I had a power failure and both my master and meta logger went down simultaneously. When I turned them back on the master process failed to start, so I ran metarestore -a but got the following error: loading objects (files,directories,etc.) ... ok loading names ... loading edge: 7527,DSC01862.JPG->7554 error: child not found error can't read metadata from file: metadata.mfs.back So I went to the metalogger and got the same error. Now I am not sure what to try next. Any help would be appreciated. Tuukka ----------------------------------------------------------------------------- vRanger cuts backup time in half-while increasing security. With the market-leading solution for virtual backup and recovery, you get blazing-fast, flexible, and affordable data protection. Download your free trial now. http://p.sf.net/sfu/quest-d2dcopy1 _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Tuukka L. <tlu...@gm...> - 2011-05-29 01:35:43
|
I had a power failure and both my master and meta logger went down simultaneously. When I turned them back on the master process failed to start, so I ran metarestore -a but got the following error: loading objects (files,directories,etc.) ... ok loading names ... loading edge: 7527,DSC01862.JPG->7554 error: child not found error can't read metadata from file: metadata.mfs.back So I went to the metalogger and got the same error. Now I am not sure what to try next. Any help would be appreciated. Tuukka |
|
From: Papp T. <to...@ma...> - 2011-05-28 17:49:31
|
On 05/28/2011 03:12 PM, Papp Tamas wrote: > Does somebody have any idea? Again, more info on this: More info on this. The mfsmaster died totally, strace show nothing and the process is in state 'D': 12869 ? D< 199:42 /usr/sbin/mfsmaster Actually I'm removing files from trash. After the rm job the master node came back. log: May 28 19:30:00 backup1 mfsmaster[12869]: chunkservers status: May 28 19:30:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, port: 9422): usedspace: 7726711619584 (7196.06 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.60% May 28 19:30:00 backup1 mfsmaster[12869]: total: usedspace: 7726711619584 (7196.06 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.60% May 28 19:30:08 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/10/chunk_0000000000008A10_00000001.mfs May 28 19:30:10 backup1 mfsmount[3621]: master: tcp recv error: ETIMEDOUT (Operation timed out) (1) May 28 19:30:10 backup1 mfsmount[12917]: master: tcp recv error: ETIMEDOUT (Operation timed out) (1) May 28 19:30:11 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:11 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:14 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:14 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:17 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:17 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:18 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/11/chunk_0000000000008A11_00000001.mfs May 28 19:30:20 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:20 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:23 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:23 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:26 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:26 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:28 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/12/chunk_0000000000008A12_00000001.mfs May 28 19:30:29 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:29 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:32 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:32 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:35 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:35 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:38 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:38 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:39 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/13/chunk_0000000000008A13_00000001.mfs May 28 19:30:41 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:41 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:44 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:44 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:47 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:47 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:49 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/14/chunk_0000000000008A14_00000001.mfs May 28 19:30:41 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:41 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:44 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:44 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:47 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:47 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:49 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/14/chunk_0000000000008A14_00000001.mfs May 28 19:30:50 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:50 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:53 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:53 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:56 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:56 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:59 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/15/chunk_0000000000008A15_00000001.mfs May 28 19:30:59 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:30:59 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:02 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:02 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:05 backup1 mfschunkserver[2556]: connecting ... May 28 19:31:05 backup1 mfschunkserver[2556]: connected to Master May 28 19:31:05 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:05 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:08 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:08 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:11 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:11 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:14 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:14 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:17 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:17 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:20 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:20 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:23 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:23 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:26 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:26 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:29 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:29 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:32 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:32 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:35 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:35 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:38 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:38 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:41 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:41 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:44 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:44 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:47 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:47 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:50 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:50 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:53 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:53 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:56 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:56 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:59 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:31:59 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:02 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:02 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:05 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:05 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:08 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:08 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:11 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:11 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:14 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:14 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:15 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/16/chunk_0000000000008A16_00000001.mfs May 28 19:32:17 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:17 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:20 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:20 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:23 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:23 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:25 backup1 mfschunkserver[2556]: connecting ... May 28 19:32:25 backup1 mfschunkserver[2556]: connected to Master May 28 19:32:26 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/17/chunk_0000000000008A17_00000001.mfs May 28 19:32:26 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:26 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:29 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:29 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:32 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:32 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:35 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:35 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:36 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/18/chunk_0000000000008A18_00000001.mfs May 28 19:32:38 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:38 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:41 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:41 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:44 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:46 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/19/chunk_0000000000008A19_00000001.mfs May 28 19:32:47 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:53 backup1 mfsmount[3621]: last message repeated 2 times May 28 19:32:53 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:32:56 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/1A/chunk_0000000000008A1A_00000001.mfs May 28 19:32:59 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:33:00 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:33:05 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:33:06 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:33:06 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/1B/chunk_0000000000008A1B_00000001.mfs May 28 19:33:11 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:33:12 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:33:16 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/1C/chunk_0000000000008A1C_00000001.mfs May 28 19:33:17 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:33:18 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:33:18 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 19:33:18 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 19:33:18 backup1 mfsmaster[12869]: connection with CS(192.168.3.21) has been closed by peer May 28 19:33:18 backup1 mfsmaster[12869]: chunkserver disconnected - ip: 192.168.3.21, port: 9422, usedspace: 7726711480320 (7196.06 GiB), totalspace: 10944744390656 (10193.09 GiB) May 28 19:33:20 backup1 mfsmount[3621]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 19:33:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.20) has been closed by peer May 28 19:33:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.13) has been closed by peer May 28 19:33:23 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 19:33:23 backup1 mfsmaster[12869]: chunkserver register begin (packet version: 5) - ip: 192.168.3.21, port: 9422 May 28 19:33:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.13) has been closed by peer May 28 19:33:23 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 19:33:23 backup1 mfsmaster[12869]: chunk-server already connected !!! May 28 19:33:23 backup1 mfsmaster[12869]: connection with CS(192.168.3.21) has been closed by peer May 28 19:33:23 backup1 mfsmaster[12869]: connection with CS(192.168.3.21) has been closed by peer May 28 19:33:23 backup1 mfsmaster[12869]: chunkserver disconnected - ip: 192.168.3.21, port: 9422, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB) May 28 19:33:23 backup1 mfsmaster[12869]: chunkserver disconnected - ip: 192.168.3.21, port: 9422, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB) May 28 19:33:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.20) has been closed by peer May 28 19:33:23 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 19:33:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.13) has been closed by peer May 28 19:33:23 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 19:33:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.20) has been closed by peer May 28 19:33:23 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 19:33:23 backup1 mfsmaster[12869]: last message repeated 98 times May 28 19:33:23 backup1 mfsmount[12917]: registered to master May 28 19:33:23 backup1 mfsmount[3621]: registered to master May 28 19:33:23 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: 4817595, version: 1 - there are no valid copies May 28 19:33:23 backup1 mfsmount[12917]: file: 122301, index: 0 - can't connect to proper chunkserver (try counter: 1) May 28 19:33:24 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 19:33:25 backup1 mfschunkserver[2556]: connecting ... May 28 19:33:25 backup1 mfschunkserver[2556]: connected to Master May 28 19:33:25 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 19:33:25 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 19:33:26 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/1D/chunk_0000000000008A1D_00000001.mfs May 28 19:33:26 backup1 mfsmaster[12869]: chunkserver register begin (packet version: 5) - ip: 192.168.3.21, port: 9422 May 28 19:33:28 backup1 mfsmaster[12869]: chunkserver register end (packet version: 5) - ip: 192.168.3.21, port: 9422, usedspace: 7726711480320 (7196.06 GiB), totalspace: 10944744390656 (10193.09 GiB) May 28 19:33:28 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 19:33:28 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 19:33:36 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/1E/chunk_0000000000008A1E_00000001.mfs mfsmaster is not stopable again. After /etc/init.mfs-master stop I run it again but it's stucked, and master is again in state 'D'. tamas |
|
From: Papp T. <to...@ma...> - 2011-05-28 13:13:10
|
On 05/27/2011 10:19 AM, Papp Tamas wrote: > hi! > > Sometimes there is an error on our mini cluster. > Still Ubuntu Natty with recompiled moosefs from ppa. > > May 27 08:15:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739308400640 (7207.79 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:15:00 backup1 mfsmaster[12869]: total: usedspace: > 7739308400640 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:15:02 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B1/chunk_0000000000005EB1_00000001.mfs > May 27 08:15:12 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B2/chunk_0000000000005EB2_00000001.mfs > May 27 08:15:23 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B3/chunk_0000000000005EB3_00000001.mfs > May 27 08:15:28 backup1 mfsmount[12917]: master: tcp recv error: > ETIMEDOUT (Operation timed out) (1) > May 27 08:15:29 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:15:32 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:15:33 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B4/chunk_0000000000005EB4_00000001.mfs > May 27 08:15:35 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:15:44 backup1 mfsmount[12917]: last message repeated 3 times > May 27 08:15:44 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B5/chunk_0000000000005EB5_00000001.mfs > May 27 08:15:47 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:15:55 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:15:55 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B6/chunk_0000000000005EB6_00000001.mfs > May 27 08:15:56 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:15:59 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:16:00 backup1 mfschunkserver[2556]: connecting ... > May 27 08:16:00 backup1 mfschunkserver[2556]: connected to Master > May 27 08:16:02 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:17:05 backup1 mfsmount[12917]: last message repeated 21 times > May 27 08:17:15 backup1 mfsmount[12917]: last message repeated 3 times > May 27 08:17:15 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:17:15 backup1 mfsmaster[12869]: connection with > CS(192.168.3.21) has been closed by peer > May 27 08:17:15 backup1 mfsmaster[12869]: chunkserver disconnected - ip: > 192.168.3.21, port: 9422, usedspace: 7739308400640 (7207.79 GiB), > totalspace: 10944744390656 (10193.09 GiB) > May 27 08:17:16 backup1 mfsmaster[12869]: connection with > ML(192.168.3.13) has been closed by peer > May 27 08:17:16 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:17:17 backup1 mfsmaster[12869]: last message repeated 7 times > May 27 08:17:17 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:17:17 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:17:19 backup1 mfsmaster[12869]: last message repeated 28 times > May 27 08:17:19 backup1 mfsmount[12917]: registered to master > May 27 08:18:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:18:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:18:16 backup1 mfsmaster[12869]: chunkserver disconnected - ip: > 192.168.3.21, port: 0, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB) > May 27 08:19:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:19:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:19:14 backup1 mfsmount[12917]: file: 7340122, index: 0 - > fs_writechunk returns status 8 > May 27 08:20:00 backup1 mfsmount[12917]: last message repeated 17 times > May 27 08:20:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:20:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:20:05 backup1 mfsmount[12917]: file: 7340122, index: 0 - > fs_writechunk returns status 8 > May 27 08:21:00 backup1 mfsmount[12917]: last message repeated 7 times > May 27 08:21:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:21:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:21:02 backup1 mfsmount[12917]: file: 7340122, index: 0 - > fs_writechunk returns status 8 > May 27 08:21:29 backup1 mfsmount[12917]: last message repeated 3 times > May 27 08:21:29 backup1 mfsmount[12917]: error writing file number > 7340122: EIO (Input/output error) > May 27 08:21:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: > 4816320, version: 1 - there are no valid copies > May 27 08:21:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't > connect to proper chunkserver (try counter: 1) > May 27 08:22:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:22:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:22:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: > 4816320, version: 1 - there are no valid copies > May 27 08:22:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't > connect to proper chunkserver (try counter: 8) > May 27 08:23:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:23:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:23:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: > 4816320, version: 1 - there are no valid copies > May 27 08:23:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't > connect to proper chunkserver (try counter: 15) > May 27 08:24:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:24:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:24:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: > 4816320, version: 1 - there are no valid copies > May 27 08:24:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't > connect to proper chunkserver (try counter: 22) > May 27 08:25:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:25:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:25:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: > 4816320, version: 1 - there are no valid copies > May 27 08:25:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't > connect to proper chunkserver (try counter: 29) > May 27 08:26:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:26:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 > GiB), totalspace: 0 (0.00 GiB), usage: 0.00% > May 27 08:26:10 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B7/chunk_0000000000005EB7_00000001.mfs > May 27 08:26:10 backup1 mfschunkserver[2556]: connection reset by Master > May 27 08:26:20 backup1 mfschunkserver[2556]: connecting ... > May 27 08:26:20 backup1 mfschunkserver[2556]: connected to Master > May 27 08:26:21 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B8/chunk_0000000000005EB8_00000001.mfs > May 27 08:26:21 backup1 mfsmaster[12869]: chunkserver register begin > (packet version: 5) - ip: 192.168.3.21, port: 9422 > May 27 08:26:28 backup1 mfsmaster[12869]: chunkserver register end > (packet version: 5) - ip: 192.168.3.21, port: 9422, usedspace: > 7739308400640 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB) > May 27 08:26:31 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/B9/chunk_0000000000005EB9_00000001.mfs > May 27 08:26:41 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BA/chunk_0000000000005EBA_00000001.mfs > May 27 08:26:42 backup1 mfsmount[12917]: file: 7445943, index: 0, chunk: > 4816323, version: 1 - writeworker: connection with (C0A80315:9422) was > timed out (unfinished writes: 2; try counter: 1) > May 27 08:26:51 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:26:51 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BB/chunk_0000000000005EBB_00000001.mfs > May 27 08:27:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:27:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739309965312 (7207.79 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:27:00 backup1 mfsmaster[12869]: total: usedspace: > 7739309965312 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:27:01 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BC/chunk_0000000000005EBC_00000001.mfs > May 27 08:27:11 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BD/chunk_0000000000005EBD_00000001.mfs > May 27 08:27:21 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BE/chunk_0000000000005EBE_00000001.mfs > May 27 08:27:31 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/BF/chunk_0000000000005EBF_00000001.mfs > May 27 08:27:41 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C0/chunk_0000000000005EC0_00000001.mfs > May 27 08:27:51 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C1/chunk_0000000000005EC1_00000001.mfs > May 27 08:28:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:28:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739310948352 (7207.79 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:28:00 backup1 mfsmaster[12869]: total: usedspace: > 7739310948352 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:28:01 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C3/chunk_0000000000005EC3_00000001.mfs > May 27 08:28:11 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C4/chunk_0000000000005EC4_00000001.mfs > May 27 08:28:22 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C5/chunk_0000000000005EC5_00000001.mfs > May 27 08:28:32 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C7/chunk_0000000000005EC7_00000001.mfs > May 27 08:28:42 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C8/chunk_0000000000005EC8_00000001.mfs > May 27 08:28:52 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/C9/chunk_0000000000005EC9_00000001.mfs > May 27 08:29:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739313045504 (7207.80 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:29:00 backup1 mfsmaster[12869]: total: usedspace: > 7739313045504 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:29:02 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/CA/chunk_0000000000005ECA_00000001.mfs > May 27 08:29:12 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/CB/chunk_0000000000005ECB_00000001.mfs > May 27 08:29:22 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/CC/chunk_0000000000005ECC_00000001.mfs > May 27 08:29:26 backup1 mfsmount[12917]: master: tcp recv error: > ETIMEDOUT (Operation timed out) (1) > May 27 08:29:27 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:29:30 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:29:32 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/CD/chunk_0000000000005ECD_00000001.mfs > May 27 08:29:32 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:29:32 backup1 mfsmaster[12869]: last message repeated 2 times > May 27 08:29:32 backup1 mfsmount[12917]: registered to master > May 27 08:29:42 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/CE/chunk_0000000000005ECE_00000001.mfs > May 27 08:29:52 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D0/chunk_0000000000005ED0_00000001.mfs > May 27 08:30:02 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D1/chunk_0000000000005ED1_00000001.mfs > May 27 08:30:05 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: > 4816324, version: 1 - writeworker: connection with (C0A80315:9422) was > timed out (unfinished writes: 1; try counter: 1) > May 27 08:30:12 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D2/chunk_0000000000005ED2_00000001.mfs > May 27 08:30:22 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D3/chunk_0000000000005ED3_00000001.mfs > May 27 08:30:33 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D4/chunk_0000000000005ED4_00000001.mfs > May 27 08:30:43 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D5/chunk_0000000000005ED5_00000001.mfs > May 27 08:30:53 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D6/chunk_0000000000005ED6_00000001.mfs > May 27 08:31:03 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D7/chunk_0000000000005ED7_00000001.mfs > May 27 08:31:13 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D8/chunk_0000000000005ED8_00000001.mfs > May 27 08:31:23 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/D9/chunk_0000000000005ED9_00000001.mfs > May 27 08:31:33 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/DA/chunk_0000000000005EDA_00000001.mfs > May 27 08:31:43 backup1 mfsmount[12917]: master: tcp recv error: > ETIMEDOUT (Operation timed out) (1) > May 27 08:31:43 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/DB/chunk_0000000000005EDB_00000001.mfs > May 27 08:31:44 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:31:53 backup1 mfsmount[12917]: last message repeated 3 times > May 27 08:31:53 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/DD/chunk_0000000000005EDD_00000001.mfs > May 27 08:31:56 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:32:03 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:32:03 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/DE/chunk_0000000000005EDE_00000001.mfs > May 27 08:32:05 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:32:14 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:32:14 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/DF/chunk_0000000000005EDF_00000001.mfs > May 27 08:32:14 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:32:24 backup1 mfsmount[12917]: last message repeated 3 times > May 27 08:32:24 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E0/chunk_0000000000005EE0_00000001.mfs > May 27 08:32:25 backup1 mfschunkserver[2556]: connecting ... > May 27 08:32:25 backup1 mfschunkserver[2556]: connected to Master > May 27 08:32:26 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:33:29 backup1 mfsmount[12917]: last message repeated 21 times > May 27 08:34:25 backup1 mfsmount[12917]: last message repeated 18 times > May 27 08:34:25 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:34:25 backup1 mfsmaster[12869]: connection with > CS(192.168.3.21) has been closed by peer > May 27 08:34:25 backup1 mfsmaster[12869]: chunkserver disconnected - ip: > 192.168.3.21, port: 9422, usedspace: 7739311943680 (7207.80 GiB), > totalspace: 10944744390656 (10193.09 GiB) > May 27 08:34:26 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:34:33 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:34:33 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E1/chunk_0000000000005EE1_00000001.mfs > May 27 08:34:34 backup1 mfsmaster[12869]: connection with > ML(192.168.3.13) has been closed by peer > May 27 08:34:34 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:34:34 backup1 mfsmaster[12869]: chunkserver register begin > (packet version: 5) - ip: 192.168.3.21, port: 9422 > May 27 08:34:34 backup1 mfsmaster[12869]: connection with > ML(192.168.3.13) has been closed by peer > May 27 08:34:34 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:34:34 backup1 mfsmaster[12869]: connection with > CS(192.168.3.21) has been closed by peer > May 27 08:34:34 backup1 mfsmaster[12869]: chunkserver disconnected - ip: > 192.168.3.21, port: 9422, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB) > May 27 08:34:35 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:34:35 backup1 mfsmaster[12869]: last message repeated 2 times > May 27 08:34:35 backup1 mfsmount[12917]: master: register error (read > header: ETIMEDOUT (Operation timed out)) > May 27 08:34:35 backup1 mfsmaster[12869]: connection with > client(ip:192.168.3.21) has been closed by peer > May 27 08:34:37 backup1 mfsmaster[12869]: last message repeated 52 times > May 27 08:34:37 backup1 mfsmount[12917]: registered to master > May 27 08:34:39 backup1 mfsmount[12917]: file: 7445943, index: 0 - > fs_writechunk returns status 8 > May 27 08:34:40 backup1 mfsmount[12917]: last message repeated 2 times > May 27 08:34:40 backup1 mfschunkserver[2556]: connecting ... > May 27 08:34:40 backup1 mfschunkserver[2556]: connected to Master > May 27 08:34:40 backup1 mfsmount[12917]: file: 7445943, index: 0 - > fs_writechunk returns status 8 > May 27 08:34:41 backup1 mfsmount[12917]: file: 7445943, index: 0 - > fs_writechunk returns status 8 > May 27 08:34:41 backup1 mfsmaster[12869]: chunkserver register begin > (packet version: 5) - ip: 192.168.3.21, port: 9422 > May 27 08:34:42 backup1 mfsmaster[12869]: chunkserver register end > (packet version: 5) - ip: 192.168.3.21, port: 9422, usedspace: > 7739311943680 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB) > May 27 08:34:43 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E2/chunk_0000000000005EE2_00000001.mfs > May 27 08:34:53 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E3/chunk_0000000000005EE3_00000001.mfs > May 27 08:35:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:35:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739314479104 (7207.80 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:35:00 backup1 mfsmaster[12869]: total: usedspace: > 7739314479104 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:35:03 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E4/chunk_0000000000005EE4_00000001.mfs > May 27 08:35:14 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E5/chunk_0000000000005EE5_00000001.mfs > May 27 08:35:24 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E6/chunk_0000000000005EE6_00000001.mfs > May 27 08:35:34 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E7/chunk_0000000000005EE7_00000001.mfs > May 27 08:35:44 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E8/chunk_0000000000005EE8_00000001.mfs > May 27 08:35:54 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/E9/chunk_0000000000005EE9_00000001.mfs > May 27 08:36:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:36:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739314479104 (7207.80 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:36:00 backup1 mfsmaster[12869]: total: usedspace: > 7739314479104 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > May 27 08:36:04 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/EA/chunk_0000000000005EEA_00000001.mfs > May 27 08:36:14 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/EB/chunk_0000000000005EEB_00000001.mfs > May 27 08:36:24 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/EC/chunk_0000000000005EEC_00000001.mfs > May 27 08:36:34 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/ED/chunk_0000000000005EED_00000001.mfs > May 27 08:36:45 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/EE/chunk_0000000000005EEE_00000001.mfs > May 27 08:36:55 backup1 mfschunkserver[2556]: testing chunk: > /mnt/mfschunk1/EF/chunk_0000000000005EEF_00000001.mfs > May 27 08:37:00 backup1 mfsmaster[12869]: chunkservers status: > May 27 08:37:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, > port: 9422): usedspace: 7739314479104 (7207.80 GiB), totalspace: > 10944744390656 (10193.09 GiB), usage: 70.71% > May 27 08:37:00 backup1 mfsmaster[12869]: total: usedspace: > 7739314479104 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), > usage: 70.71% > > > Can you help me what to do with it? > Is this fuse, moosefs, kernel problem or something else? hi! I've just realized an other error. $ dirvish --vault cluster/Projects dirvish cluster/Projects:default fatal error: filesystem full dirvish cluster/Projects:default fatal error (12) -- filesystem full cluster/Projects:default post-server failed (1) log: May 28 14:58:04 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/70/chunk_0000000000008470_00000001.mfs May 28 14:58:14 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/71/chunk_0000000000008471_00000001.mfs May 28 14:58:24 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/72/chunk_0000000000008472_00000001.mfs May 28 14:58:34 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/73/chunk_0000000000008473_00000001.mfs May 28 14:58:44 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/74/chunk_0000000000008474_00000001.mfs May 28 14:58:55 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/75/chunk_0000000000008475_00000001.mfs May 28 14:59:00 backup1 mfsmaster[12869]: chunkservers status: May 28 14:59:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, port: 9422): usedspace: 7734218166272 (7203.05 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.67% May 28 14:59:00 backup1 mfsmaster[12869]: total: usedspace: 7734218166272 (7203.05 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.67% May 28 14:59:05 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/76/chunk_0000000000008476_00000001.mfs May 28 14:59:15 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/77/chunk_0000000000008477_00000001.mfs May 28 14:59:24 backup1 mfsmount[12917]: master: tcp recv error: ETIMEDOUT (Operation timed out) (1) May 28 14:59:25 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 14:59:25 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/78/chunk_0000000000008478_00000001.mfs May 28 14:59:28 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 14:59:35 backup1 mfsmount[12917]: last message repeated 2 times May 28 14:59:35 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/79/chunk_0000000000008479_00000001.mfs May 28 14:59:37 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 14:59:45 backup1 mfsmount[12917]: last message repeated 2 times May 28 14:59:45 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7A/chunk_000000000000847A_00000001.mfs May 28 14:59:46 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 14:59:55 backup1 mfsmount[12917]: last message repeated 3 times May 28 14:59:55 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7B/chunk_000000000000847B_00000001.mfs May 28 14:59:58 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 15:00:06 backup1 mfsmount[12917]: last message repeated 2 times May 28 15:00:06 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7C/chunk_000000000000847C_00000001.mfs May 28 15:00:07 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 15:00:15 backup1 mfsmount[12917]: last message repeated 2 times May 28 15:00:15 backup1 mfschunkserver[2556]: connecting ... May 28 15:00:15 backup1 mfschunkserver[2556]: connected to Master May 28 15:00:16 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 15:01:19 backup1 mfsmount[12917]: last message repeated 21 times May 28 15:02:19 backup1 mfsmount[12917]: last message repeated 19 times May 28 15:02:19 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 15:02:19 backup1 mfsmaster[12869]: connection with CS(192.168.3.21) has been closed by peer May 28 15:02:19 backup1 mfsmaster[12869]: chunkserver disconnected - ip: 192.168.3.21, port: 9422, usedspace: 7734399709184 (7203.22 GiB), totalspace: 10944744390656 (10193.09 GiB) May 28 15:02:19 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 15:02:22 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 28 15:02:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.20) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.13) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.13) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.20) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with ML(192.168.3.13) has been closed by peer May 28 15:02:23 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 28 15:02:24 backup1 mfsmaster[12869]: last message repeated 55 times May 28 15:02:24 backup1 mfsmount[12917]: registered to master May 28 15:02:25 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: 4816943, version: 1 - there are no valid copies May 28 15:02:25 backup1 mfsmount[12917]: file: 122301, index: 0 - can't connect to proper chunkserver (try counter: 1) May 28 15:02:26 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:26 backup1 mfsmount[12917]: last message repeated 2 times May 28 15:02:26 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:02:27 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:32 backup1 mfsmount[12917]: last message repeated 3 times May 28 15:02:32 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:02:33 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:38 backup1 mfsmount[12917]: last message repeated 2 times May 28 15:02:38 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:02:41 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:44 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:44 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:02:48 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:51 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:02:52 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:56 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:02:58 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:01 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:05 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:06 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:11 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:13 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:17 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:20 backup1 mfsmaster[12869]: chunkserver disconnected - ip: 192.168.3.21, port: 0, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB) May 28 15:03:22 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:23 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:25 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: 4816943, version: 1 - there are no valid copies May 28 15:03:25 backup1 mfsmount[12917]: file: 122301, index: 0 - can't connect to proper chunkserver (try counter: 8) May 28 15:03:29 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:30 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:36 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:39 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:43 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:48 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:50 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:03:57 backup1 mfsmount[12917]: file: 107222, index: 0 - fs_writechunk returns status 8 May 28 15:03:57 backup1 mfsmount[12917]: error writing file number 107222: EIO (Input/output error) May 28 15:03:58 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:04:00 backup1 mfsmaster[12869]: chunkservers status: May 28 15:04:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 28 15:04:06 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:04:25 backup1 mfsmount[12917]: last message repeated 2 times May 28 15:04:25 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: 4816943, version: 1 - there are no valid copies May 28 15:04:25 backup1 mfsmount[12917]: file: 122301, index: 0 - can't connect to proper chunkserver (try counter: 15) May 28 15:04:32 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:04:41 backup1 mfsmount[12917]: file: 4691391, index: 0 - fs_writechunk returns status 12 May 28 15:04:41 backup1 mfsmount[12917]: error writing file number 4691391: ENOSPC (No space left on device) May 28 15:04:42 backup1 mfsmount[12917]: file: 107221, index: 0 - fs_writechunk returns status 8 May 28 15:05:00 backup1 mfsmount[12917]: last message repeated 10 times May 28 15:05:00 backup1 mfsmaster[12869]: chunkservers status: May 28 15:05:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 28 15:05:00 backup1 mfsmount[12917]: file: 107221, index: 0 - fs_writechunk returns status 8 May 28 15:05:25 backup1 mfsmount[12917]: last message repeated 5 times May 28 15:05:25 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: 4816943, version: 1 - there are no valid copies May 28 15:05:25 backup1 mfsmount[12917]: file: 122301, index: 0 - can't connect to proper chunkserver (try counter: 22) May 28 15:05:27 backup1 mfsmount[12917]: file: 107221, index: 0 - fs_writechunk returns status 8 May 28 15:06:00 backup1 mfsmount[12917]: last message repeated 4 times May 28 15:06:00 backup1 mfsmaster[12869]: chunkservers status: May 28 15:06:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 28 15:06:01 backup1 mfsmount[12917]: file: 107221, index: 0 - fs_writechunk returns status 8 May 28 15:06:25 backup1 mfsmount[12917]: last message repeated 3 times May 28 15:06:25 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: 4816943, version: 1 - there are no valid copies May 28 15:06:25 backup1 mfsmount[12917]: file: 122301, index: 0 - can't connect to proper chunkserver (try counter: 29) May 28 15:06:25 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7D/chunk_000000000000847D_00000001.mfs May 28 15:06:25 backup1 mfschunkserver[2556]: connection reset by Master May 28 15:06:32 backup1 mfsmount[12917]: file: 107221, index: 0 - fs_writechunk returns status 8 May 28 15:06:35 backup1 mfschunkserver[2556]: connecting ... May 28 15:06:35 backup1 mfschunkserver[2556]: connected to Master May 28 15:06:36 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7E/chunk_000000000000847E_00000001.mfs May 28 15:06:36 backup1 mfsmaster[12869]: chunkserver register begin (packet version: 5) - ip: 192.168.3.21, port: 9422 May 28 15:06:38 backup1 mfsmaster[12869]: chunkserver register end (packet version: 5) - ip: 192.168.3.21, port: 9422, usedspace: 7734393311232 (7203.22 GiB), totalspace: 10944744390656 (10193.09 GiB) May 28 15:06:46 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/7F/chunk_000000000000847F_00000001.mfs May 28 15:06:56 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/80/chunk_0000000000008480_00000001.mfs May 28 15:07:00 backup1 mfsmaster[12869]: chunkservers status: May 28 15:07:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, port: 9422): usedspace: 7734393442304 (7203.22 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.67% May 28 15:07:00 backup1 mfsmaster[12869]: total: usedspace: 7734393442304 (7203.22 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.67% Filesystem Size Used Avail Use% Mounted on /dev/sda2 19G 1.4G 17G 8% / none 3.9G 180K 3.9G 1% /dev none 4.0G 0 4.0G 0% /dev/shm none 4.0G 716K 4.0G 1% /var/run none 4.0G 0 4.0G 0% /var/lock /dev/sda6 10T 7.1T 3.0T 71% /mnt/mfschunk1 /dev/sda3 19G 12G 6.0G 66% /var /dev/sda4 4.6G 138M 4.3G 4% /tmp mfsmaster:9421 10T 7.1T 3.0T 71% /data/backup Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 1222992 64010 1158982 6% / none 1021194 762 1020432 1% /dev none 1023079 1 1023078 1% /dev/shm none 1023079 58 1023021 1% /var/run none 1023079 1 1023078 1% /var/lock /dev/sda6 2138062720 4778154 2133284566 1% /mnt/mfschunk1 /dev/sda3 1222992 3174 1219818 1% /var /dev/sda4 305216 13 305203 1% /tmp mfsmaster:9421 1031528383 30522363 1001006020 3% /data/backup So it's definetly not full. I don't understand, what's going on:/ Does somebody have any idea? Thank you, tamas |
|
From: Upendra M. <upe...@he...> - 2011-05-27 12:04:10
|
Hi
Moosefs has problem if cache=none.
We can give either writeback or writethrough.
It also has issue if the disk is virtio.
Performance of IO becomes very very low when compared to ide bus.
I am using qemu 0.14 and mfs 1.6.20 on ubuntu 10.04.
On Thu, May 26, 2011 at 9:01 AM, Dietmar Maurer <di...@pr...> wrote:
> >
> > On 13/05/2011 19:26, Richard Chute wrote:
> > > 5385 open("/mfs/pve/images/101/vm-101-disk-1.raw",
> > > O_RDWR|O_DIRECT|O_CLOEXEC) = -1 EINVAL (Invalid argument)
> >
> > I was wrong, I am using proxmox 1.7 with MooseFS. Today, after trying to
> > ugrade to 1.8, I get the same error. :(
> >
> > There is probably a regression (or some conflicts with MooseFS) on one of
> the
> > latest proxmox updated packages:
>
> Seems Moosefs does not support O_DIRECT. Try to use another cache setting
> for your disk.
> Edit the VM config, set 'cache=writethrough' for all disks (see 'man qm')
>
> PVE 1.8 uses "cache=none" by default.
>
> - Dietmar
>
>
>
> ------------------------------------------------------------------------------
> vRanger cuts backup time in half-while increasing security.
> With the market-leading solution for virtual backup and recovery,
> you get blazing-fast, flexible, and affordable data protection.
> Download your free trial now.
> http://p.sf.net/sfu/quest-d2dcopy1
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
--
Thanks and Regards,
Upendra.M
|
|
From: Papp T. <to...@ma...> - 2011-05-27 08:19:51
|
hi! Sometimes there is an error on our mini cluster. Still Ubuntu Natty with recompiled moosefs from ppa. May 27 08:15:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, port: 9422): usedspace: 7739308400640 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.71% May 27 08:15:00 backup1 mfsmaster[12869]: total: usedspace: 7739308400640 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.71% May 27 08:15:02 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/B1/chunk_0000000000005EB1_00000001.mfs May 27 08:15:12 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/B2/chunk_0000000000005EB2_00000001.mfs May 27 08:15:23 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/B3/chunk_0000000000005EB3_00000001.mfs May 27 08:15:28 backup1 mfsmount[12917]: master: tcp recv error: ETIMEDOUT (Operation timed out) (1) May 27 08:15:29 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 27 08:15:32 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 27 08:15:33 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/B4/chunk_0000000000005EB4_00000001.mfs May 27 08:15:35 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 27 08:15:44 backup1 mfsmount[12917]: last message repeated 3 times May 27 08:15:44 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/B5/chunk_0000000000005EB5_00000001.mfs May 27 08:15:47 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 27 08:15:55 backup1 mfsmount[12917]: last message repeated 2 times May 27 08:15:55 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/B6/chunk_0000000000005EB6_00000001.mfs May 27 08:15:56 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 27 08:15:59 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 27 08:16:00 backup1 mfschunkserver[2556]: connecting ... May 27 08:16:00 backup1 mfschunkserver[2556]: connected to Master May 27 08:16:02 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 27 08:17:05 backup1 mfsmount[12917]: last message repeated 21 times May 27 08:17:15 backup1 mfsmount[12917]: last message repeated 3 times May 27 08:17:15 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 27 08:17:15 backup1 mfsmaster[12869]: connection with CS(192.168.3.21) has been closed by peer May 27 08:17:15 backup1 mfsmaster[12869]: chunkserver disconnected - ip: 192.168.3.21, port: 9422, usedspace: 7739308400640 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB) May 27 08:17:16 backup1 mfsmaster[12869]: connection with ML(192.168.3.13) has been closed by peer May 27 08:17:16 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 27 08:17:17 backup1 mfsmaster[12869]: last message repeated 7 times May 27 08:17:17 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 27 08:17:17 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 27 08:17:19 backup1 mfsmaster[12869]: last message repeated 28 times May 27 08:17:19 backup1 mfsmount[12917]: registered to master May 27 08:18:00 backup1 mfsmaster[12869]: chunkservers status: May 27 08:18:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 27 08:18:16 backup1 mfsmaster[12869]: chunkserver disconnected - ip: 192.168.3.21, port: 0, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB) May 27 08:19:00 backup1 mfsmaster[12869]: chunkservers status: May 27 08:19:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 27 08:19:14 backup1 mfsmount[12917]: file: 7340122, index: 0 - fs_writechunk returns status 8 May 27 08:20:00 backup1 mfsmount[12917]: last message repeated 17 times May 27 08:20:00 backup1 mfsmaster[12869]: chunkservers status: May 27 08:20:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 27 08:20:05 backup1 mfsmount[12917]: file: 7340122, index: 0 - fs_writechunk returns status 8 May 27 08:21:00 backup1 mfsmount[12917]: last message repeated 7 times May 27 08:21:00 backup1 mfsmaster[12869]: chunkservers status: May 27 08:21:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 27 08:21:02 backup1 mfsmount[12917]: file: 7340122, index: 0 - fs_writechunk returns status 8 May 27 08:21:29 backup1 mfsmount[12917]: last message repeated 3 times May 27 08:21:29 backup1 mfsmount[12917]: error writing file number 7340122: EIO (Input/output error) May 27 08:21:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: 4816320, version: 1 - there are no valid copies May 27 08:21:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't connect to proper chunkserver (try counter: 1) May 27 08:22:00 backup1 mfsmaster[12869]: chunkservers status: May 27 08:22:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 27 08:22:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: 4816320, version: 1 - there are no valid copies May 27 08:22:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't connect to proper chunkserver (try counter: 8) May 27 08:23:00 backup1 mfsmaster[12869]: chunkservers status: May 27 08:23:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 27 08:23:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: 4816320, version: 1 - there are no valid copies May 27 08:23:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't connect to proper chunkserver (try counter: 15) May 27 08:24:00 backup1 mfsmaster[12869]: chunkservers status: May 27 08:24:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 27 08:24:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: 4816320, version: 1 - there are no valid copies May 27 08:24:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't connect to proper chunkserver (try counter: 22) May 27 08:25:00 backup1 mfsmaster[12869]: chunkservers status: May 27 08:25:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 27 08:25:30 backup1 mfsmount[12917]: file: 7340122, index: 0, chunk: 4816320, version: 1 - there are no valid copies May 27 08:25:30 backup1 mfsmount[12917]: file: 7340122, index: 0 - can't connect to proper chunkserver (try counter: 29) May 27 08:26:00 backup1 mfsmaster[12869]: chunkservers status: May 27 08:26:00 backup1 mfsmaster[12869]: total: usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB), usage: 0.00% May 27 08:26:10 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/B7/chunk_0000000000005EB7_00000001.mfs May 27 08:26:10 backup1 mfschunkserver[2556]: connection reset by Master May 27 08:26:20 backup1 mfschunkserver[2556]: connecting ... May 27 08:26:20 backup1 mfschunkserver[2556]: connected to Master May 27 08:26:21 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/B8/chunk_0000000000005EB8_00000001.mfs May 27 08:26:21 backup1 mfsmaster[12869]: chunkserver register begin (packet version: 5) - ip: 192.168.3.21, port: 9422 May 27 08:26:28 backup1 mfsmaster[12869]: chunkserver register end (packet version: 5) - ip: 192.168.3.21, port: 9422, usedspace: 7739308400640 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB) May 27 08:26:31 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/B9/chunk_0000000000005EB9_00000001.mfs May 27 08:26:41 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/BA/chunk_0000000000005EBA_00000001.mfs May 27 08:26:42 backup1 mfsmount[12917]: file: 7445943, index: 0, chunk: 4816323, version: 1 - writeworker: connection with (C0A80315:9422) was timed out (unfinished writes: 2; try counter: 1) May 27 08:26:51 backup1 mfsmount[12917]: last message repeated 2 times May 27 08:26:51 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/BB/chunk_0000000000005EBB_00000001.mfs May 27 08:27:00 backup1 mfsmaster[12869]: chunkservers status: May 27 08:27:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, port: 9422): usedspace: 7739309965312 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.71% May 27 08:27:00 backup1 mfsmaster[12869]: total: usedspace: 7739309965312 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.71% May 27 08:27:01 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/BC/chunk_0000000000005EBC_00000001.mfs May 27 08:27:11 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/BD/chunk_0000000000005EBD_00000001.mfs May 27 08:27:21 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/BE/chunk_0000000000005EBE_00000001.mfs May 27 08:27:31 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/BF/chunk_0000000000005EBF_00000001.mfs May 27 08:27:41 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/C0/chunk_0000000000005EC0_00000001.mfs May 27 08:27:51 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/C1/chunk_0000000000005EC1_00000001.mfs May 27 08:28:00 backup1 mfsmaster[12869]: chunkservers status: May 27 08:28:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, port: 9422): usedspace: 7739310948352 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.71% May 27 08:28:00 backup1 mfsmaster[12869]: total: usedspace: 7739310948352 (7207.79 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.71% May 27 08:28:01 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/C3/chunk_0000000000005EC3_00000001.mfs May 27 08:28:11 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/C4/chunk_0000000000005EC4_00000001.mfs May 27 08:28:22 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/C5/chunk_0000000000005EC5_00000001.mfs May 27 08:28:32 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/C7/chunk_0000000000005EC7_00000001.mfs May 27 08:28:42 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/C8/chunk_0000000000005EC8_00000001.mfs May 27 08:28:52 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/C9/chunk_0000000000005EC9_00000001.mfs May 27 08:29:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, port: 9422): usedspace: 7739313045504 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.71% May 27 08:29:00 backup1 mfsmaster[12869]: total: usedspace: 7739313045504 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.71% May 27 08:29:02 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/CA/chunk_0000000000005ECA_00000001.mfs May 27 08:29:12 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/CB/chunk_0000000000005ECB_00000001.mfs May 27 08:29:22 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/CC/chunk_0000000000005ECC_00000001.mfs May 27 08:29:26 backup1 mfsmount[12917]: master: tcp recv error: ETIMEDOUT (Operation timed out) (1) May 27 08:29:27 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 27 08:29:30 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 27 08:29:32 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/CD/chunk_0000000000005ECD_00000001.mfs May 27 08:29:32 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 27 08:29:32 backup1 mfsmaster[12869]: last message repeated 2 times May 27 08:29:32 backup1 mfsmount[12917]: registered to master May 27 08:29:42 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/CE/chunk_0000000000005ECE_00000001.mfs May 27 08:29:52 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/D0/chunk_0000000000005ED0_00000001.mfs May 27 08:30:02 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/D1/chunk_0000000000005ED1_00000001.mfs May 27 08:30:05 backup1 mfsmount[12917]: file: 122301, index: 0, chunk: 4816324, version: 1 - writeworker: connection with (C0A80315:9422) was timed out (unfinished writes: 1; try counter: 1) May 27 08:30:12 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/D2/chunk_0000000000005ED2_00000001.mfs May 27 08:30:22 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/D3/chunk_0000000000005ED3_00000001.mfs May 27 08:30:33 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/D4/chunk_0000000000005ED4_00000001.mfs May 27 08:30:43 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/D5/chunk_0000000000005ED5_00000001.mfs May 27 08:30:53 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/D6/chunk_0000000000005ED6_00000001.mfs May 27 08:31:03 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/D7/chunk_0000000000005ED7_00000001.mfs May 27 08:31:13 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/D8/chunk_0000000000005ED8_00000001.mfs May 27 08:31:23 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/D9/chunk_0000000000005ED9_00000001.mfs May 27 08:31:33 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/DA/chunk_0000000000005EDA_00000001.mfs May 27 08:31:43 backup1 mfsmount[12917]: master: tcp recv error: ETIMEDOUT (Operation timed out) (1) May 27 08:31:43 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/DB/chunk_0000000000005EDB_00000001.mfs May 27 08:31:44 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 27 08:31:53 backup1 mfsmount[12917]: last message repeated 3 times May 27 08:31:53 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/DD/chunk_0000000000005EDD_00000001.mfs May 27 08:31:56 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 27 08:32:03 backup1 mfsmount[12917]: last message repeated 2 times May 27 08:32:03 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/DE/chunk_0000000000005EDE_00000001.mfs May 27 08:32:05 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 27 08:32:14 backup1 mfsmount[12917]: last message repeated 2 times May 27 08:32:14 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/DF/chunk_0000000000005EDF_00000001.mfs May 27 08:32:14 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 27 08:32:24 backup1 mfsmount[12917]: last message repeated 3 times May 27 08:32:24 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/E0/chunk_0000000000005EE0_00000001.mfs May 27 08:32:25 backup1 mfschunkserver[2556]: connecting ... May 27 08:32:25 backup1 mfschunkserver[2556]: connected to Master May 27 08:32:26 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 27 08:33:29 backup1 mfsmount[12917]: last message repeated 21 times May 27 08:34:25 backup1 mfsmount[12917]: last message repeated 18 times May 27 08:34:25 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 27 08:34:25 backup1 mfsmaster[12869]: connection with CS(192.168.3.21) has been closed by peer May 27 08:34:25 backup1 mfsmaster[12869]: chunkserver disconnected - ip: 192.168.3.21, port: 9422, usedspace: 7739311943680 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB) May 27 08:34:26 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 27 08:34:33 backup1 mfsmount[12917]: last message repeated 2 times May 27 08:34:33 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/E1/chunk_0000000000005EE1_00000001.mfs May 27 08:34:34 backup1 mfsmaster[12869]: connection with ML(192.168.3.13) has been closed by peer May 27 08:34:34 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 27 08:34:34 backup1 mfsmaster[12869]: chunkserver register begin (packet version: 5) - ip: 192.168.3.21, port: 9422 May 27 08:34:34 backup1 mfsmaster[12869]: connection with ML(192.168.3.13) has been closed by peer May 27 08:34:34 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 27 08:34:34 backup1 mfsmaster[12869]: connection with CS(192.168.3.21) has been closed by peer May 27 08:34:34 backup1 mfsmaster[12869]: chunkserver disconnected - ip: 192.168.3.21, port: 9422, usedspace: 0 (0.00 GiB), totalspace: 0 (0.00 GiB) May 27 08:34:35 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 27 08:34:35 backup1 mfsmaster[12869]: last message repeated 2 times May 27 08:34:35 backup1 mfsmount[12917]: master: register error (read header: ETIMEDOUT (Operation timed out)) May 27 08:34:35 backup1 mfsmaster[12869]: connection with client(ip:192.168.3.21) has been closed by peer May 27 08:34:37 backup1 mfsmaster[12869]: last message repeated 52 times May 27 08:34:37 backup1 mfsmount[12917]: registered to master May 27 08:34:39 backup1 mfsmount[12917]: file: 7445943, index: 0 - fs_writechunk returns status 8 May 27 08:34:40 backup1 mfsmount[12917]: last message repeated 2 times May 27 08:34:40 backup1 mfschunkserver[2556]: connecting ... May 27 08:34:40 backup1 mfschunkserver[2556]: connected to Master May 27 08:34:40 backup1 mfsmount[12917]: file: 7445943, index: 0 - fs_writechunk returns status 8 May 27 08:34:41 backup1 mfsmount[12917]: file: 7445943, index: 0 - fs_writechunk returns status 8 May 27 08:34:41 backup1 mfsmaster[12869]: chunkserver register begin (packet version: 5) - ip: 192.168.3.21, port: 9422 May 27 08:34:42 backup1 mfsmaster[12869]: chunkserver register end (packet version: 5) - ip: 192.168.3.21, port: 9422, usedspace: 7739311943680 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB) May 27 08:34:43 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/E2/chunk_0000000000005EE2_00000001.mfs May 27 08:34:53 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/E3/chunk_0000000000005EE3_00000001.mfs May 27 08:35:00 backup1 mfsmaster[12869]: chunkservers status: May 27 08:35:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, port: 9422): usedspace: 7739314479104 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.71% May 27 08:35:00 backup1 mfsmaster[12869]: total: usedspace: 7739314479104 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.71% May 27 08:35:03 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/E4/chunk_0000000000005EE4_00000001.mfs May 27 08:35:14 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/E5/chunk_0000000000005EE5_00000001.mfs May 27 08:35:24 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/E6/chunk_0000000000005EE6_00000001.mfs May 27 08:35:34 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/E7/chunk_0000000000005EE7_00000001.mfs May 27 08:35:44 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/E8/chunk_0000000000005EE8_00000001.mfs May 27 08:35:54 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/E9/chunk_0000000000005EE9_00000001.mfs May 27 08:36:00 backup1 mfsmaster[12869]: chunkservers status: May 27 08:36:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, port: 9422): usedspace: 7739314479104 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.71% May 27 08:36:00 backup1 mfsmaster[12869]: total: usedspace: 7739314479104 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.71% May 27 08:36:04 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/EA/chunk_0000000000005EEA_00000001.mfs May 27 08:36:14 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/EB/chunk_0000000000005EEB_00000001.mfs May 27 08:36:24 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/EC/chunk_0000000000005EEC_00000001.mfs May 27 08:36:34 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/ED/chunk_0000000000005EED_00000001.mfs May 27 08:36:45 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/EE/chunk_0000000000005EEE_00000001.mfs May 27 08:36:55 backup1 mfschunkserver[2556]: testing chunk: /mnt/mfschunk1/EF/chunk_0000000000005EEF_00000001.mfs May 27 08:37:00 backup1 mfsmaster[12869]: chunkservers status: May 27 08:37:00 backup1 mfsmaster[12869]: server 1 (ip: 192.168.3.21, port: 9422): usedspace: 7739314479104 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.71% May 27 08:37:00 backup1 mfsmaster[12869]: total: usedspace: 7739314479104 (7207.80 GiB), totalspace: 10944744390656 (10193.09 GiB), usage: 70.71% Can you help me what to do with it? Is this fuse, moosefs, kernel problem or something else? Thank you, tamas |
|
From: Robert <rsa...@ne...> - 2011-05-26 16:25:31
|
It seems like if the switch connecting some of your chunk servers to the master server acts up then you may slowly collect a few of these. Two questions: 1. How do you fix this? 2. How do I make the system a bit more robust? Increase time-outs etc.? Thanks Robert |
|
From: Dietmar M. <di...@pr...> - 2011-05-26 03:49:20
|
>
> On 13/05/2011 19:26, Richard Chute wrote:
> > 5385 open("/mfs/pve/images/101/vm-101-disk-1.raw",
> > O_RDWR|O_DIRECT|O_CLOEXEC) = -1 EINVAL (Invalid argument)
>
> I was wrong, I am using proxmox 1.7 with MooseFS. Today, after trying to
> ugrade to 1.8, I get the same error. :(
>
> There is probably a regression (or some conflicts with MooseFS) on one of the
> latest proxmox updated packages:
Seems Moosefs does not support O_DIRECT. Try to use another cache setting for your disk.
Edit the VM config, set 'cache=writethrough' for all disks (see 'man qm')
PVE 1.8 uses "cache=none" by default.
- Dietmar
|
|
From: WK <wk...@bn...> - 2011-05-26 00:41:22
|
On 5/23/2011 3:32 PM, W Kern wrote: > So now the CGI is showing 10,000+ chunks with a single copy (red), 2 > million+ chunks are now orange (2 copies) and the system is happily > increasing the 'green' 3 valid copy column. > > The problem is that it seems to be concentrating on the orange (2 copy) > files and ignoring the 10,000+ red ones that are most at risk. In the > last hour we've seen a few 'red' chunks > disappear but the vast majority of activity is occuring in the orange (2 > copy) column. > > Shouldn't the replication worry about the single copy files first? > > I also realize we could simply set the goal back to 2 let it finish that > up and THEN switch it to 3 but I'm curious as to what the community says. > Just a followup, after a day or so we still had "red" single copy chunks. Obviously the under goal routine doesn't look at how badly under goal a given chunk is. So we dropped the Goal back down to 2 and MFS immediately focused on the single copy chunks. The only problem observed was that shortly after dropping the goal back down to 2, the mount complained of connection issues and people were kicked out of their IMAP sessions. That condition returned to normal less than a minute later and no files were lost. Once the under goal of 2 was completed an hour or so later, we reset the Goal to 3 and in a few days we should be fully green. In the meantime, we have at least two copies and are not vulnerable to an additional failure. I would still suggest that the "under goal" routine might want to look first at those chunks that are more severely out of goal, then go back and fix the others. Assuming that doesn't impact overall performance. -bill |
|
From: Giovanni T. <me...@gi...> - 2011-05-25 16:21:02
|
Hi Richard,
On 13/05/2011 19:26, Richard Chute wrote:
> 5385 open("/mfs/pve/images/101/vm-101-disk-1.raw",
> O_RDWR|O_DIRECT|O_CLOEXEC) = -1 EINVAL (Invalid argument)
I was wrong, I am using proxmox 1.7 with MooseFS. Today, after trying to
ugrade to 1.8, I get the same error. :(
There is probably a regression (or some conflicts with MooseFS) on one
of the latest proxmox updated packages:
[AGGIORNATO] ksm-control-daemon 1.0-4 -> 1.0-5
[AGGIORNATO] libpve-storage-perl 1.0-16 -> 1.0-17
[AGGIORNATO] proxmox-ve-2.6.32 1.7-30 -> 1.8-33
[AGGIORNATO] pve-firmware 1.0-10 -> 1.0-11
[AGGIORNATO] pve-kernel-2.6.32-4-pve 2.6.32-30 -> 2.6.32-33
[AGGIORNATO] pve-manager 1.7-11 -> 1.8-17
[AGGIORNATO] pve-qemu-kvm 0.13.0-3 -> 0.14.0-3
[AGGIORNATO] qemu-server 1.1-28 -> 1.1-30
[AGGIORNATO] vzctl 3.0.24-1pve4 -> 3.0.26-1pve4
[AGGIORNATO] vzdump 1.2-10 -> 1.2-12
[AGGIORNATO] vzprocps 2.0.11-1dso2 -> 2.0.11-2
The only thing I tried, it's to manually downgrade pve-qemu-kvm to 0.13,
but nohting changed. After, I removed proxmox, modified my sources.list
to point to pve1.7 version, and reinstall, and everything was working again.
I'm cc-ing to pve mailing list too.
Regards,
--
Giovanni Toraldo
http://gionn.net/
|
84 messages has been excluded from this view by a project administrator.
