Menu
▾
▴
moosefs-users — Mailing list for MooseFS Users
You can subscribe to this list here.
| 2009 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
(4) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2010 |
Jan
(20) |
Feb
(11) |
Mar
(11) |
Apr
(9) |
May
(22) |
Jun
(85) |
Jul
(94) |
Aug
(80) |
Sep
(72) |
Oct
(64) |
Nov
(69) |
Dec
(89) |
| 2011 |
Jan
(72) |
Feb
(109) |
Mar
(116) |
Apr
(117) |
May
(117) |
Jun
(102) |
Jul
(91) |
Aug
(72) |
Sep
(51) |
Oct
(41) |
Nov
(55) |
Dec
(74) |
| 2012 |
Jan
(45) |
Feb
(77) |
Mar
(99) |
Apr
(113) |
May
(132) |
Jun
(75) |
Jul
(70) |
Aug
(58) |
Sep
(58) |
Oct
(37) |
Nov
(51) |
Dec
(15) |
| 2013 |
Jan
(28) |
Feb
(16) |
Mar
(25) |
Apr
(38) |
May
(23) |
Jun
(39) |
Jul
(42) |
Aug
(19) |
Sep
(41) |
Oct
(31) |
Nov
(18) |
Dec
(18) |
| 2014 |
Jan
(17) |
Feb
(19) |
Mar
(39) |
Apr
(16) |
May
(10) |
Jun
(13) |
Jul
(17) |
Aug
(13) |
Sep
(8) |
Oct
(53) |
Nov
(23) |
Dec
(7) |
| 2015 |
Jan
(35) |
Feb
(13) |
Mar
(14) |
Apr
(56) |
May
(8) |
Jun
(18) |
Jul
(26) |
Aug
(33) |
Sep
(40) |
Oct
(37) |
Nov
(24) |
Dec
(20) |
| 2016 |
Jan
(38) |
Feb
(20) |
Mar
(25) |
Apr
(14) |
May
(6) |
Jun
(36) |
Jul
(27) |
Aug
(19) |
Sep
(36) |
Oct
(24) |
Nov
(15) |
Dec
(16) |
| 2017 |
Jan
(8) |
Feb
(13) |
Mar
(17) |
Apr
(20) |
May
(28) |
Jun
(10) |
Jul
(20) |
Aug
(3) |
Sep
(18) |
Oct
(8) |
Nov
|
Dec
(5) |
| 2018 |
Jan
(15) |
Feb
(9) |
Mar
(12) |
Apr
(7) |
May
(123) |
Jun
(41) |
Jul
|
Aug
(14) |
Sep
|
Oct
(15) |
Nov
|
Dec
(7) |
| 2019 |
Jan
(2) |
Feb
(9) |
Mar
(2) |
Apr
(9) |
May
|
Jun
|
Jul
(2) |
Aug
|
Sep
(6) |
Oct
(1) |
Nov
(12) |
Dec
(2) |
| 2020 |
Jan
(2) |
Feb
|
Mar
|
Apr
(3) |
May
|
Jun
(4) |
Jul
(4) |
Aug
(1) |
Sep
(18) |
Oct
(2) |
Nov
|
Dec
|
| 2021 |
Jan
|
Feb
(3) |
Mar
|
Apr
|
May
|
Jun
|
Jul
(6) |
Aug
|
Sep
(5) |
Oct
(5) |
Nov
(3) |
Dec
|
| 2022 |
Jan
|
Feb
|
Mar
(3) |
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Davies L. <dav...@gm...> - 2012-03-13 03:37:51
|
On Tue, Mar 13, 2012 at 10:21 AM, Ken <ken...@gm...> wrote:
> Hi, Davies and Moosefs
>
> I agree Davies' opinion in previous mail, undelete metadata.mfs.bak maybe
> is the best way.
>
> Recovery is very simply.
>
> After downloaded the damaged metadata from ChenGang, I began digging the
> code, comment a few lines(filesystem.c) of "return -1" in fs_loadnodes,
> fs_loadedges, chunk_load(procedures in restore).
> And print some count, like fsnodes count, edge count.., now we know inode
> information is complete, lost a half of edge, chunks totally lost.
>
> The most important information is inode ==> chunk id(s). It's complete,
> data will be remained.
>
Great work I just did not notice that the relations between inode and chunk
id is in nodes,
I thought they are in chunk block.
I'm very sorry for the wrong conclusion for Chen Gang, he maybe will lost
data without you help :-(
And thank you for the wonderful hack.
Then collect all the chunks id, version info from disk of chunkserver, and
> write(use a python script) it to a single file(chunks.bin) which format
> same as metadata.mfs. Here, I made a mistake because of duplicate chunks
> via goal 2.
>
> Make some dirty change of chunk_load(in chunks.c), load chunks.bin instead
> of "metadata.mfs"
> At last execute "mfsmetarestore -o metadata.mfs -m metadata.mfs.part". We
> got the metadata.mfs.
>
> This took me almost 4 hours.
>
Great efficience. He should buy you a drink :)
>
> In the accident, I notice something:
> a. The reason is disk full of master server, Why damaged occured in
> metalogger? Take a close look in fs_storeall(filesystem.c). Why unlink
> ("metadata.mfs") after write metadata failed?
>
The upcoming version maybe already fixed this issue, need more check.
> b. Keep more copies metadata.mfs in metalogger and mfsmaster maybe good.
>
The upcoming version has this feature, and I have patched my cluster also.
> c. Split the huge metadata.mfs to 3 files: inode, edge, chunk maybe be
> benefet in diagnosing, performance optimization. I guess.
>
> Any suggestion?
>
> Best Regards.
> -Ken
>
>
>
>
> On Tue, Mar 13, 2012 at 9:53 AM, Davies Liu <dav...@gm...> wrote:
>
>> Hi，
>>
>> congratulation !
>>
>> Can you show some details about how to recover the data back ?
>>
>> Davies
>>
>>
>> On Tue, Mar 13, 2012 at 4:24 AM, 陈钢 <yik...@gm...> wrote:
>>
>>> Hi all. I got some message to report.
>>>
>>> The situation I faced is that I got my metadata.mfs broken when
>>> "mfsmaster restart" include the metadata.mfs on mfsmetalogger.
>>>
>>> In fact, the metadata on mfsmetalogger is "more broken" than the
>>> metadata on mfsmaster.That is wired.
>>>
>>> So, I got all my files back but half of them lost their filename.
>>>
>>> Lucky,ken...@gm... helped me,and I have a SQLite file which
>>> stored filesize and filename in it.I have already restored my important
>>> data 90 percent now.
>>>
>>> And , I wrote some script helps me cp the metadata.mfs then rsync it to
>>> another server every hour. just like then suggestion in
>>> http://www.moosefs.org/moosefs-faq.html#metadata-backup.
>>>
>>> Metadata is really important , it worse a incremental backup.
>>>
>>>
>>> 2012/3/9 陈钢 <yik...@gm...>
>>>
>>>> Maybe, 80% data can be found. i still trying.
>>>> i tried restore every metadata.mfs i have, just not work.
>>>>
>>>>
>>>> 2012/3/8 Olivier Thibault <Oli...@lm...>
>>>>
>>>>> Hi,
>>>>>
>>>>> Did you solve your problem ?
>>>>> I had few days ago a mfsmaster crash which went out of memory.
>>>>> When I tried to restart, it crashed saying that there was no
>>>>> metadata.mfs file.
>>>>> I tried "mfsmetarestore -a". It didn't work, saying that
>>>>> metadata.mfs.back was corrupted.
>>>>> There was a metadata.mfs.back file and a metadata.mfs.back.tmp file.
>>>>> metadata.mfs.back was half the size it should be.
>>>>> I restored from a daily backup the metadata.mfs file, then ran again
>>>>> 'mfsmetarestore -a', and this time, it worked. I could then start mfsmaster
>>>>> successfully.
>>>>> Did you tried that ? I mean, just restore the latest working
>>>>> metadata.mfs file ?
>>>>>
>>>>> HTH.
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Olivier
>>>>>
>>>>>
>>>>>
>>>>> Le 07/03/12 04:20, 陈钢 a écrit :
>>>>>
>>>>>> In master`s metadata.mfs,i saw nodes part is complete, part of
>>>>>> names, no free,
>>>>>> no chunks..
>>>>>>
>>>>>> 2012/3/7 Davies Liu <dav...@gm... <mailto:
>>>>>> dav...@gm...>>
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I had try to recover it from metadata.mfs and
>>>>>> metadata_ml.mfs.back, but failed.
>>>>>>
>>>>>> Because disk is full, mfsmaster had not dump all the metadata into
>>>>>> disks,
>>>>>> had part of nodes in metadata.mfs, no names, no edges, no chunks.
>>>>>> No hope to recover from the broken metadata.mfs, it's too short.
>>>>>>
>>>>>> The changelogs are also helpless, only in two days.
>>>>>>
>>>>>> The only hope is try to undelete the previous metadata_ml.mfs.back
>>>>>> from
>>>>>> metalogger machine, Chenggang had also failed, with extundelete
>>>>>> and etx3grep,
>>>>>> maybe some experts can archive this.
>>>>>>
>>>>>> The final options is to GUESS the relation between files and
>>>>>> chunks by
>>>>>> chunk id and size fo files, if data lost can not been afforded.
>>>>>> Each chunk is
>>>>>> combined with crc checksum and real data, if we know the relation
>>>>>> between
>>>>>> files and chunksever, then we can get data back. Or we can
>>>>>> contruct the
>>>>>> metadata according to the GUESS, the using mfsmaster to recover
>>>>>> them.
>>>>>>
>>>>>> Davies
>>>>>>
>>>>>> On Wed, Mar 7, 2012 at 10:56 AM, Ken <ken...@gm...
>>>>>> <mailto:ken...@gm...>> wrote:
>>>>>>
>>>>>> Hi, chengang
>>>>>>
>>>>>> I think you should try more, and post detail here. Someone
>>>>>> must resolve it.
>>>>>> Maybe you will lost some data in last few minutes, but 250T
>>>>>> should be saved.
>>>>>>
>>>>>> At first, BACKUP all files:
>>>>>> /var/lib/mfs/* on master
>>>>>> /var/lib/mfs/* on mfsmetalogger
>>>>>>
>>>>>> about restore error:
>>>>>>
>>>>>> file 'metadata.mfs.back' not found - will try
>>>>>> 'metadata_ml.mfs.back'
>>>>>> instead
>>>>>> loading objects (files,directories,etc.) ... loading node:
>>>>>> read
>>>>>> error: ENOENT (No such file or directory)
>>>>>> error
>>>>>> can't read metadata from file: .//metadata_ml.mfs.back
>>>>>>
>>>>>> How did you run mfsmetarestore? add -d options?
>>>>>> If stat(datapath + metadata_ml.mfs.back) fail, these error
>>>>>> will occur.
>>>>>> Maybe use strace will show why stat fail exactly.
>>>>>>
>>>>>> ps: I am in Beijing now and I can provide more help.
>>>>>>
>>>>>> HTH
>>>>>>
>>>>>> -Ken
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 7, 2012 at 10:21 AM, 陈钢 <yik...@gm...
>>>>>> <mailto:yik...@gm...**>> wrote:
>>>>>>
>>>>>> can not start mfsmaster with the file "78962688 Mar 6
>>>>>> 17:18
>>>>>> metadata.mfs "..
>>>>>> i tried that. :(
>>>>>>
>>>>>> 2012/3/7 Ricardo J. Barberis <
>>>>>> ric...@da...
>>>>>> <mailto:ricardo.barberis@**dattatec.com<ric...@da...>
>>>>>> >>
>>>>>>
>>>>>>
>>>>>> El Martes 06/03/2012, 陈钢 escribió:
>>>>>> > on master
>>>>>> [ ... ]
>>>>>> > -rw-r----- 1 mfs mfs 78962688 Mar 6 17:18
>>>>>> metadata.mfs
>>>>>> > -rw-r--r-- 1 root root 8 Jul 4 2011
>>>>>> metadata.mfs.empty
>>>>>> > -rw-r----- 1 mfs mfs 5984 Mar 6 12:00
>>>>>> sessions.mfs
>>>>>> > -rw-r----- 1 mfs mfs 0 Mar 6 16:46
>>>>>> sessions.mfs.tmp
>>>>>> > -rw-r----- 1 mfs mfs 131072 Mar 6 17:18
>>>>>> stats.mfs
>>>>>>
>>>>>> You have /var/lib/mfs/metadata.mfs on the master, it
>>>>>> might not
>>>>>> be corrupt
>>>>>> after all?
>>>>>>
>>>>>> I'd suggest:
>>>>>>
>>>>>> - backup /var/lib/mfs to another disk/server (for
>>>>>> later recovery
>>>>>> if needed)
>>>>>> - make sure you have free space in your main disk
>>>>>> - then simply try to start mfsmaster
>>>>>> - check mfs.cgi (web interface) if it looks OK
>>>>>>
>>>>>>
>>>>>> BUT: if you can, wait for confirmation from Michał
>>>>>> Borychowski
>>>>>> first, in case
>>>>>> what I'm telling you is not safe.
>>>>>>
>>>>>>
>>>>>> (BTW: You have Reply-To set to che...@cp...
>>>>>> <mailto:che...@cp...>, I don't know if that's
>>>>>>
>>>>>> intentional on your part)
>>>>>>
>>>>>> Hope it helps,
>>>>>> --
>>>>>> Ricardo J. Barberis
>>>>>> Senior SysAdmin / ITI
>>>>>> Dattatec.com :: Soluciones de Web Hosting
>>>>>> Tu Hosting hecho Simple!
>>>>>>
>>>>>> ------------------------------**------------
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------**
>>>>>> ------------------------------**------------------
>>>>>> Virtualization & Cloud Management Using Capacity Planning
>>>>>> Cloud computing makes use of virtualization - but cloud
>>>>>> computing
>>>>>> also focuses on allowing computing to be delivered as a
>>>>>> service.
>>>>>> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/>
>>>>>>
>>>>>> ______________________________**_________________
>>>>>> moosefs-users mailing list
>>>>>> moosefs-users@lists.**sourceforge.net<moo...@li...>
>>>>>> <mailto:moosefs-users@lists.**sourceforge.net<moo...@li...>
>>>>>> >
>>>>>>
>>>>>> https://lists.sourceforge.net/**
>>>>>> lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------**------------------------------
>>>>>> **------------------
>>>>>> Virtualization & Cloud Management Using Capacity Planning
>>>>>> Cloud computing makes use of virtualization - but cloud
>>>>>> computing
>>>>>> also focuses on allowing computing to be delivered as a
>>>>>> service.
>>>>>> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/>
>>>>>> ______________________________**_________________
>>>>>> moosefs-users mailing list
>>>>>> moosefs-users@lists.**sourceforge.net<moo...@li...>
>>>>>> <mailto:moosefs-users@lists.**sourceforge.net<moo...@li...>
>>>>>> >
>>>>>>
>>>>>> https://lists.sourceforge.net/**lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> - Davies
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------**------------------------------**
>>>>>> ------------------
>>>>>> Virtualization& Cloud Management Using Capacity Planning
>>>>>>
>>>>>> Cloud computing makes use of virtualization - but cloud computing
>>>>>> also focuses on allowing computing to be delivered as a service.
>>>>>> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ______________________________**_________________
>>>>>> moosefs-users mailing list
>>>>>> moosefs-users@lists.**sourceforge.net<moo...@li...>
>>>>>> https://lists.sourceforge.net/**lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> - Davies
>>
>
>
--
- Davies
|
|
From: Ken <ken...@gm...> - 2012-03-13 02:21:58
|
Hi, Davies and Moosefs
I agree Davies' opinion in previous mail, undelete metadata.mfs.bak maybe
is the best way.
Recovery is very simply.
After downloaded the damaged metadata from ChenGang, I began digging the
code, comment a few lines(filesystem.c) of "return -1" in fs_loadnodes,
fs_loadedges, chunk_load(procedures in restore).
And print some count, like fsnodes count, edge count.., now we know inode
information is complete, lost a half of edge, chunks totally lost.
The most important information is inode ==> chunk id(s). It's complete,
data will be remained.
Then collect all the chunks id, version info from disk of chunkserver, and
write(use a python script) it to a single file(chunks.bin) which format
same as metadata.mfs. Here, I made a mistake because of duplicate chunks
via goal 2.
Make some dirty change of chunk_load(in chunks.c), load chunks.bin instead
of "metadata.mfs"
At last execute "mfsmetarestore -o metadata.mfs -m metadata.mfs.part". We
got the metadata.mfs.
This took me almost 4 hours.
In the accident, I notice something:
a. The reason is disk full of master server, Why damaged occured in
metalogger? Take a close look in fs_storeall(filesystem.c). Why unlink
("metadata.mfs") after write metadata failed?
b. Keep more copies metadata.mfs in metalogger and mfsmaster maybe good.
c. Split the huge metadata.mfs to 3 files: inode, edge, chunk maybe be
benefet in diagnosing, performance optimization. I guess.
Any suggestion?
Best Regards.
-Ken
On Tue, Mar 13, 2012 at 9:53 AM, Davies Liu <dav...@gm...> wrote:
> Hi，
>
> congratulation !
>
> Can you show some details about how to recover the data back ?
>
> Davies
>
>
> On Tue, Mar 13, 2012 at 4:24 AM, 陈钢 <yik...@gm...> wrote:
>
>> Hi all. I got some message to report.
>>
>> The situation I faced is that I got my metadata.mfs broken when
>> "mfsmaster restart" include the metadata.mfs on mfsmetalogger.
>>
>> In fact, the metadata on mfsmetalogger is "more broken" than the metadata
>> on mfsmaster.That is wired.
>>
>> So, I got all my files back but half of them lost their filename.
>>
>> Lucky,ken...@gm... helped me,and I have a SQLite file which
>> stored filesize and filename in it.I have already restored my important
>> data 90 percent now.
>>
>> And , I wrote some script helps me cp the metadata.mfs then rsync it to
>> another server every hour. just like then suggestion in
>> http://www.moosefs.org/moosefs-faq.html#metadata-backup.
>>
>> Metadata is really important , it worse a incremental backup.
>>
>>
>> 2012/3/9 陈钢 <yik...@gm...>
>>
>>> Maybe, 80% data can be found. i still trying.
>>> i tried restore every metadata.mfs i have, just not work.
>>>
>>>
>>> 2012/3/8 Olivier Thibault <Oli...@lm...>
>>>
>>>> Hi,
>>>>
>>>> Did you solve your problem ?
>>>> I had few days ago a mfsmaster crash which went out of memory.
>>>> When I tried to restart, it crashed saying that there was no
>>>> metadata.mfs file.
>>>> I tried "mfsmetarestore -a". It didn't work, saying that
>>>> metadata.mfs.back was corrupted.
>>>> There was a metadata.mfs.back file and a metadata.mfs.back.tmp file.
>>>> metadata.mfs.back was half the size it should be.
>>>> I restored from a daily backup the metadata.mfs file, then ran again
>>>> 'mfsmetarestore -a', and this time, it worked. I could then start mfsmaster
>>>> successfully.
>>>> Did you tried that ? I mean, just restore the latest working
>>>> metadata.mfs file ?
>>>>
>>>> HTH.
>>>>
>>>> Best regards,
>>>>
>>>> Olivier
>>>>
>>>>
>>>>
>>>> Le 07/03/12 04:20, 陈钢 a écrit :
>>>>
>>>>> In master`s metadata.mfs,i saw nodes part is complete, part of names,
>>>>> no free,
>>>>> no chunks..
>>>>>
>>>>> 2012/3/7 Davies Liu <dav...@gm... <mailto:dav...@gm...
>>>>> >>
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I had try to recover it from metadata.mfs and metadata_ml.mfs.back,
>>>>> but failed.
>>>>>
>>>>> Because disk is full, mfsmaster had not dump all the metadata into
>>>>> disks,
>>>>> had part of nodes in metadata.mfs, no names, no edges, no chunks.
>>>>> No hope to recover from the broken metadata.mfs, it's too short.
>>>>>
>>>>> The changelogs are also helpless, only in two days.
>>>>>
>>>>> The only hope is try to undelete the previous metadata_ml.mfs.back
>>>>> from
>>>>> metalogger machine, Chenggang had also failed, with extundelete and
>>>>> etx3grep,
>>>>> maybe some experts can archive this.
>>>>>
>>>>> The final options is to GUESS the relation between files and chunks
>>>>> by
>>>>> chunk id and size fo files, if data lost can not been afforded.
>>>>> Each chunk is
>>>>> combined with crc checksum and real data, if we know the relation
>>>>> between
>>>>> files and chunksever, then we can get data back. Or we can contruct
>>>>> the
>>>>> metadata according to the GUESS, the using mfsmaster to recover
>>>>> them.
>>>>>
>>>>> Davies
>>>>>
>>>>> On Wed, Mar 7, 2012 at 10:56 AM, Ken <ken...@gm...
>>>>> <mailto:ken...@gm...>> wrote:
>>>>>
>>>>> Hi, chengang
>>>>>
>>>>> I think you should try more, and post detail here. Someone must
>>>>> resolve it.
>>>>> Maybe you will lost some data in last few minutes, but 250T
>>>>> should be saved.
>>>>>
>>>>> At first, BACKUP all files:
>>>>> /var/lib/mfs/* on master
>>>>> /var/lib/mfs/* on mfsmetalogger
>>>>>
>>>>> about restore error:
>>>>>
>>>>> file 'metadata.mfs.back' not found - will try
>>>>> 'metadata_ml.mfs.back'
>>>>> instead
>>>>> loading objects (files,directories,etc.) ... loading node:
>>>>> read
>>>>> error: ENOENT (No such file or directory)
>>>>> error
>>>>> can't read metadata from file: .//metadata_ml.mfs.back
>>>>>
>>>>> How did you run mfsmetarestore? add -d options?
>>>>> If stat(datapath + metadata_ml.mfs.back) fail, these error will
>>>>> occur.
>>>>> Maybe use strace will show why stat fail exactly.
>>>>>
>>>>> ps: I am in Beijing now and I can provide more help.
>>>>>
>>>>> HTH
>>>>>
>>>>> -Ken
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Mar 7, 2012 at 10:21 AM, 陈钢 <yik...@gm...
>>>>> <mailto:yik...@gm...**>> wrote:
>>>>>
>>>>> can not start mfsmaster with the file "78962688 Mar 6 17:18
>>>>> metadata.mfs "..
>>>>> i tried that. :(
>>>>>
>>>>> 2012/3/7 Ricardo J. Barberis <ric...@da...
>>>>> <mailto:ricardo.barberis@**dattatec.com<ric...@da...>
>>>>> >>
>>>>>
>>>>>
>>>>> El Martes 06/03/2012, 陈钢 escribió:
>>>>> > on master
>>>>> [ ... ]
>>>>> > -rw-r----- 1 mfs mfs 78962688 Mar 6 17:18
>>>>> metadata.mfs
>>>>> > -rw-r--r-- 1 root root 8 Jul 4 2011
>>>>> metadata.mfs.empty
>>>>> > -rw-r----- 1 mfs mfs 5984 Mar 6 12:00
>>>>> sessions.mfs
>>>>> > -rw-r----- 1 mfs mfs 0 Mar 6 16:46
>>>>> sessions.mfs.tmp
>>>>> > -rw-r----- 1 mfs mfs 131072 Mar 6 17:18
>>>>> stats.mfs
>>>>>
>>>>> You have /var/lib/mfs/metadata.mfs on the master, it
>>>>> might not
>>>>> be corrupt
>>>>> after all?
>>>>>
>>>>> I'd suggest:
>>>>>
>>>>> - backup /var/lib/mfs to another disk/server (for later
>>>>> recovery
>>>>> if needed)
>>>>> - make sure you have free space in your main disk
>>>>> - then simply try to start mfsmaster
>>>>> - check mfs.cgi (web interface) if it looks OK
>>>>>
>>>>>
>>>>> BUT: if you can, wait for confirmation from Michał
>>>>> Borychowski
>>>>> first, in case
>>>>> what I'm telling you is not safe.
>>>>>
>>>>>
>>>>> (BTW: You have Reply-To set to che...@cp...
>>>>> <mailto:che...@cp...>, I don't know if that's
>>>>>
>>>>> intentional on your part)
>>>>>
>>>>> Hope it helps,
>>>>> --
>>>>> Ricardo J. Barberis
>>>>> Senior SysAdmin / ITI
>>>>> Dattatec.com :: Soluciones de Web Hosting
>>>>> Tu Hosting hecho Simple!
>>>>>
>>>>> ------------------------------**------------
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------**
>>>>> ------------------------------**------------------
>>>>> Virtualization & Cloud Management Using Capacity Planning
>>>>> Cloud computing makes use of virtualization - but cloud
>>>>> computing
>>>>> also focuses on allowing computing to be delivered as a
>>>>> service.
>>>>> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/>
>>>>>
>>>>> ______________________________**_________________
>>>>> moosefs-users mailing list
>>>>> moosefs-users@lists.**sourceforge.net<moo...@li...>
>>>>> <mailto:moosefs-users@lists.**sourceforge.net<moo...@li...>
>>>>> >
>>>>>
>>>>> https://lists.sourceforge.net/**
>>>>> lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------**------------------------------*
>>>>> *------------------
>>>>> Virtualization & Cloud Management Using Capacity Planning
>>>>> Cloud computing makes use of virtualization - but cloud
>>>>> computing
>>>>> also focuses on allowing computing to be delivered as a service.
>>>>> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/>
>>>>> ______________________________**_________________
>>>>> moosefs-users mailing list
>>>>> moosefs-users@lists.**sourceforge.net<moo...@li...>
>>>>> <mailto:moosefs-users@lists.**sourceforge.net<moo...@li...>
>>>>> >
>>>>>
>>>>> https://lists.sourceforge.net/**lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> - Davies
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------**------------------------------**
>>>>> ------------------
>>>>> Virtualization& Cloud Management Using Capacity Planning
>>>>>
>>>>> Cloud computing makes use of virtualization - but cloud computing
>>>>> also focuses on allowing computing to be delivered as a service.
>>>>> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/>
>>>>>
>>>>>
>>>>>
>>>>> ______________________________**_________________
>>>>> moosefs-users mailing list
>>>>> moosefs-users@lists.**sourceforge.net<moo...@li...>
>>>>> https://lists.sourceforge.net/**lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>
>
> --
> - Davies
>
|
|
From: Davies L. <dav...@gm...> - 2012-03-13 01:53:46
|
Hi， congratulation ! Can you show some details about how to recover the data back ? Davies On Tue, Mar 13, 2012 at 4:24 AM, 陈钢 <yik...@gm...> wrote: > Hi all. I got some message to report. > > The situation I faced is that I got my metadata.mfs broken when "mfsmaster > restart" include the metadata.mfs on mfsmetalogger. > > In fact, the metadata on mfsmetalogger is "more broken" than the metadata > on mfsmaster.That is wired. > > So, I got all my files back but half of them lost their filename. > > Lucky,ken...@gm... helped me,and I have a SQLite file which stored > filesize and filename in it.I have already restored my important data 90 > percent now. > > And , I wrote some script helps me cp the metadata.mfs then rsync it to > another server every hour. just like then suggestion in > http://www.moosefs.org/moosefs-faq.html#metadata-backup. > > Metadata is really important , it worse a incremental backup. > > > 2012/3/9 陈钢 <yik...@gm...> > >> Maybe, 80% data can be found. i still trying. >> i tried restore every metadata.mfs i have, just not work. >> >> >> 2012/3/8 Olivier Thibault <Oli...@lm...> >> >>> Hi, >>> >>> Did you solve your problem ? >>> I had few days ago a mfsmaster crash which went out of memory. >>> When I tried to restart, it crashed saying that there was no >>> metadata.mfs file. >>> I tried "mfsmetarestore -a". It didn't work, saying that >>> metadata.mfs.back was corrupted. >>> There was a metadata.mfs.back file and a metadata.mfs.back.tmp file. >>> metadata.mfs.back was half the size it should be. >>> I restored from a daily backup the metadata.mfs file, then ran again >>> 'mfsmetarestore -a', and this time, it worked. I could then start mfsmaster >>> successfully. >>> Did you tried that ? I mean, just restore the latest working >>> metadata.mfs file ? >>> >>> HTH. >>> >>> Best regards, >>> >>> Olivier >>> >>> >>> >>> Le 07/03/12 04:20, 陈钢 a écrit : >>> >>>> In master`s metadata.mfs,i saw nodes part is complete, part of names, >>>> no free, >>>> no chunks.. >>>> >>>> 2012/3/7 Davies Liu <dav...@gm... <mailto:dav...@gm... >>>> >> >>>> >>>> >>>> Hi, >>>> >>>> I had try to recover it from metadata.mfs and metadata_ml.mfs.back, >>>> but failed. >>>> >>>> Because disk is full, mfsmaster had not dump all the metadata into >>>> disks, >>>> had part of nodes in metadata.mfs, no names, no edges, no chunks. >>>> No hope to recover from the broken metadata.mfs, it's too short. >>>> >>>> The changelogs are also helpless, only in two days. >>>> >>>> The only hope is try to undelete the previous metadata_ml.mfs.back >>>> from >>>> metalogger machine, Chenggang had also failed, with extundelete and >>>> etx3grep, >>>> maybe some experts can archive this. >>>> >>>> The final options is to GUESS the relation between files and chunks >>>> by >>>> chunk id and size fo files, if data lost can not been afforded. Each >>>> chunk is >>>> combined with crc checksum and real data, if we know the relation >>>> between >>>> files and chunksever, then we can get data back. Or we can contruct >>>> the >>>> metadata according to the GUESS, the using mfsmaster to recover them. >>>> >>>> Davies >>>> >>>> On Wed, Mar 7, 2012 at 10:56 AM, Ken <ken...@gm... >>>> <mailto:ken...@gm...>> wrote: >>>> >>>> Hi, chengang >>>> >>>> I think you should try more, and post detail here. Someone must >>>> resolve it. >>>> Maybe you will lost some data in last few minutes, but 250T >>>> should be saved. >>>> >>>> At first, BACKUP all files: >>>> /var/lib/mfs/* on master >>>> /var/lib/mfs/* on mfsmetalogger >>>> >>>> about restore error: >>>> >>>> file 'metadata.mfs.back' not found - will try >>>> 'metadata_ml.mfs.back' >>>> instead >>>> loading objects (files,directories,etc.) ... loading node: >>>> read >>>> error: ENOENT (No such file or directory) >>>> error >>>> can't read metadata from file: .//metadata_ml.mfs.back >>>> >>>> How did you run mfsmetarestore? add -d options? >>>> If stat(datapath + metadata_ml.mfs.back) fail, these error will >>>> occur. >>>> Maybe use strace will show why stat fail exactly. >>>> >>>> ps: I am in Beijing now and I can provide more help. >>>> >>>> HTH >>>> >>>> -Ken >>>> >>>> >>>> >>>> On Wed, Mar 7, 2012 at 10:21 AM, 陈钢 <yik...@gm... >>>> <mailto:yik...@gm...**>> wrote: >>>> >>>> can not start mfsmaster with the file "78962688 Mar 6 17:18 >>>> metadata.mfs ".. >>>> i tried that. :( >>>> >>>> 2012/3/7 Ricardo J. Barberis <ric...@da... >>>> <mailto:ricardo.barberis@**dattatec.com<ric...@da...> >>>> >> >>>> >>>> >>>> El Martes 06/03/2012, 陈钢 escribió: >>>> > on master >>>> [ ... ] >>>> > -rw-r----- 1 mfs mfs 78962688 Mar 6 17:18 >>>> metadata.mfs >>>> > -rw-r--r-- 1 root root 8 Jul 4 2011 >>>> metadata.mfs.empty >>>> > -rw-r----- 1 mfs mfs 5984 Mar 6 12:00 >>>> sessions.mfs >>>> > -rw-r----- 1 mfs mfs 0 Mar 6 16:46 >>>> sessions.mfs.tmp >>>> > -rw-r----- 1 mfs mfs 131072 Mar 6 17:18 >>>> stats.mfs >>>> >>>> You have /var/lib/mfs/metadata.mfs on the master, it >>>> might not >>>> be corrupt >>>> after all? >>>> >>>> I'd suggest: >>>> >>>> - backup /var/lib/mfs to another disk/server (for later >>>> recovery >>>> if needed) >>>> - make sure you have free space in your main disk >>>> - then simply try to start mfsmaster >>>> - check mfs.cgi (web interface) if it looks OK >>>> >>>> >>>> BUT: if you can, wait for confirmation from Michał >>>> Borychowski >>>> first, in case >>>> what I'm telling you is not safe. >>>> >>>> >>>> (BTW: You have Reply-To set to che...@cp... >>>> <mailto:che...@cp...>, I don't know if that's >>>> >>>> intentional on your part) >>>> >>>> Hope it helps, >>>> -- >>>> Ricardo J. Barberis >>>> Senior SysAdmin / ITI >>>> Dattatec.com :: Soluciones de Web Hosting >>>> Tu Hosting hecho Simple! >>>> >>>> ------------------------------**------------ >>>> >>>> >>>> >>>> ------------------------------** >>>> ------------------------------**------------------ >>>> Virtualization & Cloud Management Using Capacity Planning >>>> Cloud computing makes use of virtualization - but cloud >>>> computing >>>> also focuses on allowing computing to be delivered as a >>>> service. >>>> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/> >>>> >>>> ______________________________**_________________ >>>> moosefs-users mailing list >>>> moosefs-users@lists.**sourceforge.net<moo...@li...> >>>> <mailto:moosefs-users@lists.**sourceforge.net<moo...@li...> >>>> > >>>> >>>> https://lists.sourceforge.net/**lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users> >>>> >>>> >>>> >>>> ------------------------------**------------------------------** >>>> ------------------ >>>> Virtualization & Cloud Management Using Capacity Planning >>>> Cloud computing makes use of virtualization - but cloud computing >>>> also focuses on allowing computing to be delivered as a service. >>>> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/> >>>> ______________________________**_________________ >>>> moosefs-users mailing list >>>> moosefs-users@lists.**sourceforge.net<moo...@li...> >>>> <mailto:moosefs-users@lists.**sourceforge.net<moo...@li...> >>>> > >>>> >>>> https://lists.sourceforge.net/**lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users> >>>> >>>> >>>> >>>> >>>> -- >>>> - Davies >>>> >>>> >>>> >>>> >>>> ------------------------------**------------------------------** >>>> ------------------ >>>> Virtualization& Cloud Management Using Capacity Planning >>>> >>>> Cloud computing makes use of virtualization - but cloud computing >>>> also focuses on allowing computing to be delivered as a service. >>>> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/> >>>> >>>> >>>> >>>> ______________________________**_________________ >>>> moosefs-users mailing list >>>> moosefs-users@lists.**sourceforge.net<moo...@li...> >>>> https://lists.sourceforge.net/**lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users> >>>> >>> >>> >>> >> > -- - Davies |
|
From: 陈钢 <yik...@gm...> - 2012-03-12 20:24:37
|
Hi all. I got some message to report. The situation I faced is that I got my metadata.mfs broken when "mfsmaster restart" include the metadata.mfs on mfsmetalogger. In fact, the metadata on mfsmetalogger is "more broken" than the metadata on mfsmaster.That is wired. So, I got all my files back but half of them lost their filename. Lucky,ken...@gm... helped me,and I have a SQLite file which stored filesize and filename in it.I have already restored my important data 90 percent now. And , I wrote some script helps me cp the metadata.mfs then rsync it to another server every hour. just like then suggestion in http://www.moosefs.org/moosefs-faq.html#metadata-backup. Metadata is really important , it worse a incremental backup. 2012/3/9 陈钢 <yik...@gm...> > Maybe, 80% data can be found. i still trying. > i tried restore every metadata.mfs i have, just not work. > > > 2012/3/8 Olivier Thibault <Oli...@lm...> > >> Hi, >> >> Did you solve your problem ? >> I had few days ago a mfsmaster crash which went out of memory. >> When I tried to restart, it crashed saying that there was no metadata.mfs >> file. >> I tried "mfsmetarestore -a". It didn't work, saying that >> metadata.mfs.back was corrupted. >> There was a metadata.mfs.back file and a metadata.mfs.back.tmp file. >> metadata.mfs.back was half the size it should be. >> I restored from a daily backup the metadata.mfs file, then ran again >> 'mfsmetarestore -a', and this time, it worked. I could then start mfsmaster >> successfully. >> Did you tried that ? I mean, just restore the latest working metadata.mfs >> file ? >> >> HTH. >> >> Best regards, >> >> Olivier >> >> >> >> Le 07/03/12 04:20, 陈钢 a écrit : >> >>> In master`s metadata.mfs,i saw nodes part is complete, part of names, >>> no free, >>> no chunks.. >>> >>> 2012/3/7 Davies Liu <dav...@gm... <mailto:dav...@gm...>> >>> >>> >>> Hi, >>> >>> I had try to recover it from metadata.mfs and metadata_ml.mfs.back, >>> but failed. >>> >>> Because disk is full, mfsmaster had not dump all the metadata into >>> disks, >>> had part of nodes in metadata.mfs, no names, no edges, no chunks. >>> No hope to recover from the broken metadata.mfs, it's too short. >>> >>> The changelogs are also helpless, only in two days. >>> >>> The only hope is try to undelete the previous metadata_ml.mfs.back >>> from >>> metalogger machine, Chenggang had also failed, with extundelete and >>> etx3grep, >>> maybe some experts can archive this. >>> >>> The final options is to GUESS the relation between files and chunks by >>> chunk id and size fo files, if data lost can not been afforded. Each >>> chunk is >>> combined with crc checksum and real data, if we know the relation >>> between >>> files and chunksever, then we can get data back. Or we can contruct >>> the >>> metadata according to the GUESS, the using mfsmaster to recover them. >>> >>> Davies >>> >>> On Wed, Mar 7, 2012 at 10:56 AM, Ken <ken...@gm... >>> <mailto:ken...@gm...>> wrote: >>> >>> Hi, chengang >>> >>> I think you should try more, and post detail here. Someone must >>> resolve it. >>> Maybe you will lost some data in last few minutes, but 250T >>> should be saved. >>> >>> At first, BACKUP all files: >>> /var/lib/mfs/* on master >>> /var/lib/mfs/* on mfsmetalogger >>> >>> about restore error: >>> >>> file 'metadata.mfs.back' not found - will try >>> 'metadata_ml.mfs.back' >>> instead >>> loading objects (files,directories,etc.) ... loading node: >>> read >>> error: ENOENT (No such file or directory) >>> error >>> can't read metadata from file: .//metadata_ml.mfs.back >>> >>> How did you run mfsmetarestore? add -d options? >>> If stat(datapath + metadata_ml.mfs.back) fail, these error will >>> occur. >>> Maybe use strace will show why stat fail exactly. >>> >>> ps: I am in Beijing now and I can provide more help. >>> >>> HTH >>> >>> -Ken >>> >>> >>> >>> On Wed, Mar 7, 2012 at 10:21 AM, 陈钢 <yik...@gm... >>> <mailto:yik...@gm...**>> wrote: >>> >>> can not start mfsmaster with the file "78962688 Mar 6 17:18 >>> metadata.mfs ".. >>> i tried that. :( >>> >>> 2012/3/7 Ricardo J. Barberis <ric...@da... >>> <mailto:ricardo.barberis@**dattatec.com<ric...@da...> >>> >> >>> >>> >>> El Martes 06/03/2012, 陈钢 escribió: >>> > on master >>> [ ... ] >>> > -rw-r----- 1 mfs mfs 78962688 Mar 6 17:18 >>> metadata.mfs >>> > -rw-r--r-- 1 root root 8 Jul 4 2011 >>> metadata.mfs.empty >>> > -rw-r----- 1 mfs mfs 5984 Mar 6 12:00 >>> sessions.mfs >>> > -rw-r----- 1 mfs mfs 0 Mar 6 16:46 >>> sessions.mfs.tmp >>> > -rw-r----- 1 mfs mfs 131072 Mar 6 17:18 >>> stats.mfs >>> >>> You have /var/lib/mfs/metadata.mfs on the master, it >>> might not >>> be corrupt >>> after all? >>> >>> I'd suggest: >>> >>> - backup /var/lib/mfs to another disk/server (for later >>> recovery >>> if needed) >>> - make sure you have free space in your main disk >>> - then simply try to start mfsmaster >>> - check mfs.cgi (web interface) if it looks OK >>> >>> >>> BUT: if you can, wait for confirmation from Michał >>> Borychowski >>> first, in case >>> what I'm telling you is not safe. >>> >>> >>> (BTW: You have Reply-To set to che...@cp... >>> <mailto:che...@cp...>, I don't know if that's >>> >>> intentional on your part) >>> >>> Hope it helps, >>> -- >>> Ricardo J. Barberis >>> Senior SysAdmin / ITI >>> Dattatec.com :: Soluciones de Web Hosting >>> Tu Hosting hecho Simple! >>> >>> ------------------------------**------------ >>> >>> >>> >>> ------------------------------** >>> ------------------------------**------------------ >>> Virtualization & Cloud Management Using Capacity Planning >>> Cloud computing makes use of virtualization - but cloud >>> computing >>> also focuses on allowing computing to be delivered as a >>> service. >>> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/> >>> >>> ______________________________**_________________ >>> moosefs-users mailing list >>> moosefs-users@lists.**sourceforge.net<moo...@li...> >>> <mailto:moosefs-users@lists.**sourceforge.net<moo...@li...> >>> > >>> >>> https://lists.sourceforge.net/**lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users> >>> >>> >>> >>> ------------------------------**------------------------------** >>> ------------------ >>> Virtualization & Cloud Management Using Capacity Planning >>> Cloud computing makes use of virtualization - but cloud computing >>> also focuses on allowing computing to be delivered as a service. >>> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/> >>> ______________________________**_________________ >>> moosefs-users mailing list >>> moosefs-users@lists.**sourceforge.net<moo...@li...> >>> <mailto:moosefs-users@lists.**sourceforge.net<moo...@li...> >>> > >>> >>> https://lists.sourceforge.net/**lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users> >>> >>> >>> >>> >>> -- >>> - Davies >>> >>> >>> >>> >>> ------------------------------**------------------------------** >>> ------------------ >>> Virtualization& Cloud Management Using Capacity Planning >>> >>> Cloud computing makes use of virtualization - but cloud computing >>> also focuses on allowing computing to be delivered as a service. >>> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/> >>> >>> >>> >>> ______________________________**_________________ >>> moosefs-users mailing list >>> moosefs-users@lists.**sourceforge.net<moo...@li...> >>> https://lists.sourceforge.net/**lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users> >>> >> >> >> > |
|
From: C. C. <c-...@uc...> - 2012-03-09 22:15:23
|
I've been running MooseFS as a scratch space on a small computing cluster to aggregate unused storage. It has been very robust, far more so than my experiments with OrangeFS and GlusterFS. The total size of the filesystem is 6.4TB with goal set to 2, so the effective usable space is 3.2TB. Last week the node running the mfsmaster had to be rebooted due to a runaway process which had nothing to do with MooseFS but which spiked the system load so that the node became unresponsive and mfschunkserver and mfsmount connections began to time out. The filesystem where mfsmaster stored its metadata/changelogs did not fill up, nor did the system run out of physical memory. After rebooting, the mfsmetarestore rebuilt the metadata.mfs file without incident. But when the mfsmaster process is started there are numerous messages of the type: master mfsmaster[26378]: chunkserver has nonexistent chunk (0000000002CC8487_00000001), so create it for future deletion After about 24-48 hours of this mfsmaster then abruptly terminates. I have tried using an earlier subset of changelog files and using the changelogs and metadata backup files on the metadata loggers, without any success and with the same messages followed by the mfsmaster crashing after 24+ hours. Is there anything else I can try? Since this is a scratch system it's not necessary to recover it, but I'd like to find out what went wrong. -- C. Chan <c-chan at uchicago.edu> GPG Public Key registered at pgp.mit.edu |
|
From: 陈钢 <yik...@gm...> - 2012-03-08 16:00:27
|
Maybe, 80% data can be found. i still trying. i tried restore every metadata.mfs i have, just not work. 2012/3/8 Olivier Thibault <Oli...@lm...> > Hi, > > Did you solve your problem ? > I had few days ago a mfsmaster crash which went out of memory. > When I tried to restart, it crashed saying that there was no metadata.mfs > file. > I tried "mfsmetarestore -a". It didn't work, saying that metadata.mfs.back > was corrupted. > There was a metadata.mfs.back file and a metadata.mfs.back.tmp file. > metadata.mfs.back was half the size it should be. > I restored from a daily backup the metadata.mfs file, then ran again > 'mfsmetarestore -a', and this time, it worked. I could then start mfsmaster > successfully. > Did you tried that ? I mean, just restore the latest working metadata.mfs > file ? > > HTH. > > Best regards, > > Olivier > > > > Le 07/03/12 04:20, 陈钢 a écrit : > >> In master`s metadata.mfs,i saw nodes part is complete, part of names, no >> free, >> no chunks.. >> >> 2012/3/7 Davies Liu <dav...@gm... <mailto:dav...@gm...>> >> >> >> Hi, >> >> I had try to recover it from metadata.mfs and metadata_ml.mfs.back, >> but failed. >> >> Because disk is full, mfsmaster had not dump all the metadata into >> disks, >> had part of nodes in metadata.mfs, no names, no edges, no chunks. >> No hope to recover from the broken metadata.mfs, it's too short. >> >> The changelogs are also helpless, only in two days. >> >> The only hope is try to undelete the previous metadata_ml.mfs.back from >> metalogger machine, Chenggang had also failed, with extundelete and >> etx3grep, >> maybe some experts can archive this. >> >> The final options is to GUESS the relation between files and chunks by >> chunk id and size fo files, if data lost can not been afforded. Each >> chunk is >> combined with crc checksum and real data, if we know the relation >> between >> files and chunksever, then we can get data back. Or we can contruct the >> metadata according to the GUESS, the using mfsmaster to recover them. >> >> Davies >> >> On Wed, Mar 7, 2012 at 10:56 AM, Ken <ken...@gm... >> <mailto:ken...@gm...>> wrote: >> >> Hi, chengang >> >> I think you should try more, and post detail here. Someone must >> resolve it. >> Maybe you will lost some data in last few minutes, but 250T should >> be saved. >> >> At first, BACKUP all files: >> /var/lib/mfs/* on master >> /var/lib/mfs/* on mfsmetalogger >> >> about restore error: >> >> file 'metadata.mfs.back' not found - will try >> 'metadata_ml.mfs.back' >> instead >> loading objects (files,directories,etc.) ... loading node: read >> error: ENOENT (No such file or directory) >> error >> can't read metadata from file: .//metadata_ml.mfs.back >> >> How did you run mfsmetarestore? add -d options? >> If stat(datapath + metadata_ml.mfs.back) fail, these error will >> occur. >> Maybe use strace will show why stat fail exactly. >> >> ps: I am in Beijing now and I can provide more help. >> >> HTH >> >> -Ken >> >> >> >> On Wed, Mar 7, 2012 at 10:21 AM, 陈钢 <yik...@gm... >> <mailto:yik...@gm...**>> wrote: >> >> can not start mfsmaster with the file "78962688 Mar 6 17:18 >> metadata.mfs ".. >> i tried that. :( >> >> 2012/3/7 Ricardo J. Barberis <ric...@da... >> <mailto:ricardo.barberis@**dattatec.com<ric...@da...> >> >> >> >> >> El Martes 06/03/2012, 陈钢 escribió: >> > on master >> [ ... ] >> > -rw-r----- 1 mfs mfs 78962688 Mar 6 17:18 >> metadata.mfs >> > -rw-r--r-- 1 root root 8 Jul 4 2011 >> metadata.mfs.empty >> > -rw-r----- 1 mfs mfs 5984 Mar 6 12:00 >> sessions.mfs >> > -rw-r----- 1 mfs mfs 0 Mar 6 16:46 >> sessions.mfs.tmp >> > -rw-r----- 1 mfs mfs 131072 Mar 6 17:18 stats.mfs >> >> You have /var/lib/mfs/metadata.mfs on the master, it might >> not >> be corrupt >> after all? >> >> I'd suggest: >> >> - backup /var/lib/mfs to another disk/server (for later >> recovery >> if needed) >> - make sure you have free space in your main disk >> - then simply try to start mfsmaster >> - check mfs.cgi (web interface) if it looks OK >> >> >> BUT: if you can, wait for confirmation from Michał >> Borychowski >> first, in case >> what I'm telling you is not safe. >> >> >> (BTW: You have Reply-To set to che...@cp... >> <mailto:che...@cp...>, I don't know if that's >> >> intentional on your part) >> >> Hope it helps, >> -- >> Ricardo J. Barberis >> Senior SysAdmin / ITI >> Dattatec.com :: Soluciones de Web Hosting >> Tu Hosting hecho Simple! >> >> ------------------------------**------------ >> >> >> >> ------------------------------**------------------------------ >> **------------------ >> Virtualization & Cloud Management Using Capacity Planning >> Cloud computing makes use of virtualization - but cloud >> computing >> also focuses on allowing computing to be delivered as a >> service. >> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/> >> >> ______________________________**_________________ >> moosefs-users mailing list >> moosefs-users@lists.**sourceforge.net<moo...@li...> >> <mailto:moosefs-users@lists.**sourceforge.net<moo...@li...> >> > >> >> https://lists.sourceforge.net/**lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users> >> >> >> >> ------------------------------**------------------------------** >> ------------------ >> Virtualization & Cloud Management Using Capacity Planning >> Cloud computing makes use of virtualization - but cloud computing >> also focuses on allowing computing to be delivered as a service. >> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/> >> ______________________________**_________________ >> moosefs-users mailing list >> moosefs-users@lists.**sourceforge.net<moo...@li...> >> <mailto:moosefs-users@lists.**sourceforge.net<moo...@li...> >> > >> >> https://lists.sourceforge.net/**lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users> >> >> >> >> >> -- >> - Davies >> >> >> >> >> ------------------------------**------------------------------** >> ------------------ >> Virtualization& Cloud Management Using Capacity Planning >> >> Cloud computing makes use of virtualization - but cloud computing >> also focuses on allowing computing to be delivered as a service. >> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/> >> >> >> >> ______________________________**_________________ >> moosefs-users mailing list >> moosefs-users@lists.**sourceforge.net<moo...@li...> >> https://lists.sourceforge.net/**lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users> >> > > > |
|
From: Olivier T. <Oli...@lm...> - 2012-03-08 13:17:40
|
Hi, Did you solve your problem ? I had few days ago a mfsmaster crash which went out of memory. When I tried to restart, it crashed saying that there was no metadata.mfs file. I tried "mfsmetarestore -a". It didn't work, saying that metadata.mfs.back was corrupted. There was a metadata.mfs.back file and a metadata.mfs.back.tmp file. metadata.mfs.back was half the size it should be. I restored from a daily backup the metadata.mfs file, then ran again 'mfsmetarestore -a', and this time, it worked. I could then start mfsmaster successfully. Did you tried that ? I mean, just restore the latest working metadata.mfs file ? HTH. Best regards, Olivier Le 07/03/12 04:20, 陈钢 a écrit : > In master`s metadata.mfs,i saw nodes part is complete, part of names, no free, > no chunks.. > > 2012/3/7 Davies Liu <dav...@gm... <mailto:dav...@gm...>> > > Hi, > > I had try to recover it from metadata.mfs and metadata_ml.mfs.back, but failed. > > Because disk is full, mfsmaster had not dump all the metadata into disks, > had part of nodes in metadata.mfs, no names, no edges, no chunks. > No hope to recover from the broken metadata.mfs, it's too short. > > The changelogs are also helpless, only in two days. > > The only hope is try to undelete the previous metadata_ml.mfs.back from > metalogger machine, Chenggang had also failed, with extundelete and etx3grep, > maybe some experts can archive this. > > The final options is to GUESS the relation between files and chunks by > chunk id and size fo files, if data lost can not been afforded. Each chunk is > combined with crc checksum and real data, if we know the relation between > files and chunksever, then we can get data back. Or we can contruct the > metadata according to the GUESS, the using mfsmaster to recover them. > > Davies > > On Wed, Mar 7, 2012 at 10:56 AM, Ken <ken...@gm... > <mailto:ken...@gm...>> wrote: > > Hi, chengang > > I think you should try more, and post detail here. Someone must resolve it. > Maybe you will lost some data in last few minutes, but 250T should be saved. > > At first, BACKUP all files: > /var/lib/mfs/* on master > /var/lib/mfs/* on mfsmetalogger > > about restore error: > > file 'metadata.mfs.back' not found - will try 'metadata_ml.mfs.back' > instead > loading objects (files,directories,etc.) ... loading node: read > error: ENOENT (No such file or directory) > error > can't read metadata from file: .//metadata_ml.mfs.back > > How did you run mfsmetarestore? add -d options? > If stat(datapath + metadata_ml.mfs.back) fail, these error will occur. > Maybe use strace will show why stat fail exactly. > > ps: I am in Beijing now and I can provide more help. > > HTH > > -Ken > > > > On Wed, Mar 7, 2012 at 10:21 AM, 陈钢 <yik...@gm... > <mailto:yik...@gm...>> wrote: > > can not start mfsmaster with the file "78962688 Mar 6 17:18 > metadata.mfs ".. > i tried that. :( > > 2012/3/7 Ricardo J. Barberis <ric...@da... > <mailto:ric...@da...>> > > El Martes 06/03/2012, 陈钢 escribió: > > on master > [ ... ] > > -rw-r----- 1 mfs mfs 78962688 Mar 6 17:18 metadata.mfs > > -rw-r--r-- 1 root root 8 Jul 4 2011 metadata.mfs.empty > > -rw-r----- 1 mfs mfs 5984 Mar 6 12:00 sessions.mfs > > -rw-r----- 1 mfs mfs 0 Mar 6 16:46 sessions.mfs.tmp > > -rw-r----- 1 mfs mfs 131072 Mar 6 17:18 stats.mfs > > You have /var/lib/mfs/metadata.mfs on the master, it might not > be corrupt > after all? > > I'd suggest: > > - backup /var/lib/mfs to another disk/server (for later recovery > if needed) > - make sure you have free space in your main disk > - then simply try to start mfsmaster > - check mfs.cgi (web interface) if it looks OK > > > BUT: if you can, wait for confirmation from Michał Borychowski > first, in case > what I'm telling you is not safe. > > > (BTW: You have Reply-To set to che...@cp... > <mailto:che...@cp...>, I don't know if that's > intentional on your part) > > Hope it helps, > -- > Ricardo J. Barberis > Senior SysAdmin / ITI > Dattatec.com :: Soluciones de Web Hosting > Tu Hosting hecho Simple! > > ------------------------------------------ > > > > ------------------------------------------------------------------------------ > Virtualization & Cloud Management Using Capacity Planning > Cloud computing makes use of virtualization - but cloud computing > also focuses on allowing computing to be delivered as a service. > http://www.accelacomm.com/jaw/sfnl/114/51521223/ > > _______________________________________________ > moosefs-users mailing list > moo...@li... > <mailto:moo...@li...> > https://lists.sourceforge.net/lists/listinfo/moosefs-users > > > > ------------------------------------------------------------------------------ > Virtualization & Cloud Management Using Capacity Planning > Cloud computing makes use of virtualization - but cloud computing > also focuses on allowing computing to be delivered as a service. > http://www.accelacomm.com/jaw/sfnl/114/51521223/ > _______________________________________________ > moosefs-users mailing list > moo...@li... > <mailto:moo...@li...> > https://lists.sourceforge.net/lists/listinfo/moosefs-users > > > > > -- > - Davies > > > > > ------------------------------------------------------------------------------ > Virtualization& Cloud Management Using Capacity Planning > Cloud computing makes use of virtualization - but cloud computing > also focuses on allowing computing to be delivered as a service. > http://www.accelacomm.com/jaw/sfnl/114/51521223/ > > > > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: 崔赢 <cui...@gm...> - 2012-03-08 09:32:24
|
cui...@in... |
|
From: 陈钢 <yik...@gm...> - 2012-03-07 03:20:43
|
In master`s metadata.mfs,i saw nodes part is complete, part of names, no free, no chunks.. 2012/3/7 Davies Liu <dav...@gm...> > Hi, > > I had try to recover it from metadata.mfs and metadata_ml.mfs.back, but > failed. > > Because disk is full, mfsmaster had not dump all the metadata into disks, > had part of nodes in metadata.mfs, no names, no edges, no chunks. > No hope to recover from the broken metadata.mfs, it's too short. > > The changelogs are also helpless, only in two days. > > The only hope is try to undelete the previous metadata_ml.mfs.back from > metalogger machine, Chenggang had also failed, with extundelete and > etx3grep, > maybe some experts can archive this. > > The final options is to GUESS the relation between files and chunks by > chunk id and size fo files, if data lost can not been afforded. Each chunk > is > combined with crc checksum and real data, if we know the relation between > files and chunksever, then we can get data back. Or we can contruct the > metadata according to the GUESS, the using mfsmaster to recover them. > > Davies > > On Wed, Mar 7, 2012 at 10:56 AM, Ken <ken...@gm...> wrote: > >> Hi, chengang >> >> I think you should try more, and post detail here. Someone must resolve >> it. >> Maybe you will lost some data in last few minutes, but 250T should be >> saved. >> >> At first, BACKUP all files: >> /var/lib/mfs/* on master >> /var/lib/mfs/* on mfsmetalogger >> >> about restore error: >> >> file 'metadata.mfs.back' not found - will try 'metadata_ml.mfs.back' >>> instead >>> loading objects (files,directories,etc.) ... loading node: read error: >>> ENOENT (No such file or directory) >>> error >>> can't read metadata from file: .//metadata_ml.mfs.back >>> >> How did you run mfsmetarestore? add -d options? >> If stat(datapath + metadata_ml.mfs.back) fail, these error will occur. >> Maybe use strace will show why stat fail exactly. >> >> ps: I am in Beijing now and I can provide more help. >> >> HTH >> >> -Ken >> >> >> >> On Wed, Mar 7, 2012 at 10:21 AM, 陈钢 <yik...@gm...> wrote: >> >>> can not start mfsmaster with the file "78962688 Mar 6 17:18 >>> metadata.mfs ".. >>> i tried that. :( >>> >>> 2012/3/7 Ricardo J. Barberis <ric...@da...> >>> >>> El Martes 06/03/2012, 陈钢 escribió: >>>> > on master >>>> [ ... ] >>>> > -rw-r----- 1 mfs mfs 78962688 Mar 6 17:18 metadata.mfs >>>> > -rw-r--r-- 1 root root 8 Jul 4 2011 metadata.mfs.empty >>>> > -rw-r----- 1 mfs mfs 5984 Mar 6 12:00 sessions.mfs >>>> > -rw-r----- 1 mfs mfs 0 Mar 6 16:46 sessions.mfs.tmp >>>> > -rw-r----- 1 mfs mfs 131072 Mar 6 17:18 stats.mfs >>>> >>>> You have /var/lib/mfs/metadata.mfs on the master, it might not be >>>> corrupt >>>> after all? >>>> >>>> I'd suggest: >>>> >>>> - backup /var/lib/mfs to another disk/server (for later recovery if >>>> needed) >>>> - make sure you have free space in your main disk >>>> - then simply try to start mfsmaster >>>> - check mfs.cgi (web interface) if it looks OK >>>> >>>> >>>> BUT: if you can, wait for confirmation from Michał Borychowski first, >>>> in case >>>> what I'm telling you is not safe. >>>> >>>> >>>> (BTW: You have Reply-To set to che...@cp..., I don't know if >>>> that's >>>> intentional on your part) >>>> >>>> Hope it helps, >>>> -- >>>> Ricardo J. Barberis >>>> Senior SysAdmin / ITI >>>> Dattatec.com :: Soluciones de Web Hosting >>>> Tu Hosting hecho Simple! >>>> >>>> ------------------------------------------ >>>> >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> Virtualization & Cloud Management Using Capacity Planning >>> Cloud computing makes use of virtualization - but cloud computing >>> also focuses on allowing computing to be delivered as a service. >>> http://www.accelacomm.com/jaw/sfnl/114/51521223/ >>> >>> _______________________________________________ >>> moosefs-users mailing list >>> moo...@li... >>> https://lists.sourceforge.net/lists/listinfo/moosefs-users >>> >>> >> >> >> ------------------------------------------------------------------------------ >> Virtualization & Cloud Management Using Capacity Planning >> Cloud computing makes use of virtualization - but cloud computing >> also focuses on allowing computing to be delivered as a service. >> http://www.accelacomm.com/jaw/sfnl/114/51521223/ >> _______________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> >> > > > -- > - Davies > |
|
From: Davies L. <dav...@gm...> - 2012-03-07 03:13:27
|
Hi, I had try to recover it from metadata.mfs and metadata_ml.mfs.back, but failed. Because disk is full, mfsmaster had not dump all the metadata into disks, had part of nodes in metadata.mfs, no names, no edges, no chunks. No hope to recover from the broken metadata.mfs, it's too short. The changelogs are also helpless, only in two days. The only hope is try to undelete the previous metadata_ml.mfs.back from metalogger machine, Chenggang had also failed, with extundelete and etx3grep, maybe some experts can archive this. The final options is to GUESS the relation between files and chunks by chunk id and size fo files, if data lost can not been afforded. Each chunk is combined with crc checksum and real data, if we know the relation between files and chunksever, then we can get data back. Or we can contruct the metadata according to the GUESS, the using mfsmaster to recover them. Davies On Wed, Mar 7, 2012 at 10:56 AM, Ken <ken...@gm...> wrote: > Hi, chengang > > I think you should try more, and post detail here. Someone must resolve it. > Maybe you will lost some data in last few minutes, but 250T should be > saved. > > At first, BACKUP all files: > /var/lib/mfs/* on master > /var/lib/mfs/* on mfsmetalogger > > about restore error: > > file 'metadata.mfs.back' not found - will try 'metadata_ml.mfs.back' >> instead >> loading objects (files,directories,etc.) ... loading node: read error: >> ENOENT (No such file or directory) >> error >> can't read metadata from file: .//metadata_ml.mfs.back >> > How did you run mfsmetarestore? add -d options? > If stat(datapath + metadata_ml.mfs.back) fail, these error will occur. > Maybe use strace will show why stat fail exactly. > > ps: I am in Beijing now and I can provide more help. > > HTH > > -Ken > > > > On Wed, Mar 7, 2012 at 10:21 AM, 陈钢 <yik...@gm...> wrote: > >> can not start mfsmaster with the file "78962688 Mar 6 17:18 metadata.mfs >> ".. >> i tried that. :( >> >> 2012/3/7 Ricardo J. Barberis <ric...@da...> >> >> El Martes 06/03/2012, 陈钢 escribió: >>> > on master >>> [ ... ] >>> > -rw-r----- 1 mfs mfs 78962688 Mar 6 17:18 metadata.mfs >>> > -rw-r--r-- 1 root root 8 Jul 4 2011 metadata.mfs.empty >>> > -rw-r----- 1 mfs mfs 5984 Mar 6 12:00 sessions.mfs >>> > -rw-r----- 1 mfs mfs 0 Mar 6 16:46 sessions.mfs.tmp >>> > -rw-r----- 1 mfs mfs 131072 Mar 6 17:18 stats.mfs >>> >>> You have /var/lib/mfs/metadata.mfs on the master, it might not be corrupt >>> after all? >>> >>> I'd suggest: >>> >>> - backup /var/lib/mfs to another disk/server (for later recovery if >>> needed) >>> - make sure you have free space in your main disk >>> - then simply try to start mfsmaster >>> - check mfs.cgi (web interface) if it looks OK >>> >>> >>> BUT: if you can, wait for confirmation from Michał Borychowski first, in >>> case >>> what I'm telling you is not safe. >>> >>> >>> (BTW: You have Reply-To set to che...@cp..., I don't know if that's >>> intentional on your part) >>> >>> Hope it helps, >>> -- >>> Ricardo J. Barberis >>> Senior SysAdmin / ITI >>> Dattatec.com :: Soluciones de Web Hosting >>> Tu Hosting hecho Simple! >>> >>> ------------------------------------------ >>> >> >> >> >> ------------------------------------------------------------------------------ >> Virtualization & Cloud Management Using Capacity Planning >> Cloud computing makes use of virtualization - but cloud computing >> also focuses on allowing computing to be delivered as a service. >> http://www.accelacomm.com/jaw/sfnl/114/51521223/ >> >> _______________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> >> > > > ------------------------------------------------------------------------------ > Virtualization & Cloud Management Using Capacity Planning > Cloud computing makes use of virtualization - but cloud computing > also focuses on allowing computing to be delivered as a service. > http://www.accelacomm.com/jaw/sfnl/114/51521223/ > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > > -- - Davies |
|
From: 陈钢 <yik...@gm...> - 2012-03-07 03:08:49
|
I backup all files already. yes, i use metadatarestore with -d option,not work. how can i contact you ? gtalk? 2012/3/7 Ken <ken...@gm...> > Hi, chengang > > I think you should try more, and post detail here. Someone must resolve it. > Maybe you will lost some data in last few minutes, but 250T should be > saved. > > At first, BACKUP all files: > /var/lib/mfs/* on master > /var/lib/mfs/* on mfsmetalogger > > about restore error: > > file 'metadata.mfs.back' not found - will try 'metadata_ml.mfs.back' >> instead >> loading objects (files,directories,etc.) ... loading node: read error: >> ENOENT (No such file or directory) >> error >> can't read metadata from file: .//metadata_ml.mfs.back >> > How did you run mfsmetarestore? add -d options? > If stat(datapath + metadata_ml.mfs.back) fail, these error will occur. > Maybe use strace will show why stat fail exactly. > > ps: I am in Beijing now and I can provide more help. > > HTH > > -Ken > > > > On Wed, Mar 7, 2012 at 10:21 AM, 陈钢 <yik...@gm...> wrote: > >> can not start mfsmaster with the file "78962688 Mar 6 17:18 metadata.mfs >> ".. >> i tried that. :( >> >> 2012/3/7 Ricardo J. Barberis <ric...@da...> >> >> El Martes 06/03/2012, 陈钢 escribió: >>> > on master >>> [ ... ] >>> > -rw-r----- 1 mfs mfs 78962688 Mar 6 17:18 metadata.mfs >>> > -rw-r--r-- 1 root root 8 Jul 4 2011 metadata.mfs.empty >>> > -rw-r----- 1 mfs mfs 5984 Mar 6 12:00 sessions.mfs >>> > -rw-r----- 1 mfs mfs 0 Mar 6 16:46 sessions.mfs.tmp >>> > -rw-r----- 1 mfs mfs 131072 Mar 6 17:18 stats.mfs >>> >>> You have /var/lib/mfs/metadata.mfs on the master, it might not be corrupt >>> after all? >>> >>> I'd suggest: >>> >>> - backup /var/lib/mfs to another disk/server (for later recovery if >>> needed) >>> - make sure you have free space in your main disk >>> - then simply try to start mfsmaster >>> - check mfs.cgi (web interface) if it looks OK >>> >>> >>> BUT: if you can, wait for confirmation from Michał Borychowski first, in >>> case >>> what I'm telling you is not safe. >>> >>> >>> (BTW: You have Reply-To set to che...@cp..., I don't know if that's >>> intentional on your part) >>> >>> Hope it helps, >>> -- >>> Ricardo J. Barberis >>> Senior SysAdmin / ITI >>> Dattatec.com :: Soluciones de Web Hosting >>> Tu Hosting hecho Simple! >>> >>> ------------------------------------------ >>> >> >> >> >> ------------------------------------------------------------------------------ >> Virtualization & Cloud Management Using Capacity Planning >> Cloud computing makes use of virtualization - but cloud computing >> also focuses on allowing computing to be delivered as a service. >> http://www.accelacomm.com/jaw/sfnl/114/51521223/ >> >> _______________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> >> > |
|
From: Ken <ken...@gm...> - 2012-03-07 02:57:13
|
Hi, chengang I think you should try more, and post detail here. Someone must resolve it. Maybe you will lost some data in last few minutes, but 250T should be saved. At first, BACKUP all files: /var/lib/mfs/* on master /var/lib/mfs/* on mfsmetalogger about restore error: > file 'metadata.mfs.back' not found - will try 'metadata_ml.mfs.back' > instead > loading objects (files,directories,etc.) ... loading node: read error: > ENOENT (No such file or directory) > error > can't read metadata from file: .//metadata_ml.mfs.back > How did you run mfsmetarestore? add -d options? If stat(datapath + metadata_ml.mfs.back) fail, these error will occur. Maybe use strace will show why stat fail exactly. ps: I am in Beijing now and I can provide more help. HTH -Ken On Wed, Mar 7, 2012 at 10:21 AM, 陈钢 <yik...@gm...> wrote: > can not start mfsmaster with the file "78962688 Mar 6 17:18 metadata.mfs > ".. > i tried that. :( > > 2012/3/7 Ricardo J. Barberis <ric...@da...> > > El Martes 06/03/2012, 陈钢 escribió: >> > on master >> [ ... ] >> > -rw-r----- 1 mfs mfs 78962688 Mar 6 17:18 metadata.mfs >> > -rw-r--r-- 1 root root 8 Jul 4 2011 metadata.mfs.empty >> > -rw-r----- 1 mfs mfs 5984 Mar 6 12:00 sessions.mfs >> > -rw-r----- 1 mfs mfs 0 Mar 6 16:46 sessions.mfs.tmp >> > -rw-r----- 1 mfs mfs 131072 Mar 6 17:18 stats.mfs >> >> You have /var/lib/mfs/metadata.mfs on the master, it might not be corrupt >> after all? >> >> I'd suggest: >> >> - backup /var/lib/mfs to another disk/server (for later recovery if >> needed) >> - make sure you have free space in your main disk >> - then simply try to start mfsmaster >> - check mfs.cgi (web interface) if it looks OK >> >> >> BUT: if you can, wait for confirmation from Michał Borychowski first, in >> case >> what I'm telling you is not safe. >> >> >> (BTW: You have Reply-To set to che...@cp..., I don't know if that's >> intentional on your part) >> >> Hope it helps, >> -- >> Ricardo J. Barberis >> Senior SysAdmin / ITI >> Dattatec.com :: Soluciones de Web Hosting >> Tu Hosting hecho Simple! >> >> ------------------------------------------ >> > > > > ------------------------------------------------------------------------------ > Virtualization & Cloud Management Using Capacity Planning > Cloud computing makes use of virtualization - but cloud computing > also focuses on allowing computing to be delivered as a service. > http://www.accelacomm.com/jaw/sfnl/114/51521223/ > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > > |
|
From: 陈钢 <yik...@gm...> - 2012-03-07 02:21:40
|
can not start mfsmaster with the file "78962688 Mar 6 17:18 metadata.mfs ".. i tried that. :( 2012/3/7 Ricardo J. Barberis <ric...@da...> > El Martes 06/03/2012, 陈钢 escribió: > > on master > [ ... ] > > -rw-r----- 1 mfs mfs 78962688 Mar 6 17:18 metadata.mfs > > -rw-r--r-- 1 root root 8 Jul 4 2011 metadata.mfs.empty > > -rw-r----- 1 mfs mfs 5984 Mar 6 12:00 sessions.mfs > > -rw-r----- 1 mfs mfs 0 Mar 6 16:46 sessions.mfs.tmp > > -rw-r----- 1 mfs mfs 131072 Mar 6 17:18 stats.mfs > > You have /var/lib/mfs/metadata.mfs on the master, it might not be corrupt > after all? > > I'd suggest: > > - backup /var/lib/mfs to another disk/server (for later recovery if needed) > - make sure you have free space in your main disk > - then simply try to start mfsmaster > - check mfs.cgi (web interface) if it looks OK > > > BUT: if you can, wait for confirmation from Michał Borychowski first, in > case > what I'm telling you is not safe. > > > (BTW: You have Reply-To set to che...@cp..., I don't know if that's > intentional on your part) > > Hope it helps, > -- > Ricardo J. Barberis > Senior SysAdmin / ITI > Dattatec.com :: Soluciones de Web Hosting > Tu Hosting hecho Simple! > > ------------------------------------------ > |
|
From: Ricardo J. B. <ric...@da...> - 2012-03-06 22:19:52
|
El Martes 06/03/2012, 陈钢 escribió: > on master [ ... ] > -rw-r----- 1 mfs mfs 78962688 Mar 6 17:18 metadata.mfs > -rw-r--r-- 1 root root 8 Jul 4 2011 metadata.mfs.empty > -rw-r----- 1 mfs mfs 5984 Mar 6 12:00 sessions.mfs > -rw-r----- 1 mfs mfs 0 Mar 6 16:46 sessions.mfs.tmp > -rw-r----- 1 mfs mfs 131072 Mar 6 17:18 stats.mfs You have /var/lib/mfs/metadata.mfs on the master, it might not be corrupt after all? I'd suggest: - backup /var/lib/mfs to another disk/server (for later recovery if needed) - make sure you have free space in your main disk - then simply try to start mfsmaster - check mfs.cgi (web interface) if it looks OK BUT: if you can, wait for confirmation from Michał Borychowski first, in case what I'm telling you is not safe. (BTW: You have Reply-To set to che...@cp..., I don't know if that's intentional on your part) Hope it helps, -- Ricardo J. Barberis Senior SysAdmin / ITI Dattatec.com :: Soluciones de Web Hosting Tu Hosting hecho Simple! ------------------------------------------ |
|
From: 陈钢 <yik...@gm...> - 2012-03-06 14:21:40
|
on master -rw-r----- 1 mfs mfs 43376640 Mar 6 07:59 changelog.10.mfs -rw-r----- 1 mfs mfs 42741760 Mar 6 07:00 changelog.11.mfs -rw-r----- 1 mfs mfs 48666829 Mar 6 05:59 changelog.12.mfs -rw-r----- 1 mfs mfs 46485504 Mar 6 05:00 changelog.13.mfs -rw-r----- 1 mfs mfs 45221600 Mar 6 04:00 changelog.14.mfs -rw-r----- 1 mfs mfs 34729984 Mar 6 03:00 changelog.15.mfs -rw-r----- 1 mfs mfs 50851840 Mar 6 02:00 changelog.16.mfs -rw-r----- 1 mfs mfs 60547072 Mar 6 01:00 changelog.17.mfs -rw-r----- 1 mfs mfs 33427456 Mar 5 23:59 changelog.18.mfs -rw-r----- 1 mfs mfs 46690304 Mar 5 23:00 changelog.19.mfs -rw-r----- 1 mfs mfs 54243328 Mar 6 17:18 changelog.1.mfs -rw-r----- 1 mfs mfs 53063680 Mar 5 21:59 changelog.20.mfs -rw-r----- 1 mfs mfs 52146176 Mar 5 21:00 changelog.21.mfs -rw-r----- 1 mfs mfs 34467840 Mar 5 20:00 changelog.22.mfs -rw-r----- 1 mfs mfs 32673792 Mar 5 19:00 changelog.23.mfs -rw-r----- 1 mfs mfs 38252544 Mar 5 18:00 changelog.24.mfs -rw-r----- 1 mfs mfs 39833600 Mar 5 16:00 changelog.25.mfs -rw-r----- 1 mfs mfs 54104064 Mar 5 15:00 changelog.26.mfs -rw-r----- 1 mfs mfs 76775424 Mar 5 13:59 changelog.27.mfs -rw-r----- 1 mfs mfs 44556288 Mar 5 13:00 changelog.28.mfs -rw-r----- 1 mfs mfs 50315264 Mar 5 11:59 changelog.29.mfs -rw-r----- 1 mfs mfs 45498368 Mar 6 16:00 changelog.2.mfs -rw-r----- 1 mfs mfs 65257472 Mar 5 10:59 changelog.30.mfs -rw-r----- 1 mfs mfs 78256495 Mar 5 09:59 changelog.31.mfs -rw-r----- 1 mfs mfs 92380473 Mar 5 09:00 changelog.32.mfs -rw-r----- 1 mfs mfs 94376943 Mar 5 07:59 changelog.33.mfs -rw-r----- 1 mfs mfs 92183317 Mar 5 07:00 changelog.34.mfs -rw-r----- 1 mfs mfs 56623104 Mar 5 06:00 changelog.35.mfs -rw-r----- 1 mfs mfs 58347520 Mar 5 05:00 changelog.36.mfs -rw-r----- 1 mfs mfs 55304192 Mar 5 04:00 changelog.37.mfs -rw-r----- 1 mfs mfs 52674560 Mar 5 03:00 changelog.38.mfs -rw-r----- 1 mfs mfs 34627584 Mar 5 01:59 changelog.39.mfs -rw-r----- 1 mfs mfs 53587968 Mar 6 15:00 changelog.3.mfs -rw-r----- 1 mfs mfs 39833600 Mar 5 00:59 changelog.40.mfs -rw-r----- 1 mfs mfs 55279616 Mar 4 23:59 changelog.41.mfs -rw-r----- 1 mfs mfs 61194240 Mar 4 23:00 changelog.42.mfs -rw-r----- 1 mfs mfs 63688704 Mar 4 21:59 changelog.43.mfs -rw-r----- 1 mfs mfs 68395008 Mar 4 20:59 changelog.44.mfs -rw-r----- 1 mfs mfs 71020544 Mar 4 19:59 changelog.45.mfs -rw-r----- 1 mfs mfs 83447808 Mar 4 19:00 changelog.46.mfs -rw-r----- 1 mfs mfs 118579200 Mar 4 18:00 changelog.47.mfs -rw-r----- 1 mfs mfs 102060795 Mar 4 15:59 changelog.48.mfs -rw-r----- 1 mfs mfs 92660665 Mar 4 15:00 changelog.49.mfs -rw-r----- 1 mfs mfs 60923904 Mar 6 14:00 changelog.4.mfs -rw-r----- 1 mfs mfs 86800616 Mar 4 13:59 changelog.50.mfs -rw-r----- 1 mfs mfs 65785856 Mar 6 13:00 changelog.5.mfs -rw-r----- 1 mfs mfs 20377435 Mar 6 12:00 changelog.6.mfs -rw-r----- 1 mfs mfs 49433076 Mar 6 11:00 changelog.7.mfs -rw-r----- 1 mfs mfs 32522240 Mar 6 09:59 changelog.8.mfs -rw-r----- 1 mfs mfs 27201536 Mar 6 08:59 changelog.9.mfs -rw-r----- 1 mfs mfs 78962688 Mar 6 17:18 metadata.mfs -rw-r--r-- 1 root root 8 Jul 4 2011 metadata.mfs.empty -rw-r----- 1 mfs mfs 5984 Mar 6 12:00 sessions.mfs -rw-r----- 1 mfs mfs 0 Mar 6 16:46 sessions.mfs.tmp -rw-r----- 1 mfs mfs 131072 Mar 6 17:18 stats.mfs on metalogger -rw-r----- 1 mfs mfs 215719718 Mar 6 17:18 changelog_ml.0.mfs -rw-r----- 1 mfs mfs 153003570 Mar 6 16:00 changelog_ml.1.mfs -rw-r----- 1 mfs mfs 125645785 Mar 6 15:00 changelog_ml.2.mfs -rw-r----- 1 mfs mfs 101594634 Mar 6 14:00 changelog_ml.3.mfs -rw-r----- 1 mfs mfs 94872041 Mar 6 13:00 changelog_ml.4.mfs -rw-r----- 1 mfs mfs 20377747 Mar 6 12:00 changelog_ml.5.mfs -rw-r----- 1 mfs mfs 54243328 Mar 6 17:10 changelog_ml_back.0.mfs -rw-r----- 1 mfs mfs 45498368 Mar 6 17:10 changelog_ml_back.1.mfs -rw-r----- 1 mfs mfs 54468608 Mar 6 17:10 metadata_ml.mfs.back -rw-r----- 1 mfs mfs 8 Mar 6 17:37 sessions.mfs -rw-r----- 1 mfs mfs 5984 Mar 6 17:18 sessions_ml.mfs 2012/3/6 Davies Liu <dav...@gm...> > It seems that the metadata in master and metalogger are broken, > could you show us the content in /var/lib/mfs/ on master and metalogger? > > ls /var/lib/mfs/ ? > > please backup all of them first. > > > 2012/3/6 陈钢 <yik...@gm...> > >> Is there any more way to go? >> >> >> 2012/3/6 Michał Borychowski <mic...@ge...> >> >>> Hi!**** >>> >>> ** ** >>> >>> Please tell what MooseFS version do you use? What OS is on >>> master/chunks? Did you have metalogger running?**** >>> >>> ** ** >>> >>> Try to find metadata in these locations:**** >>> >>> /metadata.mfs.emergency**** >>> >>> /tmp/metadata.mfs.emergency**** >>> >>> /var/metadata.mfs.emergency**** >>> >>> /usr/metadata.mfs.emergency**** >>> >>> /usr/share/metadata.mfs.emergency**** >>> >>> /usr/local/metadata.mfs.emergency**** >>> >>> /usr/local/var/metadata.mfs.emergency**** >>> >>> /usr/local/share/metadata.mfs.emergency**** >>> >>> **** >>> >>> ** ** >>> >>> Kind regards**** >>> >>> Michał Borychowski **** >>> >>> MooseFS Support Manager**** >>> >>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _**** >>> >>> Gemius S.A.**** >>> >>> ul. Wołoska 7, 02-672 Warszawa**** >>> >>> Budynek MARS, klatka D**** >>> >>> Tel.: +4822 874-41-00**** >>> >>> Fax : +4822 874-41-01**** >>> >>> ** ** >>> >>> *From:* 陈钢 [mailto:yik...@gm...] >>> *Sent:* Tuesday, March 06, 2012 11:36 AM >>> *To:* moo...@li... >>> *Subject:* [Moosefs-users] HELP!!! 250T Data lost!**** >>> >>> ** ** >>> >>> i <moo...@li...>`m crazy, my boss will kill me .. >>> **** >>> >>> ** ** >>> >>> on mfsmaster machine, i execute "mfsmaster restart", then it crashed.*** >>> * >>> >>> ** ** >>> >>> err info is here：**** >>> >>> ======**** >>> >>> working directory: /var/lib/mfs**** >>> >>> lockfile created and locked**** >>> >>> initializing mfsmaster modules ...**** >>> >>> loading sessions ... ok**** >>> >>> sessions file has been loaded**** >>> >>> exports file has been loaded**** >>> >>> loading metadata ...**** >>> >>> loading objects (files,directories,etc.) ... loading node: read error: >>> ENOENT (No such file or directory)**** >>> >>> error**** >>> >>> init: file system manager failed !!!**** >>> >>> error occured during initialization - exiting**** >>> >>> =======**** >>> >>> ** ** >>> >>> no "metadata.mfs_back" left for me . i think it because there is no >>> space left on mfsmaster hard disk.**** >>> >>> ** ** >>> >>> ** ** >>> >>> ** ** >>> >>> then i log in mfsmetalogger server. i execute "mfsmetarestore -a -d >>> /var/lib/mfs", and i also report err! **** >>> >>> err info is here:**** >>> >>> ======**** >>> >>> file 'metadata.mfs.back' not found - will try 'metadata_ml.mfs.back' >>> instead**** >>> >>> loading objects (files,directories,etc.) ... loading node: read error: >>> ENOENT (No such file or directory)**** >>> >>> error**** >>> >>> can't read metadata from file: .//metadata_ml.mfs.back**** >>> >>> =======**** >>> >>> ** ** >>> >>> ** ** >>> >>> what can i do now?**** >>> >>> ** ** >>> >>> ** ** >>> >>> PLEASE SAVE MY LIFE.....**** >>> >> >> >> >> ------------------------------------------------------------------------------ >> >> Keep Your Developer Skills Current with LearnDevNow! >> The most comprehensive online learning library for Microsoft developers >> is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, >> Metro Style Apps, more. Free future releases when you subscribe now! >> http://p.sf.net/sfu/learndevnow-d2d >> _______________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> >> > > > -- > - Davies > |
|
From: Davies L. <dav...@gm...> - 2012-03-06 14:15:26
|
It seems that the metadata in master and metalogger are broken, could you show us the content in /var/lib/mfs/ on master and metalogger? ls /var/lib/mfs/ ? please backup all of them first. 2012/3/6 陈钢 <yik...@gm...> > Is there any more way to go? > > > 2012/3/6 Michał Borychowski <mic...@ge...> > >> Hi!**** >> >> ** ** >> >> Please tell what MooseFS version do you use? What OS is on master/chunks? >> Did you have metalogger running?**** >> >> ** ** >> >> Try to find metadata in these locations:**** >> >> /metadata.mfs.emergency**** >> >> /tmp/metadata.mfs.emergency**** >> >> /var/metadata.mfs.emergency**** >> >> /usr/metadata.mfs.emergency**** >> >> /usr/share/metadata.mfs.emergency**** >> >> /usr/local/metadata.mfs.emergency**** >> >> /usr/local/var/metadata.mfs.emergency**** >> >> /usr/local/share/metadata.mfs.emergency**** >> >> **** >> >> ** ** >> >> Kind regards**** >> >> Michał Borychowski **** >> >> MooseFS Support Manager**** >> >> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _**** >> >> Gemius S.A.**** >> >> ul. Wołoska 7, 02-672 Warszawa**** >> >> Budynek MARS, klatka D**** >> >> Tel.: +4822 874-41-00**** >> >> Fax : +4822 874-41-01**** >> >> ** ** >> >> *From:* 陈钢 [mailto:yik...@gm...] >> *Sent:* Tuesday, March 06, 2012 11:36 AM >> *To:* moo...@li... >> *Subject:* [Moosefs-users] HELP!!! 250T Data lost!**** >> >> ** ** >> >> i <moo...@li...>`m crazy, my boss will kill me ..* >> *** >> >> ** ** >> >> on mfsmaster machine, i execute "mfsmaster restart", then it crashed.**** >> >> ** ** >> >> err info is here：**** >> >> ======**** >> >> working directory: /var/lib/mfs**** >> >> lockfile created and locked**** >> >> initializing mfsmaster modules ...**** >> >> loading sessions ... ok**** >> >> sessions file has been loaded**** >> >> exports file has been loaded**** >> >> loading metadata ...**** >> >> loading objects (files,directories,etc.) ... loading node: read error: >> ENOENT (No such file or directory)**** >> >> error**** >> >> init: file system manager failed !!!**** >> >> error occured during initialization - exiting**** >> >> =======**** >> >> ** ** >> >> no "metadata.mfs_back" left for me . i think it because there is no >> space left on mfsmaster hard disk.**** >> >> ** ** >> >> ** ** >> >> ** ** >> >> then i log in mfsmetalogger server. i execute "mfsmetarestore -a -d >> /var/lib/mfs", and i also report err! **** >> >> err info is here:**** >> >> ======**** >> >> file 'metadata.mfs.back' not found - will try 'metadata_ml.mfs.back' >> instead**** >> >> loading objects (files,directories,etc.) ... loading node: read error: >> ENOENT (No such file or directory)**** >> >> error**** >> >> can't read metadata from file: .//metadata_ml.mfs.back**** >> >> =======**** >> >> ** ** >> >> ** ** >> >> what can i do now?**** >> >> ** ** >> >> ** ** >> >> PLEASE SAVE MY LIFE.....**** >> > > > > ------------------------------------------------------------------------------ > Keep Your Developer Skills Current with LearnDevNow! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-d2d > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > > -- - Davies |
|
From: Davies L. <dav...@gm...> - 2012-03-06 13:52:44
|
2012/3/6 Michał Borychowski <mic...@ge...> > Hi! > > I'm curious whether you made some tests with your solution? What was the > performance gain? Was it like 20-30% or rather 2-3%? > I had not do some benchmark for it, but we had noticed that the patch reduce the "time of data write operation" of chunk server from max 8 seconds (0.8 seconds, avg) to max 100 ms (50 ms avg). We write weblog to MFS by scribed in nearly realtime, had notice that huge amount of "writeworker: connection with (C0A8014B:9422) was timed out" in syslog of mfsmount before patching it. After the patch, the number of time out messages from 120k to about 100. Reduce the times of fsync() is necessary to improve write performance, especially for small block write operations. When delaying fsync() until fclose(), data loss will only occur when power failing. So we can touch a file when chunk server startup, then remove it when shutdown. When power failing, the flag file will not been removed, so it seems that some of chunks may not beed fully synced, it can scan all chunks with mtime after mtime of the flag file. > > Actually here we are quite skeptical about this. In one of your emails you > suggest to do fsync more rarely (eg. every 30 seconds). ut when CS is > closed cleanly, OS will do all fsyncs before closing the files but when CS > is not closed cleanly so how should we know which files to test? CS upon > startup doesn't do 'stat' on every file (it took too long time). So it > won't know which files to check. > > We could create some extra file (eg. named '.dirty') where we could save > id of the file upon opening it (and do fsync on the '.dirty' file). Upon > file closing we delete its id from '.dirty' file. When CS closes cleanly, > '.dirty' file should be empty. If not, upon starting of CS, it reads the > '.dirty' file and it scans all the chunks which are saved in this file. > > > You also gave some suggestions to use this options: > 1. FLUSH_ON_WRITE - option easy to implement, but not that secure > 2. FLUSH_DELAY - as above > 3. CHECKSUM_INITIAL - this would mean to read all chunks on all the disks > upon startup which is just impossible (in some environments would take more > than 24hrs). > > > And still we are afraid that there may be scenario that malfunction of CS > without fsyncs could cause that chunk will "return" to a proper form (in > sense of CRC) from before the save. It will mean that there would be > several "proper" copies of the same chunk, but with different content - we > cannot allow this to happen. > > Possible it would be necessary to inform master server that CS has some > 'unfsynced' chunks. So this gets still more complicated. That's why we are > curious whether the performance gain is substantial enough to start doing > this fine tuning. > > > Kind regards > Michał Borychowski > MooseFS Support Manager > > > -- - Davies |
|
From: 陈钢 <yik...@gm...> - 2012-03-06 11:52:56
|
Is there any more way to go? 2012/3/6 Michał Borychowski <mic...@ge...> > Hi!**** > > ** ** > > Please tell what MooseFS version do you use? What OS is on master/chunks? > Did you have metalogger running?**** > > ** ** > > Try to find metadata in these locations:**** > > /metadata.mfs.emergency**** > > /tmp/metadata.mfs.emergency**** > > /var/metadata.mfs.emergency**** > > /usr/metadata.mfs.emergency**** > > /usr/share/metadata.mfs.emergency**** > > /usr/local/metadata.mfs.emergency**** > > /usr/local/var/metadata.mfs.emergency**** > > /usr/local/share/metadata.mfs.emergency**** > > **** > > ** ** > > Kind regards**** > > Michał Borychowski **** > > MooseFS Support Manager**** > > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _**** > > Gemius S.A.**** > > ul. Wołoska 7, 02-672 Warszawa**** > > Budynek MARS, klatka D**** > > Tel.: +4822 874-41-00**** > > Fax : +4822 874-41-01**** > > ** ** > > *From:* 陈钢 [mailto:yik...@gm...] > *Sent:* Tuesday, March 06, 2012 11:36 AM > *To:* moo...@li... > *Subject:* [Moosefs-users] HELP!!! 250T Data lost!**** > > ** ** > > i <moo...@li...>`m crazy, my boss will kill me ..** > ** > > ** ** > > on mfsmaster machine, i execute "mfsmaster restart", then it crashed.**** > > ** ** > > err info is here：**** > > ======**** > > working directory: /var/lib/mfs**** > > lockfile created and locked**** > > initializing mfsmaster modules ...**** > > loading sessions ... ok**** > > sessions file has been loaded**** > > exports file has been loaded**** > > loading metadata ...**** > > loading objects (files,directories,etc.) ... loading node: read error: > ENOENT (No such file or directory)**** > > error**** > > init: file system manager failed !!!**** > > error occured during initialization - exiting**** > > =======**** > > ** ** > > no "metadata.mfs_back" left for me . i think it because there is no space > left on mfsmaster hard disk.**** > > ** ** > > ** ** > > ** ** > > then i log in mfsmetalogger server. i execute "mfsmetarestore -a -d > /var/lib/mfs", and i also report err! **** > > err info is here:**** > > ======**** > > file 'metadata.mfs.back' not found - will try 'metadata_ml.mfs.back' > instead**** > > loading objects (files,directories,etc.) ... loading node: read error: > ENOENT (No such file or directory)**** > > error**** > > can't read metadata from file: .//metadata_ml.mfs.back**** > > =======**** > > ** ** > > ** ** > > what can i do now?**** > > ** ** > > ** ** > > PLEASE SAVE MY LIFE.....**** > |
|
From: 陈钢 <yik...@gm...> - 2012-03-06 11:36:39
|
no,there is no metadata in those locations... what should i do ? my god ! 2012/3/6 Michał Borychowski <mic...@ge...> > Hi!**** > > ** ** > > Please tell what MooseFS version do you use? What OS is on master/chunks? > Did you have metalogger running?**** > > ** ** > > Try to find metadata in these locations:**** > > /metadata.mfs.emergency**** > > /tmp/metadata.mfs.emergency**** > > /var/metadata.mfs.emergency**** > > /usr/metadata.mfs.emergency**** > > /usr/share/metadata.mfs.emergency**** > > /usr/local/metadata.mfs.emergency**** > > /usr/local/var/metadata.mfs.emergency**** > > /usr/local/share/metadata.mfs.emergency**** > > **** > > ** ** > > Kind regards**** > > Michał Borychowski **** > > MooseFS Support Manager**** > > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _**** > > Gemius S.A.**** > > ul. Wołoska 7, 02-672 Warszawa**** > > Budynek MARS, klatka D**** > > Tel.: +4822 874-41-00**** > > Fax : +4822 874-41-01**** > > ** ** > > *From:* 陈钢 [mailto:yik...@gm...] > *Sent:* Tuesday, March 06, 2012 11:36 AM > *To:* moo...@li... > *Subject:* [Moosefs-users] HELP!!! 250T Data lost!**** > > ** ** > > i <moo...@li...>`m crazy, my boss will kill me ..** > ** > > ** ** > > on mfsmaster machine, i execute "mfsmaster restart", then it crashed.**** > > ** ** > > err info is here：**** > > ======**** > > working directory: /var/lib/mfs**** > > lockfile created and locked**** > > initializing mfsmaster modules ...**** > > loading sessions ... ok**** > > sessions file has been loaded**** > > exports file has been loaded**** > > loading metadata ...**** > > loading objects (files,directories,etc.) ... loading node: read error: > ENOENT (No such file or directory)**** > > error**** > > init: file system manager failed !!!**** > > error occured during initialization - exiting**** > > =======**** > > ** ** > > no "metadata.mfs_back" left for me . i think it because there is no space > left on mfsmaster hard disk.**** > > ** ** > > ** ** > > ** ** > > then i log in mfsmetalogger server. i execute "mfsmetarestore -a -d > /var/lib/mfs", and i also report err! **** > > err info is here:**** > > ======**** > > file 'metadata.mfs.back' not found - will try 'metadata_ml.mfs.back' > instead**** > > loading objects (files,directories,etc.) ... loading node: read error: > ENOENT (No such file or directory)**** > > error**** > > can't read metadata from file: .//metadata_ml.mfs.back**** > > =======**** > > ** ** > > ** ** > > what can i do now?**** > > ** ** > > ** ** > > PLEASE SAVE MY LIFE.....**** > |
|
From: Anh K. H. <anh...@gm...> - 2012-03-06 11:34:42
|
On Tue, 6 Mar 2012 19:24:26 +0800 陈钢 <yik...@gm...> wrote: > i run mfs-1.6.20-2. > master and chunk all run on Red Hat Enterprise Linux Server release 5 > yes i have metalogger running. First try to find the metadata on your master. If you can't locate them, you still have a chance to find metadata on your metalogger server. Regards, -- Anh K. Huynh System administrator |
|
From: 陈钢 <yik...@gm...> - 2012-03-06 11:32:40
|
i have found metadata on my metalogger server. but when i want to restore from it . err this: ====== file 'metadata.mfs.back' not found - will try 'metadata_ml.mfs.back' instead loading objects (files,directories,etc.) ... loading node: read error: ENOENT (No such file or directory) error can't read metadata from file: .//metadata_ml.mfs.back ======= 2012/3/6 Anh K. Huynh <anh...@gm...> > On Tue, 6 Mar 2012 19:24:26 +0800 > 陈钢 <yik...@gm...> wrote: > > > i run mfs-1.6.20-2. > > master and chunk all run on Red Hat Enterprise Linux Server release 5 > > yes i have metalogger running. > > First try to find the metadata on your master. If you can't locate > them, you still have a chance to find metadata on your metalogger > server. > > Regards, > > > -- > Anh K. Huynh > System administrator > |
|
From: 陈钢 <yik...@gm...> - 2012-03-06 11:24:38
|
i run mfs-1.6.20-2. master and chunk all run on Red Hat Enterprise Linux Server release 5 yes i have metalogger running. 2012/3/6 Michał Borychowski <mic...@ge...> > Hi!**** > > ** ** > > Please tell what MooseFS version do you use? What OS is on master/chunks? > Did you have metalogger running?**** > > ** ** > > Try to find metadata in these locations:**** > > /metadata.mfs.emergency**** > > /tmp/metadata.mfs.emergency**** > > /var/metadata.mfs.emergency**** > > /usr/metadata.mfs.emergency**** > > /usr/share/metadata.mfs.emergency**** > > /usr/local/metadata.mfs.emergency**** > > /usr/local/var/metadata.mfs.emergency**** > > /usr/local/share/metadata.mfs.emergency**** > > **** > > ** ** > > Kind regards**** > > Michał Borychowski **** > > MooseFS Support Manager**** > > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _**** > > Gemius S.A.**** > > ul. Wołoska 7, 02-672 Warszawa**** > > Budynek MARS, klatka D**** > > Tel.: +4822 874-41-00**** > > Fax : +4822 874-41-01**** > > ** ** > > *From:* 陈钢 [mailto:yik...@gm...] > *Sent:* Tuesday, March 06, 2012 11:36 AM > *To:* moo...@li... > *Subject:* [Moosefs-users] HELP!!! 250T Data lost!**** > > ** ** > > i <moo...@li...>`m crazy, my boss will kill me ..** > ** > > ** ** > > on mfsmaster machine, i execute "mfsmaster restart", then it crashed.**** > > ** ** > > err info is here：**** > > ======**** > > working directory: /var/lib/mfs**** > > lockfile created and locked**** > > initializing mfsmaster modules ...**** > > loading sessions ... ok**** > > sessions file has been loaded**** > > exports file has been loaded**** > > loading metadata ...**** > > loading objects (files,directories,etc.) ... loading node: read error: > ENOENT (No such file or directory)**** > > error**** > > init: file system manager failed !!!**** > > error occured during initialization - exiting**** > > =======**** > > ** ** > > no "metadata.mfs_back" left for me . i think it because there is no space > left on mfsmaster hard disk.**** > > ** ** > > ** ** > > ** ** > > then i log in mfsmetalogger server. i execute "mfsmetarestore -a -d > /var/lib/mfs", and i also report err! **** > > err info is here:**** > > ======**** > > file 'metadata.mfs.back' not found - will try 'metadata_ml.mfs.back' > instead**** > > loading objects (files,directories,etc.) ... loading node: read error: > ENOENT (No such file or directory)**** > > error**** > > can't read metadata from file: .//metadata_ml.mfs.back**** > > =======**** > > ** ** > > ** ** > > what can i do now?**** > > ** ** > > ** ** > > PLEASE SAVE MY LIFE.....**** > |
|
From: Michał B. <mic...@ge...> - 2012-03-06 11:21:42
|
Hi! Please tell what MooseFS version do you use? What OS is on master/chunks? Did you have metalogger running? Try to find metadata in these locations: /metadata.mfs.emergency /tmp/metadata.mfs.emergency /var/metadata.mfs.emergency /usr/metadata.mfs.emergency /usr/share/metadata.mfs.emergency /usr/local/metadata.mfs.emergency /usr/local/var/metadata.mfs.emergency /usr/local/share/metadata.mfs.emergency Kind regards Michał Borychowski MooseFS Support Manager _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Gemius S.A. ul. Wołoska 7, 02-672 Warszawa Budynek MARS, klatka D Tel.: +4822 874-41-00 Fax : +4822 874-41-01 From: 陈钢 [mailto:yik...@gm...] Sent: Tuesday, March 06, 2012 11:36 AM To: moo...@li... Subject: [Moosefs-users] HELP!!! 250T Data lost! <mailto:moo...@li...> i`m crazy, my boss will kill me .. on mfsmaster machine, i execute "mfsmaster restart", then it crashed. err info is here： ====== working directory: /var/lib/mfs lockfile created and locked initializing mfsmaster modules ... loading sessions ... ok sessions file has been loaded exports file has been loaded loading metadata ... loading objects (files,directories,etc.) ... loading node: read error: ENOENT (No such file or directory) error init: file system manager failed !!! error occured during initialization - exiting ======= no "metadata.mfs_back" left for me . i think it because there is no space left on mfsmaster hard disk. then i log in mfsmetalogger server. i execute "mfsmetarestore -a -d /var/lib/mfs", and i also report err! err info is here: ====== file 'metadata.mfs.back' not found - will try 'metadata_ml.mfs.back' instead loading objects (files,directories,etc.) ... loading node: read error: ENOENT (No such file or directory) error can't read metadata from file: .//metadata_ml.mfs.back ======= what can i do now? PLEASE SAVE MY LIFE..... |
|
From: Anh K. H. <anh...@gm...> - 2012-03-06 11:20:18
|
On Tue, 6 Mar 2012 18:36:16 +0800 陈钢 <yik...@gm...> wrote: > i <moo...@li...>`m crazy, my boss will kill > me .. > > on mfsmaster machine, i execute "mfsmaster restart", then it crashed. > .... > > what can i do now? Do you have any backup of your metadata files? For example, you can find the metadata files on your metalogger servers (if any) Regards, -- Anh K. Huynh System administrator |
|
From: Chris P. <ch...@ec...> - 2012-03-06 11:17:05
|
Hi Michal In some tests I was running, with 50 disks spread over 18 machines. I was using bonnie++ as its rewrite test matches some of our access pattens. The moosefs standard 1.6.20 was reporting a rewrite speed of about 1.4MB/s With a patch to remove fsyncs totally, the speed went up to about 20MB/s, which is a substantial speed increase. However, forcing a power failure while disks were writing caused chunk loss (because lots of data was in OS cache), and as you say, the blocks data and checksum may not match, which was only picked up later when using the filesystem again. This was why I was thinking about forcing a full checksum of all blocks. Noting blocks which could possibly be dirty is a more elegant way of dealing with this, though. Currently I am testing with the patch from Davies Liu on the mailing list yesterday, which runs the fsync on block close, or delayed. I am seeing about 1/3 the fsyncs of before, leading to percieved higher performance - with no numbers to back that statement up yet :) Chris On Tue, 2012-03-06 at 11:22 +0100, Michał Borychowski wrote: > Hi! > > I'm curious whether you made some tests with your solution? What was the performance gain? Was it like 20-30% or rather 2-3%? > > > Actually here we are quite skeptical about this. In one of your emails you suggest to do fsync more rarely (eg. every 30 seconds). But when CS is closed cleanly, OS will do all fsyncs before closing the files but when CS is not closed cleanly so how should we know which files to test? CS upon startup doesn't do 'stat' on every file (it took too long time). So it won't know which files to check. > > We could create some extra file (eg. named '.dirty') where we could save id of the file upon opening it (and do fsync on the '.dirty' file). Upon file closing we delete its id from '.dirty' file. When CS closes cleanly, '.dirty' file should be empty. If not, upon starting of CS, it reads the '.dirty' file and it scans all the chunks which are saved in this file. > > > You also gave some suggestions to use this options: > 1. FLUSH_ON_WRITE - option easy to implement, but not that secure > 2. FLUSH_DELAY - as above > 3. CHECKSUM_INITIAL - this would mean to read all chunks on all the disks upon startup which is just impossible (in some environments would take more than 24hrs). > > > And still we are afraid that there may be scenario that malfunction of CS without fsyncs could cause that chunk will "return" to a proper form (in sense of CRC) from before the save. It will mean that there would be several "proper" copies of the same chunk, but with different content - we cannot allow this to happen. > > Possible it would be necessary to inform master server that CS has some 'unfsynced' chunks. So this gets still more complicated. That's why we are curious whether the performance gain is substantial enough to start doing this fine tuning. > > > Kind regards > Michał Borychowski > MooseFS Support Manager > > > -- Chris Picton Executive Manager - Systems ECN Telecommunications (Pty) Ltd t: 010 590 0031 m: 079 721 8521 f: 087 941 0813 e: ch...@ec... "Lowering the cost of doing business" |
84 messages has been excluded from this view by a project administrator.
