Menu
▾
▴
Re: [Moosefs-users] HELP!!! 250T Data lost!
|
From: Davies L. <dav...@gm...> - 2012-03-13 01:53:46
|
Hi， congratulation ! Can you show some details about how to recover the data back ? Davies On Tue, Mar 13, 2012 at 4:24 AM, 陈钢 <yik...@gm...> wrote: > Hi all. I got some message to report. > > The situation I faced is that I got my metadata.mfs broken when "mfsmaster > restart" include the metadata.mfs on mfsmetalogger. > > In fact, the metadata on mfsmetalogger is "more broken" than the metadata > on mfsmaster.That is wired. > > So, I got all my files back but half of them lost their filename. > > Lucky,ken...@gm... helped me,and I have a SQLite file which stored > filesize and filename in it.I have already restored my important data 90 > percent now. > > And , I wrote some script helps me cp the metadata.mfs then rsync it to > another server every hour. just like then suggestion in > http://www.moosefs.org/moosefs-faq.html#metadata-backup. > > Metadata is really important , it worse a incremental backup. > > > 2012/3/9 陈钢 <yik...@gm...> > >> Maybe, 80% data can be found. i still trying. >> i tried restore every metadata.mfs i have, just not work. >> >> >> 2012/3/8 Olivier Thibault <Oli...@lm...> >> >>> Hi, >>> >>> Did you solve your problem ? >>> I had few days ago a mfsmaster crash which went out of memory. >>> When I tried to restart, it crashed saying that there was no >>> metadata.mfs file. >>> I tried "mfsmetarestore -a". It didn't work, saying that >>> metadata.mfs.back was corrupted. >>> There was a metadata.mfs.back file and a metadata.mfs.back.tmp file. >>> metadata.mfs.back was half the size it should be. >>> I restored from a daily backup the metadata.mfs file, then ran again >>> 'mfsmetarestore -a', and this time, it worked. I could then start mfsmaster >>> successfully. >>> Did you tried that ? I mean, just restore the latest working >>> metadata.mfs file ? >>> >>> HTH. >>> >>> Best regards, >>> >>> Olivier >>> >>> >>> >>> Le 07/03/12 04:20, 陈钢 a écrit : >>> >>>> In master`s metadata.mfs,i saw nodes part is complete, part of names, >>>> no free, >>>> no chunks.. >>>> >>>> 2012/3/7 Davies Liu <dav...@gm... <mailto:dav...@gm... >>>> >> >>>> >>>> >>>> Hi, >>>> >>>> I had try to recover it from metadata.mfs and metadata_ml.mfs.back, >>>> but failed. >>>> >>>> Because disk is full, mfsmaster had not dump all the metadata into >>>> disks, >>>> had part of nodes in metadata.mfs, no names, no edges, no chunks. >>>> No hope to recover from the broken metadata.mfs, it's too short. >>>> >>>> The changelogs are also helpless, only in two days. >>>> >>>> The only hope is try to undelete the previous metadata_ml.mfs.back >>>> from >>>> metalogger machine, Chenggang had also failed, with extundelete and >>>> etx3grep, >>>> maybe some experts can archive this. >>>> >>>> The final options is to GUESS the relation between files and chunks >>>> by >>>> chunk id and size fo files, if data lost can not been afforded. Each >>>> chunk is >>>> combined with crc checksum and real data, if we know the relation >>>> between >>>> files and chunksever, then we can get data back. Or we can contruct >>>> the >>>> metadata according to the GUESS, the using mfsmaster to recover them. >>>> >>>> Davies >>>> >>>> On Wed, Mar 7, 2012 at 10:56 AM, Ken <ken...@gm... >>>> <mailto:ken...@gm...>> wrote: >>>> >>>> Hi, chengang >>>> >>>> I think you should try more, and post detail here. Someone must >>>> resolve it. >>>> Maybe you will lost some data in last few minutes, but 250T >>>> should be saved. >>>> >>>> At first, BACKUP all files: >>>> /var/lib/mfs/* on master >>>> /var/lib/mfs/* on mfsmetalogger >>>> >>>> about restore error: >>>> >>>> file 'metadata.mfs.back' not found - will try >>>> 'metadata_ml.mfs.back' >>>> instead >>>> loading objects (files,directories,etc.) ... loading node: >>>> read >>>> error: ENOENT (No such file or directory) >>>> error >>>> can't read metadata from file: .//metadata_ml.mfs.back >>>> >>>> How did you run mfsmetarestore? add -d options? >>>> If stat(datapath + metadata_ml.mfs.back) fail, these error will >>>> occur. >>>> Maybe use strace will show why stat fail exactly. >>>> >>>> ps: I am in Beijing now and I can provide more help. >>>> >>>> HTH >>>> >>>> -Ken >>>> >>>> >>>> >>>> On Wed, Mar 7, 2012 at 10:21 AM, 陈钢 <yik...@gm... >>>> <mailto:yik...@gm...**>> wrote: >>>> >>>> can not start mfsmaster with the file "78962688 Mar 6 17:18 >>>> metadata.mfs ".. >>>> i tried that. :( >>>> >>>> 2012/3/7 Ricardo J. Barberis <ric...@da... >>>> <mailto:ricardo.barberis@**dattatec.com<ric...@da...> >>>> >> >>>> >>>> >>>> El Martes 06/03/2012, 陈钢 escribió: >>>> > on master >>>> [ ... ] >>>> > -rw-r----- 1 mfs mfs 78962688 Mar 6 17:18 >>>> metadata.mfs >>>> > -rw-r--r-- 1 root root 8 Jul 4 2011 >>>> metadata.mfs.empty >>>> > -rw-r----- 1 mfs mfs 5984 Mar 6 12:00 >>>> sessions.mfs >>>> > -rw-r----- 1 mfs mfs 0 Mar 6 16:46 >>>> sessions.mfs.tmp >>>> > -rw-r----- 1 mfs mfs 131072 Mar 6 17:18 >>>> stats.mfs >>>> >>>> You have /var/lib/mfs/metadata.mfs on the master, it >>>> might not >>>> be corrupt >>>> after all? >>>> >>>> I'd suggest: >>>> >>>> - backup /var/lib/mfs to another disk/server (for later >>>> recovery >>>> if needed) >>>> - make sure you have free space in your main disk >>>> - then simply try to start mfsmaster >>>> - check mfs.cgi (web interface) if it looks OK >>>> >>>> >>>> BUT: if you can, wait for confirmation from Michał >>>> Borychowski >>>> first, in case >>>> what I'm telling you is not safe. >>>> >>>> >>>> (BTW: You have Reply-To set to che...@cp... >>>> <mailto:che...@cp...>, I don't know if that's >>>> >>>> intentional on your part) >>>> >>>> Hope it helps, >>>> -- >>>> Ricardo J. Barberis >>>> Senior SysAdmin / ITI >>>> Dattatec.com :: Soluciones de Web Hosting >>>> Tu Hosting hecho Simple! >>>> >>>> ------------------------------**------------ >>>> >>>> >>>> >>>> ------------------------------** >>>> ------------------------------**------------------ >>>> Virtualization & Cloud Management Using Capacity Planning >>>> Cloud computing makes use of virtualization - but cloud >>>> computing >>>> also focuses on allowing computing to be delivered as a >>>> service. >>>> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/> >>>> >>>> ______________________________**_________________ >>>> moosefs-users mailing list >>>> moosefs-users@lists.**sourceforge.net<moo...@li...> >>>> <mailto:moosefs-users@lists.**sourceforge.net<moo...@li...> >>>> > >>>> >>>> https://lists.sourceforge.net/**lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users> >>>> >>>> >>>> >>>> ------------------------------**------------------------------** >>>> ------------------ >>>> Virtualization & Cloud Management Using Capacity Planning >>>> Cloud computing makes use of virtualization - but cloud computing >>>> also focuses on allowing computing to be delivered as a service. >>>> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/> >>>> ______________________________**_________________ >>>> moosefs-users mailing list >>>> moosefs-users@lists.**sourceforge.net<moo...@li...> >>>> <mailto:moosefs-users@lists.**sourceforge.net<moo...@li...> >>>> > >>>> >>>> https://lists.sourceforge.net/**lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users> >>>> >>>> >>>> >>>> >>>> -- >>>> - Davies >>>> >>>> >>>> >>>> >>>> ------------------------------**------------------------------** >>>> ------------------ >>>> Virtualization& Cloud Management Using Capacity Planning >>>> >>>> Cloud computing makes use of virtualization - but cloud computing >>>> also focuses on allowing computing to be delivered as a service. >>>> http://www.accelacomm.com/jaw/**sfnl/114/51521223/<http://www.accelacomm.com/jaw/sfnl/114/51521223/> >>>> >>>> >>>> >>>> ______________________________**_________________ >>>> moosefs-users mailing list >>>> moosefs-users@lists.**sourceforge.net<moo...@li...> >>>> https://lists.sourceforge.net/**lists/listinfo/moosefs-users<https://lists.sourceforge.net/lists/listinfo/moosefs-users> >>>> >>> >>> >>> >> > -- - Davies |
