Re: [Moosefs-users] Question about MooseFS metadata distribution/recovery

[Michał B. <mic...@ge...>]

If you have "invalid copies" you should use mfsfilerepair which will put in metadata proper file version. It's important that you have all chunkservers running while doing mfsfilerepair.


Kind regards
Michał Borychowski 
MooseFS Support Manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Gemius S.A.
ul. Wołoska 7, 02-672 Warszawa
Budynek MARS, klatka D
Tel.: +4822 874-41-00
Fax : +4822 874-41-01


-----Original Message-----
From: René Pfeiffer [mailto:ly...@lu...] 
Sent: Wednesday, December 14, 2011 11:01 AM
To: Michał Borychowski
Cc: moo...@li...
Subject: Re: [Moosefs-users] Question about MooseFS metadata distribution/recovery

On Dec 14, 2011 at 0937 +0100, Michał Borychowski appeared and said:
> Hi!
> 

> This is still too little information about how we could help you...

Sorry, I know, it's been a bit stressful since the server failure.

> What about metaloggers? Don't you have metadata_ml.mfs.back and 
> changelog_ml.*.mfs files?

Yes, I have, and we used the metalogger's data in order to set up a new master, but the access errors (invalid chunks and such) stayed.

> You could put them on ftp as tar.gz and give us a link so that we try 
> to recover them.

I will prepare an archive and make it available.

> In "emergency" situations MooseFS tries to write metadata file on the master machine in these locations:
> /metadata.mfs.emergency
> /tmp/metadata.mfs.emergency
> /var/metadata.mfs.emergency
> /usr/metadata.mfs.emergency
> /usr/share/metadata.mfs.emergency
> /usr/local/metadata.mfs.emergency
> /usr/local/var/metadata.mfs.emergency
> /usr/local/share/metadata.mfs.emergency
> 
> You can try to find them there, but as RAID went broken it is possible you won't see them.

The data rescue company gave us the salvaged data and we found not a single *.emergency file on the dump. The file might have been renamed though. I suspect that the RAID controller suffered from a data corruption condition for weeks prior to completely failing. Not even the Linux kernel got sensible warnings from the block device.

Since the RAID controller totally locked up the system I'm not sure if the master process was able to write the data to disk (especially because all storage containers were managed by the faulty RAID controller on this server).

> And it is impossible to recover metadata from the chunks themselves.

Yes, I know, but I just asked, because I thought that there's a way to scan the data on the chunkservers themselves.

> But file data are in the chunks so if you need to find something 
> specific it would be possible. Each chunk has 5kb header (plain text) 
> so simple 'grep' would be enough to find what you are looking at.

We mainly deal with image files (JPG and PNG). I guess we'll walk through the chunks this way.

> BTW. How much data was kept on this MooseFS installation?

About 7.7 GB, mostly image files and a few videos. The images are the most important data, but the application dealing with these files unfortunately used hashes and a drectory structure similar to the Squid proxy (multiple directories, path based on the hashes) to store the data. I'm not sure if the developers can reconstruct this information without the metadata.

Best regards,
René.

--
  )\._.,--....,'``.  fL  Let GNU/Linux work for you while you take a nap.
 /,   _.. \   _\  (`._ ,. R. Pfeiffer <lynx at luchs.at> + http://web.luchs.at/
`._.-(,_..'--(,_..'`-.;.'  - System administration + Consulting + Teaching - Got mail delivery problems?  http://web.luchs.at/information/blockedmail.php