Menu
▾
▴
Re: [MooseFS-Users] mfsmaster won't start - loading edge: 2502428->5211944 error: empty name
|
From: Ricardo J. B. <ric...@do...> - 2020-09-30 14:51:54
|
Cool, we're in the process of restoring from backups to a new mfs cluster but we'll reconfigure the old cluster to put it online and try this patch. It'll hopefully allow us to recover whatever we don't have in the backups (which are fairly recent but since these are mailboxes we're talking about everything helps). I'll let you know how the process goes. Thank you! El Miércoles 30/09/2020 a las 10:16, Piotr Robert Konopelko escribió: > Sorry, I forgot to add – in order to build MooseFS from sources, you need a > few dependencies. > Please install them as described here: > https://github.com/moosefs/moosefs#source-code. > > Piotr > > *Piotr Robert Konopelko* | m: +48 601 476 440 | e: > pio...@mo... > *Business & Technical Support Manager* > MooseFS Client Support Team > > WWW <https://moosefs.com> | GitHub <https://github.com/moosefs/moosefs> | > Twitter <https://twitter.com/moosefs> | Facebook > <https://www.facebook.com/moosefs> | LinkedIn > <https://www.linkedin.com/company/moosefs> > > > On Wed, Sep 30, 2020 at 3:00 PM Piotr Robert Konopelko < > > pio...@mo...> wrote: > > Hello Ricardo, > > > > After your report, we have updated the Master Server code today to let > > the "*-i*" parameter ignore empty file paths' (edges') names and > > substitute them with "*(empty <inode_number>)*" string, without quotes > > and lower / greater than marks of course. > > > > Please have a look at the following commit: > > > > https://github.com/moosefs/moosefs/commit/886ea4a703afce1b40e4853ce02101a > >8c43829f3 > > > > It will be included in the nearest MooseFS 3.0.115 release. > > > > In order to get this feature before release, please clone the MooseFS Git > > repository from GitHub: https://github.com/moosefs/moosefs, build > > binaries by running *./linux_build.sh* script inside MooseFS Git > > repository directory (this script doesn't run "*make install*", so you > > can just copy (replace) "*mfsmaster*" executable binary file to " > > */usr/sbin/mfsmaster*") and use this newly built "*mfsmaster*" binary > > (don't forget about passing "*-a*" and "*-i*" parameters to it). It > > should be able to load your metadata and substitute empty names with the > > above mentioned string. > > > > Please let me know if it worked for you and do not hesitate to contact us > > if you have any questions. > > > > Best regards, > > Piotr > > > > *Piotr Robert Konopelko* | m: +48 601 476 440 | e: > > pio...@mo... > > *Business & Technical Support Manager* > > MooseFS Client Support Team > > > > WWW <https://moosefs.com> | GitHub <https://github.com/moosefs/moosefs> | > > Twitter <https://twitter.com/moosefs> | Facebook > > <https://www.facebook.com/moosefs> | LinkedIn > > <https://www.linkedin.com/company/moosefs> > > > > > > On Wed, Sep 30, 2020 at 2:34 AM Ricardo J. Barberis < > > > > ric...@do...> wrote: > >> El Martes 29/09/2020 a las 20:55, Piotr Robert Konopelko escribió: > >> > Hello Ricardo, > >> > > >> > 1) Have you done a backup of /var/lib/mfs after the failure? If not, > >> > >> please > >> > >> > do so, just in case. > >> > >> Yes, I have a backup post-crash, before atempting to recover > >> > >> > 2) Have you tried the following sequence: > >> > > >> > - Stopping the Metalogger, > >> > - Copying all the metadata_ml* files and changelog_ml* files from > >> > Metalogger's /var/lib/mfs directory to Master Server's /var/lib/mfs > >> > directory, > >> > - Starting the Master server with "-a" parameter that makes it > >> > >> trying to > >> > >> > recover the metadata: > >> > > >> > *mfsmaster -a* > >> > > >> > ? > >> > >> Yes, I tried this sequence in the master server and also in the > >> metalogger, > >> copying the files to a new directory (so I also got a backup of the > >> files from the metalogger). > >> > >> > The above sequence is usually the best way to recover the metadata. > >> > >> I also tried 'mfsmaster -a -i' to no avail. > >> > >> > >> BTW, all of this is in my original email, sorry if I wasn't very clear. > >> > >> > Best regards, > >> > Piotr > >> > > >> > *Piotr Robert Konopelko* | m: +48 601 476 440 | e: > >> > pio...@mo... > >> > *Business & Technical Support Manager* > >> > MooseFS Client Support Team > >> > > >> > WWW <https://moosefs.com> | GitHub > >> > <https://github.com/moosefs/moosefs> > >> > > >> > Twitter <https://twitter.com/moosefs> | Facebook > >> > <https://www.facebook.com/moosefs> | LinkedIn > >> > <https://www.linkedin.com/company/moosefs> > >> > > >> > > >> > On Wed, Sep 30, 2020 at 1:46 AM Ricardo J. Barberis < > >> > > >> > ric...@do...> wrote: > >> > > Hi all, > >> > > > >> > > My mfsmaster crashed crashed today and when trying to start it I get > >> > >> this > >> > >> > > error: > >> > > > >> > > loading edge: 2502428->5211944 error: empty name > >> > > > >> > > > >> > > I tried with metadata.mfs.back, metadata.mfs.back.1 from master, and > >> > >> also > >> > >> > > metadata_ml.mfs.back and metadata_ml.mfs.back.1 from metalogger, all > >> > >> of > >> > >> > > them > >> > > fail with the same error. > >> > > > >> > > > >> > > My mfsmaster was 3.0.100 when it crashed, I upgraded to 3.0.114 but > >> > >> it's > >> > >> > > the same: > >> > > > >> > > # mfsmaster -v > >> > > version: 3.0.114-1 ; build: 1257 > >> > > > >> > > > >> > > Any hints to solve this much appreciated, as this is a production > >> > > cluster. > >> > > > >> > > > >> > > I have dumped all the metadata files and all of them have this: > >> > > > >> > > # mfsmetadump metadata.mfs.back > metadata.mfs.dump > >> > > # grep 2502428 metadata.mfs.dump | grep -1 5211944 > >> > > EDGE|p: 2502428|c: 5154814|i:0x7FFFFFFFF92D4A30|n: > >> > > 1519914956.H419257P2952.c106-dr.dattaweb.com:2,S > >> > > EDGE|p: 2502428|c: 5211944|i:0x000000095F864814|n: > >> > > EDGE|p: 2502428|c: 2920292|i:0x7FFFFFFFF947F4F2|n: > >> > > 1519809696.H296554P8952.c105-dr.dattaweb.com:2,S > >> > > > >> > > > >> > > Full output of 'mfsmaster -xx -a -i -c /etc/mfs/mfsmaster.cfg': > >> > > > >> > > open files limit has been set to: 16384 > >> > > working directory: /mnt/mailmfs/mfs > >> > > lockfile created and locked > >> > > initializing mfsmaster modules ... > >> > > exports file has been loaded > >> > > topology file has been loaded > >> > > write replication limit in old format - change limits to new format > >> > > read replication limit in old format - change limits to new format > >> > > loading metadata ... > >> > > found valid metadata file: metadata.mfs.back.1 (version: 16411110888 > >> > > ; id: 59AEE396E5130F20) > >> > > found invalid metadata file (wrong header): metadata.crc > >> > > found valid metadata file: metadata.mfs.back (version: 16412023641 ; > >> > >> id: > >> > > 59AEE396E5130F20) > >> > > chosen most recent metadata file: metadata.mfs.back (version: > >> > >> 16412023641 > >> > >> > > ; id: 59AEE396E5130F20) > >> > > loading sessions data ... ok (0.0000) > >> > > loading storage classes data ... ok (0.0000) > >> > > loading objects (files,directories,etc.) ... ok (1.1999) > >> > > loading names ... > >> > > loading edge: 2502428->5211944 error: empty name > >> > > cleaning metadata ... > >> > > cleaning objects ... done > >> > > cleaning names ... done > >> > > cleaning deletion timestamps ... done > >> > > cleaning quota definitions ... done > >> > > cleaning chunks data ...done > >> > > cleaning xattr data ...done > >> > > cleaning posix_acl data ...done > >> > > cleaning flock locks data ...done > >> > > cleaning posix locks data ...done > >> > > cleaning chunkservers data ...done > >> > > cleaning open files data ...done > >> > > cleaning sessions data ...done > >> > > cleaning storage classes data ...done > >> > > cleaning dictionary data ...done > >> > > metadata have been cleaned > >> > > error loading metadata file (metadata.mfs.back): ENOENT (No such > >> > > file > >> > >> or > >> > >> > > directory) > >> > > init: metadata manager failed !!! > >> > > error occurred during initialization - exiting -- Ricardo J. Barberis Senior SysAdmin / IT Architect DonWeb La Actitud Es Todo www.DonWeb.com _____ |
