Re: [MooseFS-Users] mfsmaster won't start - loading edge: 2502428->5211944 error: empty name

[Piotr R. K. <pio...@mo...>]

Sorry, I forgot to add – in order to build MooseFS from sources, you need a
few dependencies.
Please install them as described here:
https://github.com/moosefs/moosefs#source-code.

Piotr

*Piotr Robert Konopelko* | m: +48 601 476 440 | e:
pio...@mo...
*Business & Technical Support Manager*
MooseFS Client Support Team

WWW <https://moosefs.com> | GitHub <https://github.com/moosefs/moosefs> |
Twitter <https://twitter.com/moosefs> | Facebook
<https://www.facebook.com/moosefs> | LinkedIn
<https://www.linkedin.com/company/moosefs>


On Wed, Sep 30, 2020 at 3:00 PM Piotr Robert Konopelko <
pio...@mo...> wrote:

> Hello Ricardo,
>
> After your report, we have updated the Master Server code today to let the
> "*-i*" parameter ignore empty file paths' (edges') names and substitute
> them with "*(empty <inode_number>)*" string, without quotes and lower /
> greater than marks of course.
>
> Please have a look at the following commit:
>
> https://github.com/moosefs/moosefs/commit/886ea4a703afce1b40e4853ce02101a8c43829f3
>
> It will be included in the nearest MooseFS 3.0.115 release.
>
> In order to get this feature before release, please clone the MooseFS Git
> repository from GitHub: https://github.com/moosefs/moosefs, build
> binaries by running *./linux_build.sh* script inside MooseFS Git
> repository directory (this script doesn't run "*make install*", so you
> can just copy (replace) "*mfsmaster*" executable binary file to "
> */usr/sbin/mfsmaster*") and use this newly built "*mfsmaster*" binary
> (don't forget about passing "*-a*" and "*-i*" parameters to it). It
> should be able to load your metadata and substitute empty names with the
> above mentioned string.
>
> Please let me know if it worked for you and do not hesitate to contact us
> if you have any questions.
>
> Best regards,
> Piotr
>
> *Piotr Robert Konopelko* | m: +48 601 476 440 | e:
> pio...@mo...
> *Business & Technical Support Manager*
> MooseFS Client Support Team
>
> WWW <https://moosefs.com> | GitHub <https://github.com/moosefs/moosefs> |
> Twitter <https://twitter.com/moosefs> | Facebook
> <https://www.facebook.com/moosefs> | LinkedIn
> <https://www.linkedin.com/company/moosefs>
>
>
> On Wed, Sep 30, 2020 at 2:34 AM Ricardo J. Barberis <
> ric...@do...> wrote:
>
>> El Martes 29/09/2020 a las 20:55, Piotr Robert Konopelko escribió:
>> > Hello Ricardo,
>> >
>> > 1) Have you done a backup of /var/lib/mfs after the failure? If not,
>> please
>> > do so, just in case.
>>
>> Yes, I have a backup post-crash, before atempting to recover
>>
>> > 2) Have you tried the following sequence:
>> >
>> >    - Stopping the Metalogger,
>> >    - Copying all the metadata_ml* files and changelog_ml* files from
>> >    Metalogger's /var/lib/mfs directory to Master Server's /var/lib/mfs
>> >    directory,
>> >    - Starting the Master server with "-a" parameter that makes it
>> trying to
>> >    recover the metadata:
>> >
>> >    *mfsmaster -a*
>> >
>> > ?
>>
>> Yes, I tried this sequence in the master server and also in the
>> metalogger,
>> copying the files to a new directory (so I also got a backup of the files
>> from the metalogger).
>>
>> > The above sequence is usually the best way to recover the metadata.
>>
>> I also tried 'mfsmaster -a -i' to no avail.
>>
>>
>> BTW, all of this is in my original email, sorry if I wasn't very clear.
>>
>> > Best regards,
>> > Piotr
>> >
>> > *Piotr Robert Konopelko* | m: +48 601 476 440 | e:
>> > pio...@mo...
>> > *Business & Technical Support Manager*
>> > MooseFS Client Support Team
>> >
>> > WWW <https://moosefs.com> | GitHub <https://github.com/moosefs/moosefs>
>> |
>> > Twitter <https://twitter.com/moosefs> | Facebook
>> > <https://www.facebook.com/moosefs> | LinkedIn
>> > <https://www.linkedin.com/company/moosefs>
>> >
>> >
>> > On Wed, Sep 30, 2020 at 1:46 AM Ricardo J. Barberis <
>> >
>> > ric...@do...> wrote:
>> > > Hi all,
>> > >
>> > > My mfsmaster crashed crashed today and when trying to start it I get
>> this
>> > > error:
>> > >
>> > > loading edge: 2502428->5211944 error: empty name
>> > >
>> > >
>> > > I tried with metadata.mfs.back, metadata.mfs.back.1 from master, and
>> also
>> > > metadata_ml.mfs.back and metadata_ml.mfs.back.1 from metalogger, all
>> of
>> > > them
>> > > fail with the same error.
>> > >
>> > >
>> > > My mfsmaster was 3.0.100 when it crashed, I upgraded to 3.0.114 but
>> it's
>> > > the same:
>> > >
>> > > # mfsmaster -v
>> > > version: 3.0.114-1 ; build: 1257
>> > >
>> > >
>> > > Any hints to solve this much appreciated, as this is a production
>> > > cluster.
>> > >
>> > >
>> > > I have dumped all the metadata files and all of them have this:
>> > >
>> > > # mfsmetadump metadata.mfs.back > metadata.mfs.dump
>> > > # grep 2502428 metadata.mfs.dump | grep -1 5211944
>> > > EDGE|p:   2502428|c:   5154814|i:0x7FFFFFFFF92D4A30|n:
>> > > 1519914956.H419257P2952.c106-dr.dattaweb.com:2,S
>> > > EDGE|p:   2502428|c:   5211944|i:0x000000095F864814|n:
>> > > EDGE|p:   2502428|c:   2920292|i:0x7FFFFFFFF947F4F2|n:
>> > > 1519809696.H296554P8952.c105-dr.dattaweb.com:2,S
>> > >
>> > >
>> > > Full output of 'mfsmaster -xx -a -i -c /etc/mfs/mfsmaster.cfg':
>> > >
>> > > open files limit has been set to: 16384
>> > > working directory: /mnt/mailmfs/mfs
>> > > lockfile created and locked
>> > > initializing mfsmaster modules ...
>> > > exports file has been loaded
>> > > topology file has been loaded
>> > > write replication limit in old format - change limits to new format
>> > > read replication limit in old format - change limits to new format
>> > > loading metadata ...
>> > > found valid metadata file: metadata.mfs.back.1 (version: 16411110888 ;
>> > > id: 59AEE396E5130F20)
>> > > found invalid metadata file (wrong header): metadata.crc
>> > > found valid metadata file: metadata.mfs.back (version: 16412023641 ;
>> id:
>> > > 59AEE396E5130F20)
>> > > chosen most recent metadata file: metadata.mfs.back (version:
>> 16412023641
>> > > ; id: 59AEE396E5130F20)
>> > > loading sessions data ... ok (0.0000)
>> > > loading storage classes data ... ok (0.0000)
>> > > loading objects (files,directories,etc.) ... ok (1.1999)
>> > > loading names ...
>> > > loading edge: 2502428->5211944 error: empty name
>> > > cleaning metadata ...
>> > > cleaning objects ... done
>> > > cleaning names ... done
>> > > cleaning deletion timestamps ... done
>> > > cleaning quota definitions ... done
>> > > cleaning chunks data ...done
>> > > cleaning xattr data ...done
>> > > cleaning posix_acl data ...done
>> > > cleaning flock locks data ...done
>> > > cleaning posix locks data ...done
>> > > cleaning chunkservers data ...done
>> > > cleaning open files data ...done
>> > > cleaning sessions data ...done
>> > > cleaning storage classes data ...done
>> > > cleaning dictionary data ...done
>> > > metadata have been cleaned
>> > > error loading metadata file (metadata.mfs.back): ENOENT (No such file
>> or
>> > > directory)
>> > > init: metadata manager failed !!!
>> > > error occurred during initialization - exiting
>> --
>> Ricardo J. Barberis
>> Senior SysAdmin / IT Architect
>> DonWeb
>> La Actitud Es Todo
>> www.DonWeb.com
>>   _____
>>
>