Re: [MooseFS-Users] mfsmaster won't start - loading edge: 2502428->5211944 error: empty name

[Piotr R. K. <pio...@mo...>]

Hello Ricardo,

After your report, we have updated the Master Server code today to let the "
*-i*" parameter ignore empty file paths' (edges') names and substitute them
with "*(empty <inode_number>)*" string, without quotes and lower / greater
than marks of course.

Please have a look at the following commit:
https://github.com/moosefs/moosefs/commit/886ea4a703afce1b40e4853ce02101a8c43829f3

It will be included in the nearest MooseFS 3.0.115 release.

In order to get this feature before release, please clone the MooseFS Git
repository from GitHub: https://github.com/moosefs/moosefs, build binaries
by running *./linux_build.sh* script inside MooseFS Git repository
directory (this script doesn't run "*make install*", so you can just copy
(replace) "*mfsmaster*" executable binary file to "*/usr/sbin/mfsmaster*") and
use this newly built "*mfsmaster*" binary (don't forget about passing "*-a*"
and "*-i*" parameters to it). It should be able to load your metadata and
substitute empty names with the above mentioned string.

Please let me know if it worked for you and do not hesitate to contact us
if you have any questions.

Best regards,
Piotr

*Piotr Robert Konopelko* | m: +48 601 476 440 | e:
pio...@mo...
*Business & Technical Support Manager*
MooseFS Client Support Team

WWW <https://moosefs.com> | GitHub <https://github.com/moosefs/moosefs> |
Twitter <https://twitter.com/moosefs> | Facebook
<https://www.facebook.com/moosefs> | LinkedIn
<https://www.linkedin.com/company/moosefs>


On Wed, Sep 30, 2020 at 2:34 AM Ricardo J. Barberis <
ric...@do...> wrote:

> El Martes 29/09/2020 a las 20:55, Piotr Robert Konopelko escribió:
> > Hello Ricardo,
> >
> > 1) Have you done a backup of /var/lib/mfs after the failure? If not,
> please
> > do so, just in case.
>
> Yes, I have a backup post-crash, before atempting to recover
>
> > 2) Have you tried the following sequence:
> >
> >    - Stopping the Metalogger,
> >    - Copying all the metadata_ml* files and changelog_ml* files from
> >    Metalogger's /var/lib/mfs directory to Master Server's /var/lib/mfs
> >    directory,
> >    - Starting the Master server with "-a" parameter that makes it trying
> to
> >    recover the metadata:
> >
> >    *mfsmaster -a*
> >
> > ?
>
> Yes, I tried this sequence in the master server and also in the
> metalogger,
> copying the files to a new directory (so I also got a backup of the files
> from the metalogger).
>
> > The above sequence is usually the best way to recover the metadata.
>
> I also tried 'mfsmaster -a -i' to no avail.
>
>
> BTW, all of this is in my original email, sorry if I wasn't very clear.
>
> > Best regards,
> > Piotr
> >
> > *Piotr Robert Konopelko* | m: +48 601 476 440 | e:
> > pio...@mo...
> > *Business & Technical Support Manager*
> > MooseFS Client Support Team
> >
> > WWW <https://moosefs.com> | GitHub <https://github.com/moosefs/moosefs>
> |
> > Twitter <https://twitter.com/moosefs> | Facebook
> > <https://www.facebook.com/moosefs> | LinkedIn
> > <https://www.linkedin.com/company/moosefs>
> >
> >
> > On Wed, Sep 30, 2020 at 1:46 AM Ricardo J. Barberis <
> >
> > ric...@do...> wrote:
> > > Hi all,
> > >
> > > My mfsmaster crashed crashed today and when trying to start it I get
> this
> > > error:
> > >
> > > loading edge: 2502428->5211944 error: empty name
> > >
> > >
> > > I tried with metadata.mfs.back, metadata.mfs.back.1 from master, and
> also
> > > metadata_ml.mfs.back and metadata_ml.mfs.back.1 from metalogger, all of
> > > them
> > > fail with the same error.
> > >
> > >
> > > My mfsmaster was 3.0.100 when it crashed, I upgraded to 3.0.114 but
> it's
> > > the same:
> > >
> > > # mfsmaster -v
> > > version: 3.0.114-1 ; build: 1257
> > >
> > >
> > > Any hints to solve this much appreciated, as this is a production
> > > cluster.
> > >
> > >
> > > I have dumped all the metadata files and all of them have this:
> > >
> > > # mfsmetadump metadata.mfs.back > metadata.mfs.dump
> > > # grep 2502428 metadata.mfs.dump | grep -1 5211944
> > > EDGE|p:   2502428|c:   5154814|i:0x7FFFFFFFF92D4A30|n:
> > > 1519914956.H419257P2952.c106-dr.dattaweb.com:2,S
> > > EDGE|p:   2502428|c:   5211944|i:0x000000095F864814|n:
> > > EDGE|p:   2502428|c:   2920292|i:0x7FFFFFFFF947F4F2|n:
> > > 1519809696.H296554P8952.c105-dr.dattaweb.com:2,S
> > >
> > >
> > > Full output of 'mfsmaster -xx -a -i -c /etc/mfs/mfsmaster.cfg':
> > >
> > > open files limit has been set to: 16384
> > > working directory: /mnt/mailmfs/mfs
> > > lockfile created and locked
> > > initializing mfsmaster modules ...
> > > exports file has been loaded
> > > topology file has been loaded
> > > write replication limit in old format - change limits to new format
> > > read replication limit in old format - change limits to new format
> > > loading metadata ...
> > > found valid metadata file: metadata.mfs.back.1 (version: 16411110888 ;
> > > id: 59AEE396E5130F20)
> > > found invalid metadata file (wrong header): metadata.crc
> > > found valid metadata file: metadata.mfs.back (version: 16412023641 ;
> id:
> > > 59AEE396E5130F20)
> > > chosen most recent metadata file: metadata.mfs.back (version:
> 16412023641
> > > ; id: 59AEE396E5130F20)
> > > loading sessions data ... ok (0.0000)
> > > loading storage classes data ... ok (0.0000)
> > > loading objects (files,directories,etc.) ... ok (1.1999)
> > > loading names ...
> > > loading edge: 2502428->5211944 error: empty name
> > > cleaning metadata ...
> > > cleaning objects ... done
> > > cleaning names ... done
> > > cleaning deletion timestamps ... done
> > > cleaning quota definitions ... done
> > > cleaning chunks data ...done
> > > cleaning xattr data ...done
> > > cleaning posix_acl data ...done
> > > cleaning flock locks data ...done
> > > cleaning posix locks data ...done
> > > cleaning chunkservers data ...done
> > > cleaning open files data ...done
> > > cleaning sessions data ...done
> > > cleaning storage classes data ...done
> > > cleaning dictionary data ...done
> > > metadata have been cleaned
> > > error loading metadata file (metadata.mfs.back): ENOENT (No such file
> or
> > > directory)
> > > init: metadata manager failed !!!
> > > error occurred during initialization - exiting
> --
> Ricardo J. Barberis
> Senior SysAdmin / IT Architect
> DonWeb
> La Actitud Es Todo
> www.DonWeb.com
>   _____
>