Re: [MooseFS-Users] mfsmaster won't start - loading edge: 2502428->5211944 error: empty name

[Ricardo J. B. <ric...@do...>]

El Martes 29/09/2020 a las 20:55, Piotr Robert Konopelko escribió:
> Hello Ricardo,
>
> 1) Have you done a backup of /var/lib/mfs after the failure? If not, please
> do so, just in case.

Yes, I have a backup post-crash, before atempting to recover

> 2) Have you tried the following sequence:
>
>    - Stopping the Metalogger,
>    - Copying all the metadata_ml* files and changelog_ml* files from
>    Metalogger's /var/lib/mfs directory to Master Server's /var/lib/mfs
>    directory,
>    - Starting the Master server with "-a" parameter that makes it trying to
>    recover the metadata:
>
>    *mfsmaster -a*
>
> ?

Yes, I tried this sequence in the master server and also in the metalogger, 
copying the files to a new directory (so I also got a backup of the files 
from the metalogger).

> The above sequence is usually the best way to recover the metadata.

I also tried 'mfsmaster -a -i' to no avail.


BTW, all of this is in my original email, sorry if I wasn't very clear.

> Best regards,
> Piotr
>
> *Piotr Robert Konopelko* | m: +48 601 476 440 | e:
> pio...@mo...
> *Business & Technical Support Manager*
> MooseFS Client Support Team
>
> WWW <https://moosefs.com> | GitHub <https://github.com/moosefs/moosefs> |
> Twitter <https://twitter.com/moosefs> | Facebook
> <https://www.facebook.com/moosefs> | LinkedIn
> <https://www.linkedin.com/company/moosefs>
>
>
> On Wed, Sep 30, 2020 at 1:46 AM Ricardo J. Barberis <
>
> ric...@do...> wrote:
> > Hi all,
> >
> > My mfsmaster crashed crashed today and when trying to start it I get this
> > error:
> >
> > loading edge: 2502428->5211944 error: empty name
> >
> >
> > I tried with metadata.mfs.back, metadata.mfs.back.1 from master, and also
> > metadata_ml.mfs.back and metadata_ml.mfs.back.1 from metalogger, all of
> > them
> > fail with the same error.
> >
> >
> > My mfsmaster was 3.0.100 when it crashed, I upgraded to 3.0.114 but it's
> > the same:
> >
> > # mfsmaster -v
> > version: 3.0.114-1 ; build: 1257
> >
> >
> > Any hints to solve this much appreciated, as this is a production
> > cluster.
> >
> >
> > I have dumped all the metadata files and all of them have this:
> >
> > # mfsmetadump metadata.mfs.back > metadata.mfs.dump
> > # grep 2502428 metadata.mfs.dump | grep -1 5211944
> > EDGE|p:   2502428|c:   5154814|i:0x7FFFFFFFF92D4A30|n:
> > 1519914956.H419257P2952.c106-dr.dattaweb.com:2,S
> > EDGE|p:   2502428|c:   5211944|i:0x000000095F864814|n:
> > EDGE|p:   2502428|c:   2920292|i:0x7FFFFFFFF947F4F2|n:
> > 1519809696.H296554P8952.c105-dr.dattaweb.com:2,S
> >
> >
> > Full output of 'mfsmaster -xx -a -i -c /etc/mfs/mfsmaster.cfg':
> >
> > open files limit has been set to: 16384
> > working directory: /mnt/mailmfs/mfs
> > lockfile created and locked
> > initializing mfsmaster modules ...
> > exports file has been loaded
> > topology file has been loaded
> > write replication limit in old format - change limits to new format
> > read replication limit in old format - change limits to new format
> > loading metadata ...
> > found valid metadata file: metadata.mfs.back.1 (version: 16411110888 ;
> > id: 59AEE396E5130F20)
> > found invalid metadata file (wrong header): metadata.crc
> > found valid metadata file: metadata.mfs.back (version: 16412023641 ; id:
> > 59AEE396E5130F20)
> > chosen most recent metadata file: metadata.mfs.back (version: 16412023641
> > ; id: 59AEE396E5130F20)
> > loading sessions data ... ok (0.0000)
> > loading storage classes data ... ok (0.0000)
> > loading objects (files,directories,etc.) ... ok (1.1999)
> > loading names ...
> > loading edge: 2502428->5211944 error: empty name
> > cleaning metadata ...
> > cleaning objects ... done
> > cleaning names ... done
> > cleaning deletion timestamps ... done
> > cleaning quota definitions ... done
> > cleaning chunks data ...done
> > cleaning xattr data ...done
> > cleaning posix_acl data ...done
> > cleaning flock locks data ...done
> > cleaning posix locks data ...done
> > cleaning chunkservers data ...done
> > cleaning open files data ...done
> > cleaning sessions data ...done
> > cleaning storage classes data ...done
> > cleaning dictionary data ...done
> > metadata have been cleaned
> > error loading metadata file (metadata.mfs.back): ENOENT (No such file or
> > directory)
> > init: metadata manager failed !!!
> > error occurred during initialization - exiting
-- 
Ricardo J. Barberis
Senior SysAdmin / IT Architect
DonWeb
La Actitud Es Todo
www.DonWeb.com
  _____