Re: [MooseFS-Users] mfsmaster won't start - loading edge: 2502428->5211944 error: empty name

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

That's great, thank you for the reply. Please definitely try this patch and
let us know.
Looking forward to hearing from you.

Best regards,
Piotr

*Piotr Robert Konopelko* | m: +48 601 476 440 | e:
pio...@mo...
*Business & Technical Support Manager*
MooseFS Client Support Team

WWW <https://moosefs.com> | GitHub <https://github.com/moosefs/moosefs> |
Twitter <https://twitter.com/moosefs> | Facebook
<https://www.facebook.com/moosefs> | LinkedIn
<https://www.linkedin.com/company/moosefs>

On Wed, Sep 30, 2020 at 4:51 PM Ricardo J. Barberis <
ric...@do...> wrote:

> Cool, we're in the process of restoring from backups to a new mfs cluster
> but
> we'll reconfigure the old cluster to put it online and try this patch.
>
> It'll hopefully allow us to recover whatever we don't have in the backups
> (which are fairly recent but since these are mailboxes we're talking about
> everything helps).
>
> I'll let you know how the process goes.
>
> Thank you!
>
> El Miércoles 30/09/2020 a las 10:16, Piotr Robert Konopelko escribió:
> > Sorry, I forgot to add – in order to build MooseFS from sources, you
> need a
> > few dependencies.
> > Please install them as described here:
> > https://github.com/moosefs/moosefs#source-code.
> >
> > Piotr
> >
> > *Piotr Robert Konopelko* | m: +48 601 476 440 | e:
> > pio...@mo...
> > *Business & Technical Support Manager*
> > MooseFS Client Support Team
> >
> > WWW <https://moosefs.com> | GitHub <https://github.com/moosefs/moosefs>
> |
> > Twitter <https://twitter.com/moosefs> | Facebook
> > <https://www.facebook.com/moosefs> | LinkedIn
> > <https://www.linkedin.com/company/moosefs>
> >
> >
> > On Wed, Sep 30, 2020 at 3:00 PM Piotr Robert Konopelko <
> >
> > pio...@mo...> wrote:
> > > Hello Ricardo,
> > >
> > > After your report, we have updated the Master Server code today to let
> > > the "*-i*" parameter ignore empty file paths' (edges') names and
> > > substitute them with "*(empty <inode_number>)*" string, without quotes
> > > and lower / greater than marks of course.
> > >
> > > Please have a look at the following commit:
> > >
> > >
> https://github.com/moosefs/moosefs/commit/886ea4a703afce1b40e4853ce02101a
> > >8c43829f3
> > >
> > > It will be included in the nearest MooseFS 3.0.115 release.
> > >
> > > In order to get this feature before release, please clone the MooseFS
> Git
> > > repository from GitHub: https://github.com/moosefs/moosefs, build
> > > binaries by running *./linux_build.sh* script inside MooseFS Git
> > > repository directory (this script doesn't run "*make install*", so you
> > > can just copy (replace) "*mfsmaster*" executable binary file to "
> > > */usr/sbin/mfsmaster*") and use this newly built "*mfsmaster*" binary
> > > (don't forget about passing "*-a*" and "*-i*" parameters to it). It
> > > should be able to load your metadata and substitute empty names with
> the
> > > above mentioned string.
> > >
> > > Please let me know if it worked for you and do not hesitate to contact
> us
> > > if you have any questions.
> > >
> > > Best regards,
> > > Piotr
> > >
> > > *Piotr Robert Konopelko* | m: +48 601 476 440 | e:
> > > pio...@mo...
> > > *Business & Technical Support Manager*
> > > MooseFS Client Support Team
> > >
> > > WWW <https://moosefs.com> | GitHub <https://github.com/moosefs/moosefs>
> |
> > > Twitter <https://twitter.com/moosefs> | Facebook
> > > <https://www.facebook.com/moosefs> | LinkedIn
> > > <https://www.linkedin.com/company/moosefs>
> > >
> > >
> > > On Wed, Sep 30, 2020 at 2:34 AM Ricardo J. Barberis <
> > >
> > > ric...@do...> wrote:
> > >> El Martes 29/09/2020 a las 20:55, Piotr Robert Konopelko escribió:
> > >> > Hello Ricardo,
> > >> >
> > >> > 1) Have you done a backup of /var/lib/mfs after the failure? If not,
> > >>
> > >> please
> > >>
> > >> > do so, just in case.
> > >>
> > >> Yes, I have a backup post-crash, before atempting to recover
> > >>
> > >> > 2) Have you tried the following sequence:
> > >> >
> > >> >    - Stopping the Metalogger,
> > >> >    - Copying all the metadata_ml* files and changelog_ml* files from
> > >> >    Metalogger's /var/lib/mfs directory to Master Server's
> /var/lib/mfs
> > >> >    directory,
> > >> >    - Starting the Master server with "-a" parameter that makes it
> > >>
> > >> trying to
> > >>
> > >> >    recover the metadata:
> > >> >
> > >> >    *mfsmaster -a*
> > >> >
> > >> > ?
> > >>
> > >> Yes, I tried this sequence in the master server and also in the
> > >> metalogger,
> > >> copying the files to a new directory (so I also got a backup of the
> > >> files from the metalogger).
> > >>
> > >> > The above sequence is usually the best way to recover the metadata.
> > >>
> > >> I also tried 'mfsmaster -a -i' to no avail.
> > >>
> > >>
> > >> BTW, all of this is in my original email, sorry if I wasn't very
> clear.
> > >>
> > >> > Best regards,
> > >> > Piotr
> > >> >
> > >> > *Piotr Robert Konopelko* | m: +48 601 476 440 | e:
> > >> > pio...@mo...
> > >> > *Business & Technical Support Manager*
> > >> > MooseFS Client Support Team
> > >> >
> > >> > WWW <https://moosefs.com> | GitHub
> > >> > <https://github.com/moosefs/moosefs>
> > >> >
> > >> > Twitter <https://twitter.com/moosefs> | Facebook
> > >> > <https://www.facebook.com/moosefs> | LinkedIn
> > >> > <https://www.linkedin.com/company/moosefs>
> > >> >
> > >> >
> > >> > On Wed, Sep 30, 2020 at 1:46 AM Ricardo J. Barberis <
> > >> >
> > >> > ric...@do...> wrote:
> > >> > > Hi all,
> > >> > >
> > >> > > My mfsmaster crashed crashed today and when trying to start it I
> get
> > >>
> > >> this
> > >>
> > >> > > error:
> > >> > >
> > >> > > loading edge: 2502428->5211944 error: empty name
> > >> > >
> > >> > >
> > >> > > I tried with metadata.mfs.back, metadata.mfs.back.1 from master,
> and
> > >>
> > >> also
> > >>
> > >> > > metadata_ml.mfs.back and metadata_ml.mfs.back.1 from metalogger,
> all
> > >>
> > >> of
> > >>
> > >> > > them
> > >> > > fail with the same error.
> > >> > >
> > >> > >
> > >> > > My mfsmaster was 3.0.100 when it crashed, I upgraded to 3.0.114
> but
> > >>
> > >> it's
> > >>
> > >> > > the same:
> > >> > >
> > >> > > # mfsmaster -v
> > >> > > version: 3.0.114-1 ; build: 1257
> > >> > >
> > >> > >
> > >> > > Any hints to solve this much appreciated, as this is a production
> > >> > > cluster.
> > >> > >
> > >> > >
> > >> > > I have dumped all the metadata files and all of them have this:
> > >> > >
> > >> > > # mfsmetadump metadata.mfs.back > metadata.mfs.dump
> > >> > > # grep 2502428 metadata.mfs.dump | grep -1 5211944
> > >> > > EDGE|p:   2502428|c:   5154814|i:0x7FFFFFFFF92D4A30|n:
> > >> > > 1519914956.H419257P2952.c106-dr.dattaweb.com:2,S
> > >> > > EDGE|p:   2502428|c:   5211944|i:0x000000095F864814|n:
> > >> > > EDGE|p:   2502428|c:   2920292|i:0x7FFFFFFFF947F4F2|n:
> > >> > > 1519809696.H296554P8952.c105-dr.dattaweb.com:2,S
> > >> > >
> > >> > >
> > >> > > Full output of 'mfsmaster -xx -a -i -c /etc/mfs/mfsmaster.cfg':
> > >> > >
> > >> > > open files limit has been set to: 16384
> > >> > > working directory: /mnt/mailmfs/mfs
> > >> > > lockfile created and locked
> > >> > > initializing mfsmaster modules ...
> > >> > > exports file has been loaded
> > >> > > topology file has been loaded
> > >> > > write replication limit in old format - change limits to new
> format
> > >> > > read replication limit in old format - change limits to new format
> > >> > > loading metadata ...
> > >> > > found valid metadata file: metadata.mfs.back.1 (version:
> 16411110888
> > >> > > ; id: 59AEE396E5130F20)
> > >> > > found invalid metadata file (wrong header): metadata.crc
> > >> > > found valid metadata file: metadata.mfs.back (version:
> 16412023641 ;
> > >>
> > >> id:
> > >> > > 59AEE396E5130F20)
> > >> > > chosen most recent metadata file: metadata.mfs.back (version:
> > >>
> > >> 16412023641
> > >>
> > >> > > ; id: 59AEE396E5130F20)
> > >> > > loading sessions data ... ok (0.0000)
> > >> > > loading storage classes data ... ok (0.0000)
> > >> > > loading objects (files,directories,etc.) ... ok (1.1999)
> > >> > > loading names ...
> > >> > > loading edge: 2502428->5211944 error: empty name
> > >> > > cleaning metadata ...
> > >> > > cleaning objects ... done
> > >> > > cleaning names ... done
> > >> > > cleaning deletion timestamps ... done
> > >> > > cleaning quota definitions ... done
> > >> > > cleaning chunks data ...done
> > >> > > cleaning xattr data ...done
> > >> > > cleaning posix_acl data ...done
> > >> > > cleaning flock locks data ...done
> > >> > > cleaning posix locks data ...done
> > >> > > cleaning chunkservers data ...done
> > >> > > cleaning open files data ...done
> > >> > > cleaning sessions data ...done
> > >> > > cleaning storage classes data ...done
> > >> > > cleaning dictionary data ...done
> > >> > > metadata have been cleaned
> > >> > > error loading metadata file (metadata.mfs.back): ENOENT (No such
> > >> > > file
> > >>
> > >> or
> > >>
> > >> > > directory)
> > >> > > init: metadata manager failed !!!
> > >> > > error occurred during initialization - exiting
> --
> Ricardo J. Barberis
> Senior SysAdmin / IT Architect
> DonWeb
> La Actitud Es Todo
> www.DonWeb.com
>   _____
>

Re: [MooseFS-Users] mfsmaster won't start - loading edge: 2502428->5211944 error: empty name

Fault tolerant, POSIX-compliant, Net Distributed Storage / File System

Re: [MooseFS-Users] mfsmaster won't start - loading edge: 2502428->5211944 error: empty name