Re: [MooseFS-Users] mfsmaster won't start - loading edge: 2502428->5211944 error: empty name

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

OK, mfsmaster started, mentioning:

loading edge: 2502428->5211944 error: empty name
loading edge: 2502428->5211944 empty filename replaced by %28empty 5211944%29
ok (4.3644)

Then I started one mfschunkserver (previously moving .chunkdb and .metaid out
of the way) and it scanned fine.

Finally, I was able to mount it locally!  \o/

Now to continue restoring files to the new cluster :)

Thanks again!

PS: in order to compile moosefs on a CentOS 7.8 minimal installation I had to
install these additional packages:

autoconf
automake
libtool

PPS: full output of mfsmaster:

# mfsmaster -a -i -c /etc/mfs/mfsmaster.cfg
open files limit has been set to: 16384
working directory: /mnt/mailmfs/mfs
lockfile created and locked
initializing mfsmaster modules ...
exports file has been loaded
topology file has been loaded
write replication limit in old format - change limits to new format
read replication limit in old format - change limits to new format
loading metadata ...
loading sessions data ... ok (0.0000)
loading storage classes data ... ok (0.0000)
loading objects (files,directories,etc.) ... ok (0.6527)
loading names ... 
loading edge: 2502428->5211944 error: empty name
loading edge: 2502428->5211944 empty filename replaced by %28empty 5211944%29
ok (4.3644)
loading deletion timestamps ... ok (0.0046)
loading quota definitions ... ok (0.0004)
loading xattr data ... ok (0.0000)
loading posix_acl data ... ok (0.0000)
loading open files data ... ok (0.0014)
loading flock_locks data ... ok (0.0000)
loading posix_locks data ... ok (0.0000)
loading chunkservers data ... ok (0.0000)
loading chunks data ... ok (0.2275)
checking filesystem consistency ... ok
connecting files and chunks ... ok
all inodes: 12285650
directory inodes: 511090
file inodes: 11774560
chunks: 11668123
metadata file has been loaded
stats file has been loaded
master <-> metaloggers module: listen on *:9419
master <-> chunkservers module: listen on *:9420
main master server module: listen on *:9421
mfsmaster daemon initialized properly

El Miércoles 30/09/2020 a las 12:12, Piotr Robert Konopelko escribió:
> That's great, thank you for the reply. Please definitely try this patch and
> let us know.
> Looking forward to hearing from you.
>
> Best regards,
> Piotr
>
> *Piotr Robert Konopelko* | m: +48 601 476 440 | e:
> pio...@mo...
> *Business & Technical Support Manager*
> MooseFS Client Support Team
>
> WWW <https://moosefs.com> | GitHub <https://github.com/moosefs/moosefs> |
> Twitter <https://twitter.com/moosefs> | Facebook
> <https://www.facebook.com/moosefs> | LinkedIn
> <https://www.linkedin.com/company/moosefs>
>
>
> On Wed, Sep 30, 2020 at 4:51 PM Ricardo J. Barberis <
>
> ric...@do...> wrote:
> > Cool, we're in the process of restoring from backups to a new mfs cluster
> > but
> > we'll reconfigure the old cluster to put it online and try this patch.
> >
> > It'll hopefully allow us to recover whatever we don't have in the backups
> > (which are fairly recent but since these are mailboxes we're talking
> > about everything helps).
> >
> > I'll let you know how the process goes.
> >
> > Thank you!
> >
> > El Miércoles 30/09/2020 a las 10:16, Piotr Robert Konopelko escribió:
> > > Sorry, I forgot to add – in order to build MooseFS from sources, you
> >
> > need a
> >
> > > few dependencies.
> > > Please install them as described here:
> > > https://github.com/moosefs/moosefs#source-code.
> > >
> > > Piotr
> > >
> > > *Piotr Robert Konopelko* | m: +48 601 476 440 | e:
> > > pio...@mo...
> > > *Business & Technical Support Manager*
> > > MooseFS Client Support Team
> > >
> > > WWW <https://moosefs.com> | GitHub <https://github.com/moosefs/moosefs>
> > >
> > > Twitter <https://twitter.com/moosefs> | Facebook
> > > <https://www.facebook.com/moosefs> | LinkedIn
> > > <https://www.linkedin.com/company/moosefs>
> > >
> > >
> > > On Wed, Sep 30, 2020 at 3:00 PM Piotr Robert Konopelko <
> > >
> > > pio...@mo...> wrote:
> > > > Hello Ricardo,
> > > >
> > > > After your report, we have updated the Master Server code today to
> > > > let the "*-i*" parameter ignore empty file paths' (edges') names and
> > > > substitute them with "*(empty <inode_number>)*" string, without
> > > > quotes and lower / greater than marks of course.
> > > >
> > > > Please have a look at the following commit:
> >
> > https://github.com/moosefs/moosefs/commit/886ea4a703afce1b40e4853ce02101a
> >
> > > >8c43829f3
> > > >
> > > > It will be included in the nearest MooseFS 3.0.115 release.
> > > >
> > > > In order to get this feature before release, please clone the MooseFS
> >
> > Git
> >
> > > > repository from GitHub: https://github.com/moosefs/moosefs, build
> > > > binaries by running *./linux_build.sh* script inside MooseFS Git
> > > > repository directory (this script doesn't run "*make install*", so
> > > > you can just copy (replace) "*mfsmaster*" executable binary file to "
> > > > */usr/sbin/mfsmaster*") and use this newly built "*mfsmaster*" binary
> > > > (don't forget about passing "*-a*" and "*-i*" parameters to it). It
> > > > should be able to load your metadata and substitute empty names with
> >
> > the
> >
> > > > above mentioned string.
> > > >
> > > > Please let me know if it worked for you and do not hesitate to
> > > > contact
> >
> > us
> >
> > > > if you have any questions.
> > > >
> > > > Best regards,
> > > > Piotr
> > > >
> > > > *Piotr Robert Konopelko* | m: +48 601 476 440 | e:
> > > > pio...@mo...
> > > > *Business & Technical Support Manager*
> > > > MooseFS Client Support Team
> > > >
> > > > WWW <https://moosefs.com> | GitHub
> > > > <https://github.com/moosefs/moosefs>
> > > >
> > > > Twitter <https://twitter.com/moosefs> | Facebook
> > > > <https://www.facebook.com/moosefs> | LinkedIn
> > > > <https://www.linkedin.com/company/moosefs>
> > > >
> > > >
> > > > On Wed, Sep 30, 2020 at 2:34 AM Ricardo J. Barberis <
> > > >
> > > > ric...@do...> wrote:
> > > >> El Martes 29/09/2020 a las 20:55, Piotr Robert Konopelko escribió:
> > > >> > Hello Ricardo,
> > > >> >
> > > >> > 1) Have you done a backup of /var/lib/mfs after the failure? If
> > > >> > not,
> > > >>
> > > >> please
> > > >>
> > > >> > do so, just in case.
> > > >>
> > > >> Yes, I have a backup post-crash, before atempting to recover
> > > >>
> > > >> > 2) Have you tried the following sequence:
> > > >> >
> > > >> >    - Stopping the Metalogger,
> > > >> >    - Copying all the metadata_ml* files and changelog_ml* files
> > > >> > from Metalogger's /var/lib/mfs directory to Master Server's
> >
> > /var/lib/mfs
> >
> > > >> >    directory,
> > > >> >    - Starting the Master server with "-a" parameter that makes it
> > > >>
> > > >> trying to
> > > >>
> > > >> >    recover the metadata:
> > > >> >
> > > >> >    *mfsmaster -a*
> > > >> >
> > > >> > ?
> > > >>
> > > >> Yes, I tried this sequence in the master server and also in the
> > > >> metalogger,
> > > >> copying the files to a new directory (so I also got a backup of the
> > > >> files from the metalogger).
> > > >>
> > > >> > The above sequence is usually the best way to recover the
> > > >> > metadata.
> > > >>
> > > >> I also tried 'mfsmaster -a -i' to no avail.
> > > >>
> > > >>
> > > >> BTW, all of this is in my original email, sorry if I wasn't very
> >
> > clear.
> >
> > > >> > Best regards,
> > > >> > Piotr
> > > >> >
> > > >> > *Piotr Robert Konopelko* | m: +48 601 476 440 | e:
> > > >> > pio...@mo...
> > > >> > *Business & Technical Support Manager*
> > > >> > MooseFS Client Support Team
> > > >> >
> > > >> > WWW <https://moosefs.com> | GitHub
> > > >> > <https://github.com/moosefs/moosefs>
> > > >> >
> > > >> > Twitter <https://twitter.com/moosefs> | Facebook
> > > >> > <https://www.facebook.com/moosefs> | LinkedIn
> > > >> > <https://www.linkedin.com/company/moosefs>
> > > >> >
> > > >> >
> > > >> > On Wed, Sep 30, 2020 at 1:46 AM Ricardo J. Barberis <
> > > >> >
> > > >> > ric...@do...> wrote:
> > > >> > > Hi all,
> > > >> > >
> > > >> > > My mfsmaster crashed crashed today and when trying to start it I
> >
> > get
> >
> > > >> this
> > > >>
> > > >> > > error:
> > > >> > >
> > > >> > > loading edge: 2502428->5211944 error: empty name
> > > >> > >
> > > >> > >
> > > >> > > I tried with metadata.mfs.back, metadata.mfs.back.1 from master,
> >
> > and
> >
> > > >> also
> > > >>
> > > >> > > metadata_ml.mfs.back and metadata_ml.mfs.back.1 from metalogger,
> >
> > all
> >
> > > >> of
> > > >>
> > > >> > > them
> > > >> > > fail with the same error.
> > > >> > >
> > > >> > >
> > > >> > > My mfsmaster was 3.0.100 when it crashed, I upgraded to 3.0.114
> >
> > but
> >
> > > >> it's
> > > >>
> > > >> > > the same:
> > > >> > >
> > > >> > > # mfsmaster -v
> > > >> > > version: 3.0.114-1 ; build: 1257
> > > >> > >
> > > >> > >
> > > >> > > Any hints to solve this much appreciated, as this is a
> > > >> > > production cluster.
> > > >> > >
> > > >> > >
> > > >> > > I have dumped all the metadata files and all of them have this:
> > > >> > >
> > > >> > > # mfsmetadump metadata.mfs.back > metadata.mfs.dump
> > > >> > > # grep 2502428 metadata.mfs.dump | grep -1 5211944
> > > >> > > EDGE|p:   2502428|c:   5154814|i:0x7FFFFFFFF92D4A30|n:
> > > >> > > 1519914956.H419257P2952.c106-dr.dattaweb.com:2,S
> > > >> > > EDGE|p:   2502428|c:   5211944|i:0x000000095F864814|n:
> > > >> > > EDGE|p:   2502428|c:   2920292|i:0x7FFFFFFFF947F4F2|n:
> > > >> > > 1519809696.H296554P8952.c105-dr.dattaweb.com:2,S
> > > >> > >
> > > >> > >
> > > >> > > Full output of 'mfsmaster -xx -a -i -c /etc/mfs/mfsmaster.cfg':
> > > >> > >
> > > >> > > open files limit has been set to: 16384
> > > >> > > working directory: /mnt/mailmfs/mfs
> > > >> > > lockfile created and locked
> > > >> > > initializing mfsmaster modules ...
> > > >> > > exports file has been loaded
> > > >> > > topology file has been loaded
> > > >> > > write replication limit in old format - change limits to new
> >
> > format
> >
> > > >> > > read replication limit in old format - change limits to new
> > > >> > > format loading metadata ...
> > > >> > > found valid metadata file: metadata.mfs.back.1 (version:
> >
> > 16411110888
> >
> > > >> > > ; id: 59AEE396E5130F20)
> > > >> > > found invalid metadata file (wrong header): metadata.crc
> > > >> > > found valid metadata file: metadata.mfs.back (version:
> >
> > 16412023641 ;
> >
> > > >> id:
> > > >> > > 59AEE396E5130F20)
> > > >> > > chosen most recent metadata file: metadata.mfs.back (version:
> > > >>
> > > >> 16412023641
> > > >>
> > > >> > > ; id: 59AEE396E5130F20)
> > > >> > > loading sessions data ... ok (0.0000)
> > > >> > > loading storage classes data ... ok (0.0000)
> > > >> > > loading objects (files,directories,etc.) ... ok (1.1999)
> > > >> > > loading names ...
> > > >> > > loading edge: 2502428->5211944 error: empty name
> > > >> > > cleaning metadata ...
> > > >> > > cleaning objects ... done
> > > >> > > cleaning names ... done
> > > >> > > cleaning deletion timestamps ... done
> > > >> > > cleaning quota definitions ... done
> > > >> > > cleaning chunks data ...done
> > > >> > > cleaning xattr data ...done
> > > >> > > cleaning posix_acl data ...done
> > > >> > > cleaning flock locks data ...done
> > > >> > > cleaning posix locks data ...done
> > > >> > > cleaning chunkservers data ...done
> > > >> > > cleaning open files data ...done
> > > >> > > cleaning sessions data ...done
> > > >> > > cleaning storage classes data ...done
> > > >> > > cleaning dictionary data ...done
> > > >> > > metadata have been cleaned
> > > >> > > error loading metadata file (metadata.mfs.back): ENOENT (No such
> > > >> > > file
> > > >>
> > > >> or
> > > >>
> > > >> > > directory)
> > > >> > > init: metadata manager failed !!!
> > > >> > > error occurred during initialization - exiting
-- 
Ricardo J. Barberis
Senior SysAdmin / IT Architect
DonWeb
La Actitud Es Todo
www.DonWeb.com

Re: [MooseFS-Users] mfsmaster won't start - loading edge: 2502428->5211944 error: empty name

Fault tolerant, POSIX-compliant, Net Distributed Storage / File System

Re: [MooseFS-Users] mfsmaster won't start - loading edge: 2502428->5211944 error: empty name