Menu
▾
▴
moosefs-users
|
From: kuer ku <ku...@gm...> - 2010-07-28 12:52:39
|
hi, all I cannot shutdown moosefs master normally. When shutdown, it shows : working directory: /usr/local/moosefs/bin/master sending SIGTERM to lock owner (pid:23172) waiting for termination ... 10s 20s 30s 40s 50s give up Something must be wrong, what should I do ? thanks -- kuer |
|
From: kuer ku <ku...@gm...> - 2010-07-28 12:59:59
|
hi,
I just want to know why master stucked, but I find nothing wrong in
/var/log/messages.
>From /var/log/messages, it seems that master still work, but why it does NOT
exit when got SIGTERM ?
Are there any other way that I can find some useful messages ?
thanks
-- kuer
On Wed, Jul 28, 2010 at 8:52 PM, kuer ku <ku...@gm...> wrote:
> hi, all
>
> I cannot shutdown moosefs master normally. When shutdown, it shows :
>
> working directory: /usr/local/moosefs/bin/master
> sending SIGTERM to lock owner (pid:23172)
> waiting for termination ... 10s 20s 30s 40s 50s give up
>
> Something must be wrong, what should I do ?
>
> thanks
>
> -- kuer
>
|
|
From: kuer ku <ku...@gm...> - 2010-07-28 13:50:33
|
hi, all, I find some strange things in changelog, $ tail -f changelog.0.mfs 143616569: 1280322540|EMPTYRESERVED():0 143616570: 1280322600|FREEINODES():0 143616571: 1280322600|EMPTYRESERVED():0 143616572: 1280322600|EMPTYTRASH():0,0 143616573: 1280322660|FREEINODES():0 143616574: 1280322660|EMPTYRESERVED():0 143616575: 1280322720|FREEINODES():0 143616576: 1280322720|EMPTYRESERVED():0 143616577: 1280322780|FREEINODES():0 143616578: 1280322780|EMPTYRESERVED():0 what does this means, how to fix it ? Jul 28 21:19:00 meta1 mfsmaster[23172]: chunkservers status: Jul 28 21:19:00 meta1 mfsmaster[23172]: server 1 (ip: 221.194.134.189, port: 19322): usedspace: 943864180736 (879.04 GiB), totalspace: 1500164128768 (1397.14 GiB), usage: 62.92% Jul 28 21:19:00 meta1 mfsmaster[23172]: server 2 (ip: 221.194.134.187, port: 19322): usedspace: 957016182784 (891.29 GiB), totalspace: 1500164128768 (1397.14 GiB), usage: 63.79% Jul 28 21:19:00 meta1 mfsmaster[23172]: server 3 (ip: 221.194.134.181, port: 19322): usedspace: 1898559021056 (1768.17 GiB), totalspace: 3000328257536 (2794.27 GiB), usage: 63.28% Jul 28 21:19:00 meta1 mfsmaster[23172]: server 4 (ip: 221.194.134.186, port: 19322): usedspace: 940963352576 (876.34 GiB), totalspace: 1500164128768 (1397.14 GiB), usage: 62.72% Jul 28 21:19:00 meta1 mfsmaster[23172]: server 5 (ip: 221.194.134.184, port: 19322): usedspace: 944276942848 (879.43 GiB), totalspace: 1500164128768 (1397.14 GiB), usage: 62.94% Jul 28 21:19:00 meta1 mfsmaster[23172]: server 6 (ip: 221.194.134.190, port: 19322): usedspace: 1893327695872 (1763.30 GiB), totalspace: 3000328257536 (2794.27 GiB), usage: 63.10% Jul 28 21:19:00 meta1 mfsmaster[23172]: server 7 (ip: 221.194.134.188, port: 19322): usedspace: 957261549568 (891.52 GiB), totalspace: 1500164128768 (1397.14 GiB), usage: 63.81% Jul 28 21:19:00 meta1 mfsmaster[23172]: server 8 (ip: 221.194.134.185, port: 19322): usedspace: 957269495808 (891.53 GiB), totalspace: 1500164128768 (1397.14 GiB), usage: 63.81% Jul 28 21:19:00 meta1 mfsmaster[23172]: server 9 (ip: 221.194.134.183, port: 19322): usedspace: 957314211840 (891.57 GiB), totalspace: 1500164128768 (1397.14 GiB), usage: 63.81% Jul 28 21:19:00 meta1 mfsmaster[23172]: server 10 (ip: 221.194.134.182, port: 19322): usedspace: 956960980992 (891.24 GiB), totalspace: 1500164128768 (1397.14 GiB), usage: 63.79% Jul 28 21:19:00 meta1 mfsmaster[23172]: total: usedspace: 11406813614080 (10623.42 GiB), totalspace: 18001969545216 (16765.64 GiB), usage: 63.36% there are still free space on chunkservers ( almost 40% free space ) what resources runs out ??? how to fix them ? thanks -- kuer On Wed, Jul 28, 2010 at 8:59 PM, kuer ku <ku...@gm...> wrote: > hi, > > I just want to know why master stucked, but I find nothing wrong in > /var/log/messages. > > From /var/log/messages, it seems that master still work, but why it does > NOT exit when got SIGTERM ? > > Are there any other way that I can find some useful messages ? > > thanks > > -- kuer > > > On Wed, Jul 28, 2010 at 8:52 PM, kuer ku <ku...@gm...> wrote: > >> hi, all >> >> I cannot shutdown moosefs master normally. When shutdown, it shows : >> >> working directory: /usr/local/moosefs/bin/master >> sending SIGTERM to lock owner (pid:23172) >> waiting for termination ... 10s 20s 30s 40s 50s give up >> >> Something must be wrong, what should I do ? >> >> thanks >> >> -- kuer >> > > |
|
From: Michał B. <mic...@ge...> - 2010-08-09 12:49:44
|
These messages (143616569: 1280322540|EMPTYRESERVED():0) are normal ones.
Functions run periodically delete free i-nodes, deleted and "reserved"
files. 0 means there was nothing to do.
It may happen that there are still some operations on the clients' side. If
nothing else happens you can always kill the master server process (in this
case it would be "kill -9 23172"), wait till the process ends, run
"mfsmetarestore -a" and run the master again by: "mfsmaster start".
Are there any other messages in syslog close to these operations?
Kind regards
Michał Borychowski
MooseFS Support Manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Gemius S.A.
ul. Wołoska 7, 02-672 Warszawa
Budynek MARS, klatka D
Tel.: +4822 874-41-00
Fax : +4822 874-41-01
From: kuer ku [mailto:ku...@gm...]
Sent: Wednesday, July 28, 2010 3:22 PM
To: moo...@li...
Subject: Re: [Moosefs-users] what should I do IF cannot shutdown master
normally ??
hi, all,
I find some strange things in changelog,
$ tail -f changelog.0.mfs
143616569: 1280322540|EMPTYRESERVED():0
143616570: 1280322600|FREEINODES():0
143616571: 1280322600|EMPTYRESERVED():0
143616572: 1280322600|EMPTYTRASH():0,0
143616573: 1280322660|FREEINODES():0
143616574: 1280322660|EMPTYRESERVED():0
143616575: 1280322720|FREEINODES():0
143616576: 1280322720|EMPTYRESERVED():0
143616577: 1280322780|FREEINODES():0
143616578: 1280322780|EMPTYRESERVED():0
what does this means, how to fix it ?
Jul 28 21:19:00 meta1 mfsmaster[23172]: chunkservers status:
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 1 (ip: 221.194.134.189, port:
19322): usedspace: 943864180736 (879.04 GiB), totalspace: 1500164128768
(1397.14 GiB), usage: 62.92%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 2 (ip: 221.194.134.187, port:
19322): usedspace: 957016182784 (891.29 GiB), totalspace: 1500164128768
(1397.14 GiB), usage: 63.79%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 3 (ip: 221.194.134.181, port:
19322): usedspace: 1898559021056 (1768.17 GiB), totalspace: 3000328257536
(2794.27 GiB), usage: 63.28%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 4 (ip: 221.194.134.186, port:
19322): usedspace: 940963352576 (876.34 GiB), totalspace: 1500164128768
(1397.14 GiB), usage: 62.72%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 5 (ip: 221.194.134.184, port:
19322): usedspace: 944276942848 (879.43 GiB), totalspace: 1500164128768
(1397.14 GiB), usage: 62.94%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 6 (ip: 221.194.134.190, port:
19322): usedspace: 1893327695872 (1763.30 GiB), totalspace: 3000328257536
(2794.27 GiB), usage: 63.10%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 7 (ip: 221.194.134.188, port:
19322): usedspace: 957261549568 (891.52 GiB), totalspace: 1500164128768
(1397.14 GiB), usage: 63.81%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 8 (ip: 221.194.134.185, port:
19322): usedspace: 957269495808 (891.53 GiB), totalspace: 1500164128768
(1397.14 GiB), usage: 63.81%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 9 (ip: 221.194.134.183, port:
19322): usedspace: 957314211840 (891.57 GiB), totalspace: 1500164128768
(1397.14 GiB), usage: 63.81%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 10 (ip: 221.194.134.182,
port: 19322): usedspace: 956960980992 (891.24 GiB), totalspace:
1500164128768 (1397.14 GiB), usage: 63.79%
Jul 28 21:19:00 meta1 mfsmaster[23172]: total: usedspace: 11406813614080
(10623.42 GiB), totalspace: 18001969545216 (16765.64 GiB), usage: 63.36%
there are still free space on chunkservers ( almost 40% free space )
what resources runs out ???
how to fix them ?
thanks
-- kuer
On Wed, Jul 28, 2010 at 8:59 PM, kuer ku <ku...@gm...> wrote:
hi,
I just want to know why master stucked, but I find nothing wrong in
/var/log/messages.
>From /var/log/messages, it seems that master still work, but why it does NOT
exit when got SIGTERM ?
Are there any other way that I can find some useful messages ?
thanks
-- kuer
On Wed, Jul 28, 2010 at 8:52 PM, kuer ku <ku...@gm...> wrote:
hi, all
I cannot shutdown moosefs master normally. When shutdown, it shows :
working directory: /usr/local/moosefs/bin/master
sending SIGTERM to lock owner (pid:23172)
waiting for termination ... 10s 20s 30s 40s 50s give up
Something must be wrong, what should I do ?
thanks
-- kuer
|
|
From: Michał B. <mic...@ge...> - 2010-08-10 08:00:26
|
Hi Kuer!
No, there is no limit for i-nodes in the master server. This message just
means how many i-nodes were deleted by the function FREEINODES which
releases not used i-nodes. In your example the function just deleted 0 not
used i-nodes. BTW - maybe we will resign from logging this information at
all as nothing was released so metadata was also not changed. So these are
really normal messages, nothing to worry about.
Doing "kill -9 mfsmaster" you would not lose anything. There is a tool
mfsmetarestore which restores the metadata. If you feel you need it, you can
make a backup of /usr/local/var/mfs and additionaly issue:
"/usr/local/sbin/mfsmetadump /usr/local/var/mfs/metadata.mfs.back" and check
if all the files are visible in this dump.
Kind regards
Michał
From: kuer ku [mailto:ku...@gm...]
Sent: Tuesday, August 10, 2010 3:38 AM
To: Michał Borychowski
Subject: Re: [Moosefs-users] what should I do IF cannot shutdown master
normally ??
Hi, Michal,
I do NOT want to "kill -9 master", to avoid information losing.
No other sepcial messages in syslog when MFS is hung up.
I noticed another message :
$ tail -f changelog.0.mfs
143616569: 1280322540|EMPTYRESERVED():0
143616570: 1280322600|FREEINODES():0
143616571: 1280322600|EMPTYRESERVED():0
143616572: 1280322600|EMPTYTRASH():0,0
143616573: 1280322660|FREEINODES():0
^^^^^^^^^^^^^^^^^^^^^^
FREEINODES() == 0
I guess there are no more inodes in master, so no new file to create, and so
the MFS hang up.
BUT I do not understand the relationship between MFS diskspace and INODES
capacity.
SO I attach another mfschunkserver to Master, so the system ran again.
QUESTION : is there any document describing the relationship between
diskspace and INODES capacity ??
-- kuer
2010/8/9 Michał Borychowski <mic...@ge...>
These messages (143616569: 1280322540|EMPTYRESERVED():0) are normal ones.
Functions run periodically delete free i-nodes, deleted and "reserved"
files. 0 means there was nothing to do.
It may happen that there are still some operations on the clients' side. If
nothing else happens you can always kill the master server process (in this
case it would be "kill -9 23172"), wait till the process ends, run
"mfsmetarestore -a" and run the master again by: "mfsmaster start".
Are there any other messages in syslog close to these operations?
Kind regards
Michał Borychowski
MooseFS Support Manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Gemius S.A.
ul. Wołoska 7, 02-672 Warszawa
Budynek MARS, klatka D
Tel.: +4822 874-41-00
Fax : +4822 874-41-01
From: kuer ku [mailto:ku...@gm...]
Sent: Wednesday, July 28, 2010 3:22 PM
To: moo...@li...
Subject: Re: [Moosefs-users] what should I do IF cannot shutdown master
normally ??
hi, all,
I find some strange things in changelog,
$ tail -f changelog.0.mfs
143616569: 1280322540|EMPTYRESERVED():0
143616570: 1280322600|FREEINODES():0
143616571: 1280322600|EMPTYRESERVED():0
143616572: 1280322600|EMPTYTRASH():0,0
143616573: 1280322660|FREEINODES():0
143616574: 1280322660|EMPTYRESERVED():0
143616575: 1280322720|FREEINODES():0
143616576: 1280322720|EMPTYRESERVED():0
143616577: 1280322780|FREEINODES():0
143616578: 1280322780|EMPTYRESERVED():0
what does this means, how to fix it ?
Jul 28 21:19:00 meta1 mfsmaster[23172]: chunkservers status:
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 1 (ip: 221.194.134.189, port:
19322): usedspace: 943864180736 (879.04 GiB), totalspace: 1500164128768
(1397.14 GiB), usage: 62.92%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 2 (ip: 221.194.134.187, port:
19322): usedspace: 957016182784 (891.29 GiB), totalspace: 1500164128768
(1397.14 GiB), usage: 63.79%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 3 (ip: 221.194.134.181, port:
19322): usedspace: 1898559021056 (1768.17 GiB), totalspace: 3000328257536
(2794.27 GiB), usage: 63.28%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 4 (ip: 221.194.134.186, port:
19322): usedspace: 940963352576 (876.34 GiB), totalspace: 1500164128768
(1397.14 GiB), usage: 62.72%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 5 (ip: 221.194.134.184, port:
19322): usedspace: 944276942848 (879.43 GiB), totalspace: 1500164128768
(1397.14 GiB), usage: 62.94%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 6 (ip: 221.194.134.190, port:
19322): usedspace: 1893327695872 (1763.30 GiB), totalspace: 3000328257536
(2794.27 GiB), usage: 63.10%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 7 (ip: 221.194.134.188, port:
19322): usedspace: 957261549568 (891.52 GiB), totalspace: 1500164128768
(1397.14 GiB), usage: 63.81%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 8 (ip: 221.194.134.185, port:
19322): usedspace: 957269495808 (891.53 GiB), totalspace: 1500164128768
(1397.14 GiB), usage: 63.81%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 9 (ip: 221.194.134.183, port:
19322): usedspace: 957314211840 (891.57 GiB), totalspace: 1500164128768
(1397.14 GiB), usage: 63.81%
Jul 28 21:19:00 meta1 mfsmaster[23172]: server 10 (ip: 221.194.134.182,
port: 19322): usedspace: 956960980992 (891.24 GiB), totalspace:
1500164128768 (1397.14 GiB), usage: 63.79%
Jul 28 21:19:00 meta1 mfsmaster[23172]: total: usedspace: 11406813614080
(10623.42 GiB), totalspace: 18001969545216 (16765.64 GiB), usage: 63.36%
there are still free space on chunkservers ( almost 40% free space )
what resources runs out ???
how to fix them ?
thanks
-- kuer
On Wed, Jul 28, 2010 at 8:59 PM, kuer ku <ku...@gm...> wrote:
hi,
I just want to know why master stucked, but I find nothing wrong in
/var/log/messages.
>From /var/log/messages, it seems that master still work, but why it does NOT
exit when got SIGTERM ?
Are there any other way that I can find some useful messages ?
thanks
-- kuer
On Wed, Jul 28, 2010 at 8:52 PM, kuer ku <ku...@gm...> wrote:
hi, all
I cannot shutdown moosefs master normally. When shutdown, it shows :
working directory: /usr/local/moosefs/bin/master
sending SIGTERM to lock owner (pid:23172)
waiting for termination ... 10s 20s 30s 40s 50s give up
Something must be wrong, what should I do ?
thanks
-- kuer
|
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X