moosefs-users — Mailing list for MooseFS Users

[Moosefs-users] Compile on Nexenta Fails

[Robert D. <ro...@in...>]

Hello,

 

When compiling moosefs on NexentaStor 3.0.4(Community), I use the following
configure options:

 

./configure --disable-mfsmaster --disable-mfscgi --disable-mfscgiserv
--disable-mfsmount

 

After configuring, make returns the following output:

 

Making all in mfschunkserver

make[2]: Entering directory `/root/mfs-1.6.20-2/mfschunkserver'

gcc -DHAVE_CONFIG_H -I. -I..  -I../mfscommon -DMFSMAXFILES=10000
-D_USE_PTHREADS  -DAPPNAME=mfschunkserver -D__EXTENSIONS__ -D_REENTRANT
-pthreads -std=c99 -g -O2 -W -Wall -Wshadow -pedantic -MT
mfschunkserver-bgjobs.o -MD -MP -MF .deps/mfschunkserver-bgjobs.Tpo -c -o
mfschunkserver-bgjobs.o `test -f 'bgjobs.c' || echo './'`bgjobs.c

mv -f .deps/mfschunkserver-bgjobs.Tpo .deps/mfschunkserver-bgjobs.Po

gcc -DHAVE_CONFIG_H -I. -I..  -I../mfscommon -DMFSMAXFILES=10000
-D_USE_PTHREADS  -DAPPNAME=mfschunkserver -D__EXTENSIONS__ -D_REENTRANT
-pthreads -std=c99 -g -O2 -W -Wall -Wshadow -pedantic -MT
mfschunkserver-csserv.o -MD -MP -MF .deps/mfschunkserver-csserv.Tpo -c -o
mfschunkserver-csserv.o `test -f 'csserv.c' || echo './'`csserv.c

mv -f .deps/mfschunkserver-csserv.Tpo .deps/mfschunkserver-csserv.Po

gcc -DHAVE_CONFIG_H -I. -I..  -I../mfscommon -DMFSMAXFILES=10000
-D_USE_PTHREADS  -DAPPNAME=mfschunkserver -D__EXTENSIONS__ -D_REENTRANT
-pthreads -std=c99 -g -O2 -W -Wall -Wshadow -pedantic -MT
mfschunkserver-hddspacemgr.o -MD -MP -MF
.deps/mfschunkserver-hddspacemgr.Tpo -c -o mfschunkserver-hddspacemgr.o
`test -f 'hddspacemgr.c' || echo './'`hddspacemgr.c

hddspacemgr.c: In function 'hdd_chunk_remove':

hddspacemgr.c:667: warning: pointer targets in passing argument 1 of
'munmap' differ in signedness

hddspacemgr.c:675: warning: pointer targets in passing argument 1 of
'munmap' differ in signedness

hddspacemgr.c: In function 'hdd_chunk_get':

hddspacemgr.c:802: warning: pointer targets in passing argument 1 of
'munmap' differ in signedness

hddspacemgr.c:810: warning: pointer targets in passing argument 1 of
'munmap' differ in signedness

hddspacemgr.c: In function 'hdd_check_folders':

hddspacemgr.c:1070: warning: pointer targets in passing argument 1 of
'munmap' differ in signedness

hddspacemgr.c:1078: warning: pointer targets in passing argument 1 of
'munmap' differ in signedness

hddspacemgr.c: In function 'chunk_emptycrc':

hddspacemgr.c:1290: warning: pointer targets in assignment differ in
signedness

hddspacemgr.c: In function 'chunk_readcrc':

hddspacemgr.c:1328: warning: pointer targets in assignment differ in
signedness

hddspacemgr.c:1343: warning: pointer targets in passing argument 1 of
'munmap' differ in signedness

hddspacemgr.c: In function 'chunk_freecrc':

hddspacemgr.c:1355: warning: pointer targets in passing argument 1 of
'munmap' differ in signedness

hddspacemgr.c: In function 'hdd_delayed_ops':

hddspacemgr.c:1505: warning: pointer targets in passing argument 1 of
'munmap' differ in signedness

hddspacemgr.c: In function 'hdd_io_begin':

hddspacemgr.c:1608: warning: pointer targets in assignment differ in
signedness

hddspacemgr.c: In function 'hdd_term':

hddspacemgr.c:3669: warning: pointer targets in passing argument 1 of
'munmap' differ in signedness

hddspacemgr.c:3677: warning: pointer targets in passing argument 1 of
'munmap' differ in signedness

mv -f .deps/mfschunkserver-hddspacemgr.Tpo
.deps/mfschunkserver-hddspacemgr.Po

gcc -DHAVE_CONFIG_H -I. -I..  -I../mfscommon -DMFSMAXFILES=10000
-D_USE_PTHREADS  -DAPPNAME=mfschunkserver -D__EXTENSIONS__ -D_REENTRANT
-pthreads -std=c99 -g -O2 -W -Wall -Wshadow -pedantic -MT
mfschunkserver-masterconn.o -MD -MP -MF .deps/mfschunkserver-masterconn.Tpo
-c -o mfschunkserver-masterconn.o `test -f 'masterconn.c' || echo
'./'`masterconn.c

mv -f .deps/mfschunkserver-masterconn.Tpo .deps/mfschunkserver-masterconn.Po

gcc -DHAVE_CONFIG_H -I. -I..  -I../mfscommon -DMFSMAXFILES=10000
-D_USE_PTHREADS  -DAPPNAME=mfschunkserver -D__EXTENSIONS__ -D_REENTRANT
-pthreads -std=c99 -g -O2 -W -Wall -Wshadow -pedantic -MT
mfschunkserver-replicator.o -MD -MP -MF .deps/mfschunkserver-replicator.Tpo
-c -o mfschunkserver-replicator.o `test -f 'replicator.c' || echo
'./'`replicator.c

mv -f .deps/mfschunkserver-replicator.Tpo .deps/mfschunkserver-replicator.Po

gcc -DHAVE_CONFIG_H -I. -I..  -I../mfscommon -DMFSMAXFILES=10000
-D_USE_PTHREADS  -DAPPNAME=mfschunkserver -D__EXTENSIONS__ -D_REENTRANT
-pthreads -std=c99 -g -O2 -W -Wall -Wshadow -pedantic -MT
mfschunkserver-chartsdata.o -MD -MP -MF .deps/mfschunkserver-chartsdata.Tpo
-c -o mfschunkserver-chartsdata.o `test -f 'chartsdata.c' || echo
'./'`chartsdata.c

mv -f .deps/mfschunkserver-chartsdata.Tpo .deps/mfschunkserver-chartsdata.Po

gcc -DHAVE_CONFIG_H -I. -I..  -I../mfscommon -DMFSMAXFILES=10000
-D_USE_PTHREADS  -DAPPNAME=mfschunkserver -D__EXTENSIONS__ -D_REENTRANT
-pthreads -std=c99 -g -O2 -W -Wall -Wshadow -pedantic -MT
mfschunkserver-main.o -MD -MP -MF .deps/mfschunkserver-main.Tpo -c -o
mfschunkserver-main.o `test -f '../mfscommon/main.c' || echo
'./'`../mfscommon/main.c

../mfscommon/main.c: In function 'changeugid':

../mfscommon/main.c:548: error: too many arguments to function 'getgrnam_r'

../mfscommon/main.c:561: error: too many arguments to function 'getpwuid_r'

../mfscommon/main.c:569: error: too many arguments to function 'getpwnam_r'

make[2]: *** [mfschunkserver-main.o] Error 1

make[2]: Leaving directory `/root/mfs-1.6.20-2/mfschunkserver'

make[1]: *** [all-recursive] Error 1

make[1]: Leaving directory `/root/mfs-1.6.20-2'

make: *** [all] Error 2

 

 

Ideas/Suggestions/Work-Around? 

[Moosefs-users] speed up balancing

[Thomas S H. <tha...@gm...>]

I am currently migrating my moosefs environment into a new setup, new OS,
new partitioning, changing out disks for larger disks etc.

So I need to be able to take a chunkserver down, delete the chunks on it,
and then have it rebalance as quickly as possible back into the overall
moosefs cluster, what is the fastest way to do this? Should I make the
chunks for removal before taking them offline? Will that speed up the
balancing?

-Thomas S Hatch

Re: [Moosefs-users] chunk distribution & striping

[Ethan T. <et...@et...>]

Thanks for the replies.  I get the philosophy of the striping, it is just different than what I expected.

It would be neat if Moose had pluggable chunk distribution.  The built in one has priorities to level the usage of the disks in the system which results in a lot of striping.  I for one would prefer that chunks be allocated on the same disk as other chunks for the file, and rebalanced to 'defragment' files onto fewer disks.  There is also the idea of local chunkservers, you could implement a chunk distributor that tended to place new chunks on a server local to the client.  An interface could be developed so that implementations could even be scripted.  WIld idea, I know, but just a thought.

Ethan

On Mar 4, 2011, at 2:29 AM, Steve wrote:

> 
> Hi,
> 
> 
> 
> The idea is that when you loose the first drive moosefs will re-address the
> balance by making copies. Unless you loose 3 chunkservers at the same time
> (highly unlikely) youd be fine. More chunkservers than the highest goal are
> required for security. More the better. 
> 
> 
> 
> Steve 
> 
> 
> 
> 
> 
> -------Original Message------- 
> 
> 
> 
> From: Ethan Tuttle 
> 
> Date: 04/03/2011 09:38:46 
> 
> To: moo...@li... 
> 
> Subject: [Moosefs-users] chunk distribution & striping 
> 
> 
> 
> I have a question about Moose FS' chunk distribution algorithm. Due to the
> striping, an entire set of files with a given goal can be lost rather than
> just the files on failed devices. In other words, with 10 disks and all
> files goal=3, you likely lose the the entire filesystem if three disks go
> down. 
> 
> 
> 
> Will MooseFS be improved in this area? Perhaps the chunk distribution could
> tend to put files on the same disks? I realize this isn't a trivial problem
> but I'm curious if it is on the roadmap. 
> 
> 
> 
> Regards, 
> 
> 
> 
> Ethan 
> 
> -----------------------------------------------------------------------------
> 
> 
> What You Don't Know About Data Connectivity CAN Hurt You 
> 
> This paper provides an overview of data connectivity, details 
> 
> its effect on application quality, and explores various alternative 
> 
> solutions. http://p.sf.net/sfu/progress-d2d 
> 
> _______________________________________________ 
> 
> moosefs-users mailing list 
> 
> moo...@li... 
> 
> https://lists.sourceforge.net/lists/listinfo/moosefs-users 
> 
> 

Re: [Moosefs-users] Core Dumped from mfsmount with Autofs

[Flow J. <fl...@gm...>]

I tried to re-compile mfsmount with the "free(freecblockshead)" line 
commented out. Now our servers (which keep running 7x24) are happy, no 
more core files. However, core files still gets generated on our 
workstations when they reboot. The core is generated from the 
"read_data_term" line right after the "write_data_term" line mentioned 
previously.

Hopefully this will also get fixed in next release, and will even be 
better if I can have a quick solution / patch for the issue.

Thanks
Flow

On 03/01/2011 11:37 PM, Flow Jiang wrote:
> Michal,
>
> Glad to know that this error could be simply solved by commenting out 
> that line and will try tomorrow to see if it fixes this issue.
>
> It does annoying since each core file takes about 170M and I tried to 
> disable the core dump but failed. So hopefully we can have a better 
> solution in the next release.
>
> Thanks
> Flow
>
> On 03/01/2011 09:00 PM, Michal Borychowski wrote:
>> Hi!
>>
>> This error is not a serious one. It may happen only upon exits. If these
>> errors are annoying a quick solution is to comment out the
>> "free(freecblockshead)" line, recompile mfsmount and run again. We'll
>> prepare a better solution in the next release.
>>
>>
>> Kind regards
>> Michał

Re: [Moosefs-users] chunk distribution & striping

[Thomas S H. <tha...@gm...>]

Also, it is better this way, I have lost 3 servers at once (on purpose to
see what the behavior is) and the nice thing about the distribution of
chunks is that you end up with very few goal 0 files, and the file system as
a whole is still available. And I am know that I only need to get any one of
the servers back online to restore a state of complete availability.

Re: [Moosefs-users] mfs* pid files?

[Giovanni T. <gt...@li...>]

Hi,

thanks to everyone, I've modified Debian defaults scripts to make use of
pidof for saving in /var/run/mfs/*.pid the pidfiles of mfs processes.

I'm attaching the diff, maybe could be included in default mfs scripts.

Bye.

-- 
Giovanni Toraldo
http://www.libersoft.it/

Re: [Moosefs-users] chunk distribution & striping

[Steve <st...@bo...>]

Hi,



The idea is that when you loose the first drive moosefs will re-address the
balance by making copies. Unless you loose 3 chunkservers at the same time
(highly unlikely) youd be fine. More chunkservers than the highest goal are
required for security. More the better. 

 

Steve 

 

 

-------Original Message------- 

 

From: Ethan Tuttle 

Date: 04/03/2011 09:38:46 

To: moo...@li... 

Subject: [Moosefs-users] chunk distribution & striping 

 

I have a question about Moose FS' chunk distribution algorithm. Due to the
striping, an entire set of files with a given goal can be lost rather than
just the files on failed devices. In other words, with 10 disks and all
files goal=3, you likely lose the the entire filesystem if three disks go
down. 

 

Will MooseFS be improved in this area? Perhaps the chunk distribution could
tend to put files on the same disks? I realize this isn't a trivial problem
but I'm curious if it is on the roadmap. 

 

Regards, 

 

Ethan 

-----------------------------------------------------------------------------
 

What You Don't Know About Data Connectivity CAN Hurt You 

This paper provides an overview of data connectivity, details 

its effect on application quality, and explores various alternative 

solutions. http://p.sf.net/sfu/progress-d2d 

_______________________________________________ 

moosefs-users mailing list 

moo...@li... 

https://lists.sourceforge.net/lists/listinfo/moosefs-users 

 

[Moosefs-users] chunk distribution & striping

[Ethan T. <et...@et...>]

I have a question about Moose FS' chunk distribution algorithm.  Due to the striping, an entire set of files with a given goal can be lost rather than just the files on failed devices.  In other words, with 10 disks and all files goal=3, you likely lose the the entire filesystem if three disks go down.

Will MooseFS be improved in this area?  Perhaps the chunk distribution could tend to put files on the same disks?  I realize this isn't a trivial problem but I'm curious if it is on the roadmap.

Regards,

Ethan

Re: [Moosefs-users] mfs* pid files?

[Ricardo J. B. <ric...@da...>]

El Jue 03 Marzo 2011, Thomas S Hatch escribió:
> Thats some fine lsof skills you have there,

Not really, I'm a quick reader and skimmed through 'man lsof' (check the 
second version in another mail!!).

> although I am going to argue that pidof is simpler, and it should be less
> intrusive, since lsof needs to read in the table of open file descriptors:
>
> pidof -o %PPID -x /usr/sbin/mfsmaster

Well, acording to time your pidof version is consistenly three times faster 
than my last lsof version, so this one wins :)

In my defense, I said it was a dirty way!! :P

Cheers.

> On Thu, Mar 3, 2011 at 3:22 PM, Ricardo J. Barberis <
>
> ric...@da...> wrote:
> > El Jue 03 Marzo 2011, Giovanni Toraldo escribió:
> > > Il 03/03/2011 11:20, Michal Borychowski ha scritto:
> > > > Hi!
> > > >
> > > > Please have a look at these two answers:
> > > > http://www.moosefs.org/moosefs-faq.html#master-pid
> > >
> > > Uh, there isn't something more scripting friendly? :P
> >
> > Quick and dirty if you're on Linux (watch out for line-wrapping):
> >
> > lsof -X -n | grep $(awk '/^DATA_PATH/ {print $NF}'
> > /etc/mfs/mfsmaster.cfg)/.mfsmaster.lock | awk '{print $2}'
> >
> > > Every daemon that forks in background should write a PID somewhere, why
> > > mfs did not, even inside the lock file?

-- 
Ricardo J. Barberis
Senior SysAdmin / ITI
Dattatec.com :: Soluciones de Web Hosting
Tu Hosting hecho Simple!

Re: [Moosefs-users] mfs* pid files?

[Ricardo J. B. <ric...@da...>]

El Jue 03 Marzo 2011, Ricardo J. Barberis escribió:
> El Jue 03 Marzo 2011, Giovanni Toraldo escribió:
> > Il 03/03/2011 11:20, Michal Borychowski ha scritto:
> > > Hi!
> > >
> > > Please have a look at these two answers:
> > > http://www.moosefs.org/moosefs-faq.html#master-pid
> >
> > Uh, there isn't something more scripting friendly? :P
>
> Quick and dirty if you're on Linux (watch out for line-wrapping):
>
> lsof -X -n | grep $(awk '/^DATA_PATH/ {print $NF}' /etc/mfs/mfsmaster.cfg)/.mfsmaster.lock | awk '{print $2}'

OK, now the not-so-dirty way ;)

lsof -X -n -t $(awk '/^DATA_PATH/ {print $NF"/.mfsmaster.lock"}' /etc/mfs/mfsmaster.cfg)

> > Every daemon that forks in background should write a PID somewhere, why
> > mfs did not, even inside the lock file?
> >
> > Thanks.
>
> Cheers,

Cheers again :)
-- 
Ricardo J. Barberis
Senior SysAdmin / ITI
Dattatec.com :: Soluciones de Web Hosting
Tu Hosting hecho Simple!

Re: [Moosefs-users] mfs* pid files?

[Ricardo J. B. <ric...@da...>]

El Jue 03 Marzo 2011, Giovanni Toraldo escribió:
> Il 03/03/2011 11:20, Michal Borychowski ha scritto:
> > Hi!
> >
> > Please have a look at these two answers:
> > http://www.moosefs.org/moosefs-faq.html#master-pid
>
> Uh, there isn't something more scripting friendly? :P

Quick and dirty if you're on Linux (watch out for line-wrapping):

lsof -X -n | grep $(awk '/^DATA_PATH/ {print $NF}' /etc/mfs/mfsmaster.cfg)/.mfsmaster.lock | awk '{print $2}'


> Every daemon that forks in background should write a PID somewhere, why
> mfs did not, even inside the lock file?
>
> Thanks.

Cheers,
-- 
Ricardo J. Barberis
Senior SysAdmin / ITI
Dattatec.com :: Soluciones de Web Hosting
Tu Hosting hecho Simple!

------------------------------------------

Nota de confidencialidad: Este mensaje y los archivos adjuntos al mismo
son confidenciales, de uso exclusivo para el destinatario del mismo. La
divulgación y/o uso del mismo sin autorización por parte de Dattatec.com
queda prohibida. Dattatec.com no se hace responsable del mensaje por la
falsificación y/o alteración del mismo.
De no ser Ud. el destinatario del mismo y lo ha recibido por error, por
favor notifique al remitente y elimínelo de su sistema.

Confidentiality Note: This message and any attachments (the message) are
confidential and intended solely for the addressees. Any unauthorised use
or dissemination is prohibited by Dattatec.com. Dattatec.com shall not be
liable for the message if altered or falsified.
If you are not the intended addressee of this message, please cancel it
immediately and inform the sender.

Nota de Confidencialidade: Esta mensagem e seus eventuais anexos podem
conter dados confidenciais ou privilegiados. Se você os recebeu por engano
ou não é um dos destinatários aos quais ela foi endereçada, por favor
destrua-a e a todos os seus eventuais anexos ou copias realizadas,
imediatamente.
É proibida a retenção, distribuição, divulgação ou utilização de quaisquer
informações aqui contidas. Por favor, informe-nos sobre o recebimento
indevido desta mensagem, retornando-a para o autor.

Re: [Moosefs-users] Chunks

[Ricardo J. B. <ric...@da...>]

El Jue 03 Marzo 2011, Steve escribió:
> In your first scenario why wouldn't you just mirror the drives ?

With only one chunkserver? Yes.

> -------Original Message-------
>
> From: randall
> Date: 03/03/2011 10:14:00
> To: moo...@li...
> Subject: Re: [Moosefs-users] Chunks
>
> On 03/02/2011 11:40 AM, Michal Borychowski wrote:
> > The main purpose of MooseFS system is security not the space savings. And
> > solution with RAID6 is not that secure. We generally advise not to use
> > any RAIDs and using at least goal=2.
>
> Wondering about this.
>
> when no raid and goal=2, this would mean when using multiple disks per
> server that each disk would be a seperate chunk location.
>
> Can see the use of this when you use 1 single server as each copy would
> reside on 2 seperate disks so you are somewhat protected against disk
> failure.
>
> but when you have 2 servers with each 12 disks (24 chunk locations),
> does each chunk reside on 2 seperate servers giving protection against
> server failure?
>
> did read somewhere there is work done on "location awareness" spreading
> each chunks over racks
>
> Randall

-- 
Ricardo J. Barberis
Senior SysAdmin / ITI
Dattatec.com :: Soluciones de Web Hosting
Tu Hosting hecho Simple!

Re: [Moosefs-users] Chunks

[Ricardo J. B. <ric...@da...>]

El Jue 03 Marzo 2011, randall escribió:
> On 03/02/2011 11:40 AM, Michal Borychowski wrote:
> > The main purpose of MooseFS system is security not the space savings. And
> > solution with RAID6 is not that secure. We generally advise not to use
> > any RAIDs and using at least goal=2.
>
> Wondering about this.
>
> when no raid and goal=2, this would mean when using multiple disks per
> server that each disk would be a seperate chunk location.
>
> Can see the use of this when you use 1 single server as each copy would
> reside on 2 seperate disks so you are somewhat protected against disk
> failure.

I don't think so, goal=2 means one chunk in each chunkserver.

If you have only one chunckserver your files would be undergoal, though I'm 
not sure if one chunkserver alone is treated diferently.

> but when you have 2 servers with each 12 disks (24 chunk locations),
> does each chunk reside on 2 seperate servers giving protection against
> server failure?

Exactly.

You can see where each chunk is located with mfsfileinfo /path/to/file.

Regards,
-- 
Ricardo J. Barberis
Senior SysAdmin / ITI
Dattatec.com :: Soluciones de Web Hosting
Tu Hosting hecho Simple!

Re: [Moosefs-users] mfs* pid files?

[jose m. <let...@us...>]

El jue, 03-03-2011 a las 11:58 +0100, Giovanni Toraldo escribió:
> Il 03/03/2011 11:20, Michal Borychowski ha scritto:
> > Hi!
> > 
> > Please have a look at these two answers:
> > http://www.moosefs.org/moosefs-faq.html#master-pid 
> 
> Uh, there isn't something more scripting friendly? :P
> 
> Every daemon that forks in background should write a PID somewhere, why
> mfs did not, even inside the lock file?
> 
> Thanks.
> 

* Use webmin, alive servers module is friendly for centralized alerting
system, and syslog errors, write error, read error, over mail, mobil,
with google calendar, other provider ......

* Form the mailing of the alert after at least 2+ cross-checks of the
error, in order to avoid to alert circumstantial errors.

Re: [Moosefs-users] mfs* pid files?

[Thomas S H. <tha...@gm...>]

Here is my Arch Linux init script, I am just using the pidof application to
detect the pid,

http://projects.archlinux.org/svntogit/community.git/tree/mfs/trunk/mfsmaster

I think that using ps or pidof is a far more reliable way to detect if a
process is running, since often times a crashed daemon can leave behind the
pid file

I hope this helps!

On Thu, Mar 3, 2011 at 3:58 AM, Giovanni Toraldo <gt...@li...> wrote:

> Il 03/03/2011 11:20, Michal Borychowski ha scritto:
> > Hi!
> >
> > Please have a look at these two answers:
> > http://www.moosefs.org/moosefs-faq.html#master-pid
>
> Uh, there isn't something more scripting friendly? :P
>
> Every daemon that forks in background should write a PID somewhere, why
> mfs did not, even inside the lock file?
>
> Thanks.
>
> --
> Giovanni Toraldo
> http://www.libersoft.it/
>
>
> ------------------------------------------------------------------------------
> Free Software Download: Index, Search & Analyze Logs and other IT data in
> Real-Time with Splunk. Collect, index and harness all the fast moving IT
> data
> generated by your applications, servers and devices whether physical,
> virtual
> or in the cloud. Deliver compliance at lower cost and gain new business
> insights. http://p.sf.net/sfu/splunk-dev2dev
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>

Re: [Moosefs-users] Uneven distribution of files over physical storage

[Thomas S H. <tha...@gm...>]

That would leave me to suspect that your servers would level out quite well,
I am curious to see how this progresses, were all of your servers turned on
at the same time? or were the chunkservers added one at a time?

On Thu, Mar 3, 2011 at 1:29 AM, Stas Oskin <sta...@gm...> wrote:

> Goal of 2.
>
>
> On Wed, Mar 2, 2011 at 11:15 PM, Thomas S Hatch <tha...@gm...>wrote:
>
>> What is the goal you are using on your files?
>>
>> On Wed, Mar 2, 2011 at 1:31 PM, Stas Oskin <sta...@gm...> wrote:
>>
>>> The usage of the first disk is growing and probably going to completely
>>> fill-up soon, should it cause any problems?
>>>
>>> The used version is 1.6.17 by the way,
>>>
>>> Thanks.
>>>
>>>
>>> On Wed, Mar 2, 2011 at 10:30 PM, Stas Oskin <sta...@gm...>wrote:
>>>
>>>> Hi.
>>>>
>>>> I have some strange problem, where the files are distributed un-evenly
>>>> over physical storage.
>>>>
>>>> We have 5 disks per test server, with each of server have it's disks
>>>> filled out in following fashion:
>>>> 85%
>>>> 45%
>>>> 45%
>>>> 45%
>>>> 45%
>>>>
>>>> The 1st disk is relatively small (100 GB), while the rest are large (1.8
>>>> TB).
>>>>
>>>> Any idea if this situation is normal and the first disk supposed to fill
>>>> up, or there should be some kind of distribution amongst the disks?
>>>>
>>>> Thanks.
>>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Free Software Download: Index, Search & Analyze Logs and other IT data in
>>> Real-Time with Splunk. Collect, index and harness all the fast moving IT
>>> data
>>> generated by your applications, servers and devices whether physical,
>>> virtual
>>> or in the cloud. Deliver compliance at lower cost and gain new business
>>> insights. http://p.sf.net/sfu/splunk-dev2dev
>>> _______________________________________________
>>> moosefs-users mailing list
>>> moo...@li...
>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>>
>>>
>>
>

Re: [Moosefs-users] fs_writechunk returns status 11

[Steve W. <st...@pu...>]

Hi Michal,

Thanks for the help!  We found that the problematic file was a .history 
file used by tcsh.  The user was running a large number of scripts which 
spawned shells to run jobs with each one attempting to read/write the 
same .history file.  He now modified his startup script to only access 
the .history file for interactive shells and this has eliminated almost 
all of these messages in the logs.

Steve

On 03/02/2011 06:07 AM, Michal Borychowski wrote:
> Hi Steve!
>
> Regarding your first question this means "CHUNK_LOCKED" - it may happen when
> several clients try to write to the same file. The message doesn't mean
> anything bad but it is better to avoid such situations.
>
> Regarding your second question please have a look here:
> http://www.moosefs.org/moosefs-faq.html#error-messages and the two following
> answers.
>
>
> Best regards
> Michal
>
>
>
>
> -----Original Message-----
> From: Steve Wilson [mailto:st...@pu...]
> Sent: Tuesday, March 01, 2011 3:07 PM
> To: moo...@li...
> Subject: Re: [Moosefs-users] fs_writechunk returns status 11
>
> On 03/01/2011 08:42 AM, Steve Wilson wrote:
>> Hi,
>>
>> For a couple peak periods of time this morning, my logs were full of
>> messages like the following:
>>      Mar  1 07:49:49 stanley mfsmount[1253]: file: 278364, index: 0 -
>> fs_writechunk returns status 11
>>      Mar  1 07:49:49 stanley mfsmount[1253]: file: 278364, index: 1 -
>> fs_writechunk returns status 11
>>
>> They all reference the same file: 278364.  Does anyone know what this
>> means?  And, if so, what I should do about it?
>>
>> Thanks!
>>
>> Steve
>>
> A further look showed up a handful of other files but 99% of the
> messages refer to that one file.
>
> Additionally, I saw several messages like the following from last
> night's logs:
>       Mar  1 00:35:03 stanley mfsmount[1253]: file: 161779, index: 0,
> chunk: 310854, version: 1 - writeworker: connection with (80D2302F:9422)
> was timed out (unfinished writes: 5; try counter: 1)
>       Mar  1 00:35:05 stanley mfsmount[1253]: file: 161779, index: 0,
> chunk: 310854, version: 1 - writeworker: connection with (80D2302F:9422)
> was timed out (unfinished writes: 5; try counter: 1)
>
> Thanks,
> Steve
>
>
> ----------------------------------------------------------------------------
> --
> Free Software Download: Index, Search&  Analyze Logs and other IT data in
> Real-Time with Splunk. Collect, index and harness all the fast moving IT
> data
> generated by your applications, servers and devices whether physical,
> virtual
> or in the cloud. Deliver compliance at lower cost and gain new business
> insights. http://p.sf.net/sfu/splunk-dev2dev
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>

-- 
Steven M. Wilson, Systems and Network Manager
Markey Center for Structural Biology
Purdue University
(765) 496-1946

Re: [Moosefs-users] mfs* pid files?

[Giovanni T. <gt...@li...>]

Il 03/03/2011 11:20, Michal Borychowski ha scritto:
> Hi!
> 
> Please have a look at these two answers:
> http://www.moosefs.org/moosefs-faq.html#master-pid 

Uh, there isn't something more scripting friendly? :P

Every daemon that forks in background should write a PID somewhere, why
mfs did not, even inside the lock file?

Thanks.

-- 
Giovanni Toraldo
http://www.libersoft.it/

Re: [Moosefs-users] mfs* pid files?

[Michal B. <mic...@ge...>]

Hi!

Please have a look at these two answers:
http://www.moosefs.org/moosefs-faq.html#master-pid 
http://www.moosefs.org/moosefs-faq.html#master-online


Regards
Michal

-----Original Message-----
From: Giovanni Toraldo [mailto:gt...@li...] 
Sent: Thursday, March 03, 2011 11:08 AM
To: moo...@li...
Subject: [Moosefs-users] mfs* pid files?

Hi,

I would like to configure a monitoring program for my MooseFS
components, and I was searching for pid files for mfsmaster,
mfsmetalogger and mfschunkserver.

Unfortunately, I didn't find anything both in /var/run/mfs (although the
folder it's created and chowned in the default init.d script) and
/var/lib/mfs (only empty hidden lock files are present).

I also tried to add -m --pidfile to start-stop-daemon on my init.d
script, but  it probably forks on startup from main process because I
get a wrong PID in it.

Thanks.

-- 
Giovanni Toraldo
http://www.libersoft.it/

----------------------------------------------------------------------------
--
Free Software Download: Index, Search & Analyze Logs and other IT data in 
Real-Time with Splunk. Collect, index and harness all the fast moving IT
data 
generated by your applications, servers and devices whether physical,
virtual
or in the cloud. Deliver compliance at lower cost and gain new business 
insights. http://p.sf.net/sfu/splunk-dev2dev 
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] Chunks

[randall <ra...@so...>]

On 03/02/2011 11:40 AM, Michal Borychowski wrote:

> The main purpose of MooseFS system is security not the space savings. And
> solution with RAID6 is not that secure. We generally advise not to use any
> RAIDs and using at least goal=2. 
> 

Wondering about this.

when no raid and goal=2, this would mean when using multiple disks per
server that each disk would be a seperate chunk location.

Can see the use of this when you use 1 single server as each copy would
reside on 2 seperate disks so you are somewhat protected against disk
failure.

but when you have 2 servers with each 12 disks (24 chunk locations),
does each chunk reside on 2 seperate servers giving protection against
server failure?

did read somewhere there is work done on "location awareness" spreading
each chunks over racks

Randall

[Moosefs-users] mfs* pid files?

[Giovanni T. <gt...@li...>]

Hi,

I would like to configure a monitoring program for my MooseFS
components, and I was searching for pid files for mfsmaster,
mfsmetalogger and mfschunkserver.

Unfortunately, I didn't find anything both in /var/run/mfs (although the
folder it's created and chowned in the default init.d script) and
/var/lib/mfs (only empty hidden lock files are present).

I also tried to add -m --pidfile to start-stop-daemon on my init.d
script, but  it probably forks on startup from main process because I
get a wrong PID in it.

Thanks.

-- 
Giovanni Toraldo
http://www.libersoft.it/

Re: [Moosefs-users] Chunks

[Michal B. <mic...@ge...>]

Hi!

Do you mean why we chose ext3 for the chunkservers? When we started building
our infrastructure ext4 was even not available then.


Regards
Michal


-----Original Message-----
From: Steve [mailto:st...@bo...] 
Sent: Wednesday, March 02, 2011 6:31 PM
To: Heiko Schröter
Cc: moo...@li...
Subject: Re: [Moosefs-users] Chunks


What is the feature in moosefs that makes you choose it over say ext4 ? 

 

I'm not understanding this yet!

 

-------Original Message------- 

 

From: Michal Borychowski 

Date: 02/03/2011 16:08:20 

To: 'Heiko Schröter' 

Cc: 'moosefs-users' 

Subject: Re: [Moosefs-users] Chunks 

 

Hi Heiko! 

 

We undestand your point and can see that using RAID6 + goal=1 is a little 

bit more economical than RAW HDD + goal=2 but this is not a huge difference 

as you still need some disks for the RAID6. 

 

The main purpose of MooseFS system is security not the space savings. And 

solution with RAID6 is not that secure. We generally advise not to use any 

RAIDs and using at least goal=2. 

 

Module responsible for rebalancing chunks operates on chunks (independently 

of the files). Each chunk is treated individually while making different 

operations. So the requested change is not just a matter of providing a 

simple patch, it would be a change in the "philosophy" of the system and 

unfortunately won't be possible. 

 

 

Kind regards 

Michal 

 

 

-----Original Message----- 

From: Heiko Schröter [mailto:sch...@iu...] 

Sent: Monday, February 28, 2011 4:13 PM 

To: mic...@ge... 

Subject: Chunks 

 

Hi Michael, 

 

sorry for seeing your post to the list too late before posting my 

screenshots. 

I can see your point, but to us it would be very important to have a 

"chunkabilty" or striping in other fs of one. 

 

The reason: 

 

We receive a lot of large satellite data files per day. (400MB to 2GB per 

file). 

The storage space is limited (because of the governmental funding), so we 

need to keep risks to a certain minimum with the ressources given. 

 

We are running Hardware raid6 on our chunkservers. So there is some safety 

margin here. 

 

But we need to make sure that in case of a total breakdown of a chunkserver 

only some files are lost to 100%, and not all files beeing damaged to a 

certain extend and therefore irrecoverable. 

 

So if I could be of any help testing a patch I would very much appreciate 

it. 

 

Thanks for your time looking into this. 

 

Regards 

Heiko 

 

 

 

 

 

 

Hi Heiko! 

 

You are definitely right! I made a mistake writing all chunks of the file 

with goal=1 would reside just on one chunkserver. Each chunk of the file 

would go (more or less by random) to different chunkservers. 

 

On the other hand we again focus on the point that using goal=1 is almost 

useless. Unless these are some temporary, unimportant files. The expectation


of distributed file system is to keep at least two copies of the file :) 

 

Thank you for being conscious :) 

 

 

----------------------------------------------------------------------------
-
 

Free Software Download: Index, Search & Analyze Logs and other IT data in 

Real-Time with Splunk. Collect, index and harness all the fast moving IT
data 

generated by your applications, servers and devices whether physical,
virtual 

or in the cloud. Deliver compliance at lower cost and gain new business 

insights. http://p.sf.net/sfu/splunk-dev2dev 

_______________________________________________ 

moosefs-users mailing list 

moo...@li... 

https://lists.sourceforge.net/lists/listinfo/moosefs-users 

 

----------------------------------------------------------------------------
--
Free Software Download: Index, Search & Analyze Logs and other IT data in 
Real-Time with Splunk. Collect, index and harness all the fast moving IT
data 
generated by your applications, servers and devices whether physical,
virtual
or in the cloud. Deliver compliance at lower cost and gain new business 
insights. http://p.sf.net/sfu/splunk-dev2dev 
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] Chunks

[Thomas S H. <tha...@gm...>]

Over ext4? moosefs is a different storage layer than ext4, do you mean xfs
over ext4? ext3 over ext4?

On Wed, Mar 2, 2011 at 10:30 AM, Steve <st...@bo...> wrote:

>
> What is the feature in moosefs that makes you choose it over say ext4 ?
>
>
>
> I'm not understanding this yet!
>
>
>
> -------Original Message-------
>
>
>
> From: Michal Borychowski
>
> Date: 02/03/2011 16:08:20
>
> To: 'Heiko Schröter'
>
> Cc: 'moosefs-users'
>
> Subject: Re: [Moosefs-users] Chunks
>
>
>
> Hi Heiko!
>
>
>
> We undestand your point and can see that using RAID6 + goal=1 is a little
>
> bit more economical than RAW HDD + goal=2 but this is not a huge difference
>
> as you still need some disks for the RAID6.
>
>
>
> The main purpose of MooseFS system is security not the space savings. And
>
> solution with RAID6 is not that secure. We generally advise not to use any
>
> RAIDs and using at least goal=2.
>
>
>
> Module responsible for rebalancing chunks operates on chunks (independently
>
> of the files). Each chunk is treated individually while making different
>
> operations. So the requested change is not just a matter of providing a
>
> simple patch, it would be a change in the "philosophy" of the system and
>
> unfortunately won't be possible.
>
>
>
>
>
> Kind regards
>
> Michal
>
>
>
>
>
> -----Original Message-----
>
> From: Heiko Schröter [mailto:sch...@iu...]
>
> Sent: Monday, February 28, 2011 4:13 PM
>
> To: mic...@ge...
>
> Subject: Chunks
>
>
>
> Hi Michael,
>
>
>
> sorry for seeing your post to the list too late before posting my
>
> screenshots.
>
> I can see your point, but to us it would be very important to have a
>
> "chunkabilty" or striping in other fs of one.
>
>
>
> The reason:
>
>
>
> We receive a lot of large satellite data files per day. (400MB to 2GB per
>
> file).
>
> The storage space is limited (because of the governmental funding), so we
>
> need to keep risks to a certain minimum with the ressources given.
>
>
>
> We are running Hardware raid6 on our chunkservers. So there is some safety
>
> margin here.
>
>
>
> But we need to make sure that in case of a total breakdown of a chunkserver
>
> only some files are lost to 100%, and not all files beeing damaged to a
>
> certain extend and therefore irrecoverable.
>
>
>
> So if I could be of any help testing a patch I would very much appreciate
>
> it.
>
>
>
> Thanks for your time looking into this.
>
>
>
> Regards
>
> Heiko
>
>
>
>
>
>
>
>
>
>
>
>
>
> Hi Heiko!
>
>
>
> You are definitely right! I made a mistake writing all chunks of the file
>
> with goal=1 would reside just on one chunkserver. Each chunk of the file
>
> would go (more or less by random) to different chunkservers.
>
>
>
> On the other hand we again focus on the point that using goal=1 is almost
>
> useless. Unless these are some temporary, unimportant files. The
> expectation
>
>
> of distributed file system is to keep at least two copies of the file :)
>
>
>
> Thank you for being conscious :)
>
>
>
>
>
>
> -----------------------------------------------------------------------------
>
>
> Free Software Download: Index, Search & Analyze Logs and other IT data in
>
> Real-Time with Splunk. Collect, index and harness all the fast moving IT
> data
>
> generated by your applications, servers and devices whether physical,
> virtual
>
> or in the cloud. Deliver compliance at lower cost and gain new business
>
> insights. http://p.sf.net/sfu/splunk-dev2dev
>
> _______________________________________________
>
> moosefs-users mailing list
>
> moo...@li...
>
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>
>
>
> ------------------------------------------------------------------------------
> Free Software Download: Index, Search & Analyze Logs and other IT data in
> Real-Time with Splunk. Collect, index and harness all the fast moving IT
> data
> generated by your applications, servers and devices whether physical,
> virtual
> or in the cloud. Deliver compliance at lower cost and gain new business
> insights. http://p.sf.net/sfu/splunk-dev2dev
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>

Re: [Moosefs-users] Uneven distribution of files over physical storage

[Stas O. <sta...@gm...>]

The usage of the first disk is growing and probably going to completely
fill-up soon, should it cause any problems?

The used version is 1.6.17 by the way,

Thanks.

On Wed, Mar 2, 2011 at 10:30 PM, Stas Oskin <sta...@gm...> wrote:

> Hi.
>
> I have some strange problem, where the files are distributed un-evenly over
> physical storage.
>
> We have 5 disks per test server, with each of server have it's disks filled
> out in following fashion:
> 85%
> 45%
> 45%
> 45%
> 45%
>
> The 1st disk is relatively small (100 GB), while the rest are large (1.8
> TB).
>
> Any idea if this situation is normal and the first disk supposed to fill
> up, or there should be some kind of distribution amongst the disks?
>
> Thanks.
>

[Moosefs-users] Uneven distribution of files over physical storage

[Stas O. <sta...@gm...>]

Hi.

I have some strange problem, where the files are distributed un-evenly over
physical storage.

We have 5 disks per test server, with each of server have it's disks filled
out in following fashion:
85%
45%
45%
45%
45%

The 1st disk is relatively small (100 GB), while the rest are large (1.8
TB).

Any idea if this situation is normal and the first disk supposed to fill up,
or there should be some kind of distribution amongst the disks?

Thanks.