moosefs-users — Mailing list for MooseFS Users

Re: [Moosefs-users] Reliably detecting MFS is online

[Michal B. <mic...@ge...>]

Hi!

When the master server goes down while mfsmount is already running, mfsmount
doesn't disconnect the mounted resource and files awaiting to be saved would
stay quite long in the queue while trying to reconnect to the master server.
After a specified number of tries they eventually return EIO - "input/output
error". On the other hand it is not possible to start mfsmount when the
master doesn't work.

There are several ways to assure that the master server is online, we
present them below. 

1. You can check if you can connect to the port of the master server.

2. In order to assure that a MooseFS resource is mounted it is enough to
check the inode number - MooseFS root will always have inode equal to 1. For
example if we have MooseFS installation in "/mnt/mfs" then "stat /mnt/mfs"
command (in Linux) will show:

$ stat /mnt/mfs

File: `/mnt/mfs'

Size: xxxxxx Blocks: xxx IO Block: 4096 directory

Device: 13h/19d Inode: 1 Links: xx

(...)

 

3. Additionaly "mfsmount" simulates in the root folder a hidden file
".stats". So when MooseFS is mounted we would get statistics data of
mfsmount, eg.:

 

$ cat /mnt/mfs/.stats

fuse_ops.statfs: 241

fuse_ops.access: 0

fuse_ops.lookup-cached: 707553

fuse_ops.lookup: 603335

fuse_ops.getattr-cached: 24927

fuse_ops.getattr: 687750

fuse_ops.setattr: 24018

fuse_ops.mknod: 0

fuse_ops.unlink: 23083

fuse_ops.mkdir: 4

fuse_ops.rmdir: 1

fuse_ops.symlink: 3

fuse_ops.readlink: 454

fuse_ops.rename: 269

(...)

 

4. If you want to be sure that master server properly responds you need to
try to read the goal of any object, specifically of the root folder:

$ mfsgetgoal /mnt/mfs

/mnt/mfs: 2

If you get a proper goal of the root folder you can be sure that the master
server responds properly (hasn't hung up).

 

Regards

Michal

 

 

 

 

 

 

> Hi.

> 

> I wonder how other MFS users detect if their cluster is online?

> Because if the MFS goes down, from application point of view, the mount

> directory still up, and it will continue writing to local disk

> 

> We solve this by placing a special control file in root of the cluster,
but

> I wonder if there is a better solution, perhaps on MFS level?

> 

> Regards.

[Moosefs-users] Crashing mfsmaster

[Thomas S H. <tha...@gm...>]

We are getting these kernel errors:
Dec 30 20:17:34 localhost kernel: [1818169.391998] mfschunkserver: page
allocation failure. order:2, mode:0x4020
Dec 30 20:17:34 localhost kernel: [1818169.392008] Pid: 8290, comm:
mfschunkserver Not tainted 2.6.32-25-server #45-Ubuntu
Dec 30 20:17:34 localhost kernel: [1818169.392014] Call Trace:
Dec 30 20:17:34 localhost kernel: [1818169.392019]  <IRQ>
 [<ffffffff810f9a2e>] __alloc_pages_slowpath+0x56e/0x580
Dec 30 20:17:34 localhost kernel: [1818169.392046]  [<ffffffff810f9bb1>]
__alloc_pages_nodemask+0x171/0x180
Dec 30 20:17:34 localhost kernel: [1818169.392055]  [<ffffffff81131df2>]
kmalloc_large_node+0x62/0xb0
Dec 30 20:17:34 localhost kernel: [1818169.392063]  [<ffffffff81136409>]
__kmalloc_node_track_caller+0x109/0x160
Dec 30 20:17:34 localhost kernel: [1818169.392073]  [<ffffffff81468d8d>] ?
dev_alloc_skb+0x1d/0x40
Dec 30 20:17:34 localhost kernel: [1818169.392078]  [<ffffffff81468aa0>]
__alloc_skb+0x80/0x190
Dec 30 20:17:34 localhost kernel: [1818169.392084]  [<ffffffff81468d8d>]
dev_alloc_skb+0x1d/0x40
Dec 30 20:17:34 localhost kernel: [1818169.392109]  [<ffffffffa001af27>]
nv_alloc_rx_optimized+0x197/0x270 [forcedeth]
Dec 30 20:17:34 localhost kernel: [1818169.392120]  [<ffffffffa001a369>] ?
T.936+0x269/0x2a0 [forcedeth]
Dec 30 20:17:34 localhost kernel: [1818169.392130]  [<ffffffffa001c09c>]
nv_nic_irq_optimized+0xdc/0x330 [forcedeth]
Dec 30 20:17:34 localhost kernel: [1818169.392138]  [<ffffffff810c4030>]
handle_IRQ_event+0x60/0x170
Dec 30 20:17:34 localhost kernel: [1818169.392145]  [<ffffffff810c6472>]
handle_edge_irq+0xd2/0x170
Dec 30 20:17:34 localhost kernel: [1818169.392152]  [<ffffffff81014d12>]
handle_irq+0x22/0x30
Dec 30 20:17:34 localhost kernel: [1818169.392161]  [<ffffffff8155f29c>]
do_IRQ+0x6c/0xf0
Dec 30 20:17:34 localhost kernel: [1818169.392166]  [<ffffffff81012b13>]
ret_from_intr+0x0/0x11
Dec 30 20:17:34 localhost kernel: [1818169.392174]  [<ffffffff8106d494>] ?
__do_softirq+0xd4/0x1e0
Dec 30 20:17:34 localhost kernel: [1818169.392180]  [<ffffffff810c4030>] ?
handle_IRQ_event+0x60/0x170
Dec 30 20:17:34 localhost kernel: [1818169.392187]  [<ffffffff810132ec>] ?
call_softirq+0x1c/0x30
Dec 30 20:17:34 localhost kernel: [1818169.392192]  [<ffffffff81014cb5>] ?
do_softirq+0x65/0xa0
Dec 30 20:17:34 localhost kernel: [1818169.392197]  [<ffffffff8106d315>] ?
irq_exit+0x85/0x90
Dec 30 20:17:34 localhost kernel: [1818169.392203]  [<ffffffff8155f2a5>] ?
do_IRQ+0x75/0xf0
Dec 30 20:17:34 localhost kernel: [1818169.392208]  [<ffffffff81012b13>] ?
ret_from_intr+0x0/0x11
Dec 30 20:17:34 localhost kernel: [1818169.392211]  <EOI>
 [<ffffffff812bc2ad>] ? copy_user_generic_string+0x2d/0x40
Dec 30 20:17:34 localhost kernel: [1818169.392227]  [<ffffffff814af860>] ?
tcp_sendmsg+0x860/0xa20
Dec 30 20:17:34 localhost kernel: [1818169.392236]  [<ffffffff814630cc>] ?
sock_aio_write+0x13c/0x150
Dec 30 20:17:34 localhost kernel: [1818169.392245]  [<ffffffff8114378a>] ?
do_sync_write+0xfa/0x140
Dec 30 20:17:34 localhost kernel: [1818169.392253]  [<ffffffff8105a254>] ?
try_to_wake_up+0x284/0x380
Dec 30 20:17:34 localhost kernel: [1818169.392261]  [<ffffffff81084240>] ?
autoremove_wake_function+0x0/0x40
Dec 30 20:17:34 localhost kernel: [1818169.392269]  [<ffffffff8101078c>] ?
__switch_to+0x1ac/0x320
Dec 30 20:17:34 localhost kernel: [1818169.392276]  [<ffffffff81057850>] ?
finish_task_switch+0x50/0xe0
Dec 30 20:17:34 localhost kernel: [1818169.392285]  [<ffffffff81251fd6>] ?
security_file_permission+0x16/0x20
Dec 30 20:17:34 localhost kernel: [1818169.392292]  [<ffffffff81143b54>] ?
vfs_write+0x184/0x1a0
Dec 30 20:17:34 localhost kernel: [1818169.392298]  [<ffffffff811442f1>] ?
sys_write+0x51/0x80
Dec 30 20:17:34 localhost kernel: [1818169.392305]  [<ffffffff810121b2>] ?
system_call_fastpath+0x16/0x1b

For mfschunkservers, master and metaloggers, when we see them for the
mfsmaster then the mfsmaster starts to go crazy and all of the chunkservers
disconnect, when I stop the mfsmaster the metadata file is not created and I
have to use metalogger data or the metadata.back.tmp file.

We noticed that if we echo 3 > /proc/sys/vm/drop_caches then the kernel
errors go away, but we can't afford to have these crashes, this is the
second time this week.

This is a catastrophic problem, please let me know what you think, right now
our guess is that it is related to a kernel bug.

We are running 1.6.19 on Ubuntu 10.04 kernel 2.6.32.

Thanks

-Tom Hatch

Re: [Moosefs-users] 1.6.19 RPM\\\'s?

[Laurent W. <lw...@hy...>]

On Thu, 30 Dec 2010 11:20:04 +0200
Stas Oskin <sta...@gm...> wrote:

> Hi.
> 
> Thanks, I have the RPMForge repo defined, there are no latest MFS packages
> there yet.
> 
> Any idea when they will become available?
They're in the queue to be rebuild.
If you're in a hurry, you can use
http://svn.rpmforge.net/svn/trunk/rpms/mfs/ to build your own packages.
HTH,
-- 
Laurent Wandrebeck
HYGEOS, Earth Observation Department / Observation de la Terre
Euratechnologies
165 Avenue de Bretagne
59000 Lille, France
tel: +33 3 20 08 24 98
http://www.hygeos.com
GPG fingerprint/Empreinte GPG: F5CA 37A4 6D03 A90C 7A1D  2A62 54E6 EF2C
D17C F64C

Re: [Moosefs-users] 1.6.19 RPM\\\'s?

[Stas O. <sta...@gm...>]

Hi.

Thanks, I have the RPMForge repo defined, there are no latest MFS packages
there yet.

Any idea when they will become available?

Regards.

On Wed, Dec 29, 2010 at 5:34 PM, MooseFS <co...@mo...> wrote:

> Hi!
>
> The recommendation from RPMForge is as follows:
>
>
> MooseFS packages for RHEL/CentOS 3/4/5/6 are available from the RPMforge
> repository. Instructions for configuring RPMforge are here:
> http://wiki.centos.org/AdditionalResources/Repositories/RPMForge
>
>
> MooseFS clients need the 'mfs-client' package (it also contains utilities
> like mfssetgoal or mfssnapshot for administration); servers need the 'mfs'
> package (it has the daemons and separate init scripts for master server,
> metalogger and chunk server).  The 'mfs-cgi' package provides the MooseFS
> status CGI, configured to be served by Apache.
>
>
> Regards
> -Michal
>
> > Not from the RPMForge at least.
> >
> > On Thu, Dec 23, 2010 at 12:02 PM, MooseFS <co...@mo...> wrote:
> >
> > > It should already be ready. Steve received sources yesterday. Are the
> rpms
> > > not available?
> > >
> > >
> > > Regards
> > > Michal
> > >
> > > > Hi.
> > > >
> > > > Any idea when RPMForge will be updated with the latest versions of
> MFS?
> > > >
> > > > Thanks.
> > >
> > >
> > >
>
>
>

Re: [Moosefs-users] jumbo frames

[Stas O. <sta...@gm...>]

Hi Michal.

Thanks for the answer, I will make sure to address the main list in future.

Regards.

On Thu, Dec 30, 2010 at 9:31 AM, MooseFS <co...@mo...> wrote:

> I remember you had a question about jumbo frames - they should be enabled
> on the *whole* network. Otherwise network would not be efficient.
>
> PS. Please in the future send questions to
> moo...@li... - there are already about 100 users
> where we share experiences.
>
> Thanks!
>
>
> Regards
> Michal
>
>
>

Re: [Moosefs-users] How many mfs clients can be supported by the same master at the same time!!

[Michal B. <mic...@ge...>]

Hi!

From: yanbh [mailto:ya...@in...] 
Sent: Thursday, December 30, 2010 7:49 AM
To: moosefs-users
Subject: [Moosefs-users] How many mfs clients can be supported by the same
master at the same time!!

 

Dear all!

 

I have some questions about MFS, as following!

1. how many mfsclient can  be supported by the same master server at the
same time?

what about 100 client mounting the same mfs master  at the same? what about
the performace?

[MB] 100 clients would not be a problem. Performance should also not be
affected (metadata in mfs master are kept in RAM for speed). Please have a
look here: http://80.48.16.122/mfs.cgi?masterport=9421
<http://80.48.16.122/mfs.cgi?masterport=9421&mastername=bellona_main&section
s=MS> &mastername=bellona_main&sections=MS for our installation.

 

2. we know the mfs can only support a file whose size less than 2T, this is
a bad limitation, when to remove it ?

any plan?

[MB] Is it really a big problem for you? Can't you divide the files to be
not bigger than 2TB? For the moment removing this limiation doesn't have big
priority.

 

 

whatever, Thx for the developers giving us a execlent DFS!!

[MB] :)

 

If you need any further assistance please let us know.

 

Kind regards

Michał Borychowski 

MooseFS Support Manager

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Gemius S.A.

ul. Wołoska 7, 02-672 Warszawa

Budynek MARS, klatka D

Tel.: +4822 874-41-00

Fax : +4822 874-41-01

 

Best

Regards!

 

2010-12-30 

  _____  

yanbh

[Moosefs-users] How many mfs clients can be supported by the same master at the same time!!

[yanbh <ya...@in...>]

Dear all!

I have some questions about MFS, as following!
1. how many mfsclient can  be supported by the same master server at the same time?
what about 100 client mounting the same mfs master  at the same? what about the performace?

2. we know the mfs can only support a file whose size less than 2T, this is a bad limitation, when to remove it ?
any plan?

whatever, Thx for the developers giving us a execlent DFS!!

Best
Regards!

2010-12-30 



yanbh

[Moosefs-users] How many mfs clients can be supported by the same master at the same time!!

[yanbh <ya...@in...>]

Dear all!

I have some questions about MFS, as following!
1. how many mfsclient can  be supported by the same master server at the same time?
what about 100 client mounting the same mfs master  at the same? what about the performace?

2. we know the mfs can only support a file whose size less than 2T, this is a bad limitation, when to remove it ?
any plan?

whatever, Thx for the developers giving us a execlent DFS!!

Best
Regards!

2010-12-30 



yanbh

Re: [Moosefs-users] nfs export of mfs mount point

[harrylogin <har...@16...>]

Message: 1
Date: Thu, 23 Dec 2010 14:25:14 -0700
From: Jun Cheol Park <jun...@gm...>
Subject: [Moosefs-users] nfs export of mfs mount point
To: moosefs-users <moo...@li...>
Message-ID:
<AANLkTinUhcnc4=MqOOnZMTFf_VUYeuCa=szk...@ma...>
Content-Type: text/plain; charset=ISO-8859-1
Hi,
Using nfs (v4 on CentOS 5.5) on top of mfs is definitely silly.
However, some applications that I plan to use can run only with nfs,
so I tried to export mfs mount point as an nfs export. For easiness,
right now I am not using any iptables rules on any node.
On the nfs server (10.1.2.22, at the same time, as a mfs client):
# cat /etc/exports
/mnt/mfs 10.*(rw,no_root_squash)
# df | grep mfs
mfs#10.1.2.22:9421 on /mnt/mfs type fuse (rw,allow_other,default_permissions)
On the nfs client (10.1.2.24):
I got the following error.
# mount -t nfs 10.1.2.22:/mnt/mfs /mnt/nfs_mfs/
mount: 10.1.2.22:/mnt/mfs failed, reason given by server: Permission denied
# tail /var/log/messages
Dec 23 14:23:23 host1a mountd[17279]: authenticated mount request from
10.1.2.24:805 for /mnt/mfs (/mnt/mfs)
Dec 23 14:23:23 host1a mountd[17279]: Cannot export /mnt/mfs, possibly
unsupported filesystem or fsid= required
Is there anyone who knows a workaround?
Thanks,
-Jun


---------------------------------
hi,may be you can change /etc/exports like this

/mnt/mfs/test    *(rw,fsid=10)

Thanks,
-harryzhang

2010-12-30 

[Moosefs-users] Remount mfs-client

[Thomas S H. <tha...@gm...>]

I just upgraded to moosefs 1.6.19 and I need to remount my clients with the
new stuff. When I try:

$ mount -o remount <mount point>

it fails, is there a way to remount an mfsmount without first unmounting it?

-Tom

Re: [Moosefs-users] parallel (ie: slow) writes acroos a WAN link?

[Michal B. <mic...@ge...>]

Hi!

 

Yes, if you save a large file in location A you'd have to wait till it's
written also in a remote location B. If you have a really quick WAN
connection there would be no problem (eg. 1 Gbit/s). If your WAN connection
is only 10-30Mbit/s the system would be very very slow and it won't be
convenient to use it at all. So it just depends on the speed of the
connection.

 

 

Kind regards

Michał Borychowski 

MooseFS Support Manager

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Gemius S.A.

ul. Wołoska 7, 02-672 Warszawa

Budynek MARS, klatka D

Tel.: +4822 874-41-00

Fax : +4822 874-41-01

 

From: Linux Expert [mailto:lin...@gm...] 
Sent: Tuesday, December 21, 2010 7:32 PM
To: moo...@li...
Subject: [Moosefs-users] parallel (ie: slow) writes acroos a WAN link?

 

I'm researching how to setup a 2-server filesystem with complete live
replication across a WAN link.  The total FS size is 1TB and each server
will hold a complete replica, with users on each end modifying files.  Users
run windows and will access the files via Samba. 

 

For various reasons, I think MooseFS is the best solution but I had a
question regarding writes. Moose will have a goal of 2 for the entire 1TB
directory.  If a user on Site A saves a large file, will they have to wait
until that file is saved to the remote node across the WAN link, or will
MooseFS save it to the local server the users is connected to, and replicate
it asynchronously in the background?  

Re: [Moosefs-users] about the size of chunk[help]

[Steve <st...@bo...>]

http://www.moosefs.org/moosefs-faq.html#modify_chunk



The moosefs website has excellent documentation 

 

 

 

 

-------Original Message------- 

 

From: Wang hanwen 

Date: 29/12/2010 15:03:08 

To: moo...@li... 

Subject: [Moosefs-users] about the size of chunk[help] 

 

hi: 

I have a question about the size of every chunk. 

we know the file will be devided into some chunks stored in chunkservers
however I can not get the information about the size of every chunk and want
to know whether the size can be set,such as 2M,8M,even 64M 

 

thanks 

yours wanghw 

 

 

 

 

 

-----------------------------------------------------------------------------
 

Learn how Oracle Real Application Clusters (RAC) One Node allows customers 

to consolidate database storage, standardize their database environment, and
 

should the need arise, upgrade to a full multi-node Oracle RAC database 

without downtime or disruption 

http://p.sf.net/sfu/oracle-sfdevnl 

 

 

 

 

 

_______________________________________________ 

moosefs-users mailing list 

moo...@li... 

https://lists.sourceforge.net/lists/listinfo/moosefs-users 

 

Re: [Moosefs-users] FreeBSD port update

[Michal B. <mic...@ge...>]

It's undergoing. Chifeng has prepared them and they wait for approval


Regards
Michal


-----Original Message-----
From: Alexander Akhobadze [mailto:akh...@ri...] 
Sent: Wednesday, December 29, 2010 2:47 PM
To: moo...@li...
Subject: [Moosefs-users] FreeBSD port update

Hi All!

How about update port in FreeBSD ports collection
by fresh MooseFS v 1.6.19 ?

wbr
Alexander Akhobadze


----------------------------------------------------------------------------
--
Learn how Oracle Real Application Clusters (RAC) One Node allows customers
to consolidate database storage, standardize their database environment,
and, 
should the need arise, upgrade to a full multi-node Oracle RAC database 
without downtime or disruption
http://p.sf.net/sfu/oracle-sfdevnl
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] FreeBSD port update

[<leo...@ar...>]

Hi!
Of course, port is good,
but if you just need the new version installed, 
it didn't make any problem to install it from the sources on FreeBSD.


On Wed, 29 Dec 2010 16:46:30 +0300, Alexander Akhobadze
<akh...@ri...> wrote:
> Hi All!
> 
> How about update port in FreeBSD ports collection
> by fresh MooseFS v 1.6.19 ?
> 
> wbr
> Alexander Akhobadze
> 
> 
>
------------------------------------------------------------------------------
> Learn how Oracle Real Application Clusters (RAC) One Node allows
customers
> to consolidate database storage, standardize their database environment,
> and, 
> should the need arise, upgrade to a full multi-node Oracle RAC database 
> without downtime or disruption
> http://p.sf.net/sfu/oracle-sfdevnl
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

[Moosefs-users] FreeBSD port update

[Alexander A. <akh...@ri...>]

Hi All!

How about update port in FreeBSD ports collection
by fresh MooseFS v 1.6.19 ?

wbr
Alexander Akhobadze

[Moosefs-users] about the size of chunk[help]

[Wang h. <wan...@gm...>]

hi:
    I have a question about the size of every chunk.
    we know the file will be devided into some chunks stored in
chunkservers,however i can not get the information about the size of every
chunk and want to know whether the size can be set,such as 2M,8M,even 64M

thanks
yours wanghw

Re: [Moosefs-users] Strange chunk of goal 0, 1 copy

[Michal B. <mic...@ge...>]

Please go to the web monitor and on the "Master Charts" check the "chunk deletions (per minute)" chart. Is there any information about deleted chunks? You can also send to me a screenshot of them.


Kind regards
Michal


-----Original Message-----
From: leo...@ar... [mailto:leo...@ar...] 
Sent: Wednesday, December 29, 2010 11:02 AM
To: Michal Borychowski
Cc: moo...@li...
Subject: RE: [Moosefs-users] Strange chunk of goal 0, 1 copy

Interesting, ... 
More than a day have passed, there should be plenty of idle time, 
the filesystem is almost not written to or read from. The discussed chunk
is still there.
And is it normal in this situation that ...

---------------------------------
%sudo find /mnt/mfs_meta/
/mnt/mfs_meta/
/mnt/mfs_meta/trash
/mnt/mfs_meta/trash/undel
/mnt/mfs_meta/reserved
%
---------------------------------

shows no deleted files?...



On Wed, 29 Dec 2010 09:30:50 +0100, "Michal Borychowski"
<mic...@ge...> wrote:
> Hi Leonid!
> 
> This is normal. This is a way to mark chunks belonging to the
non-existing
> files (i.e. deleted ones). Deleting of files is done asynchronously in
> MooseFS. First a file is removed from metadata and its chunks are marked
as
> unnecessary (goal=0). Later the chunks are removed in an "idle" time.
This
> is much more effective than to erase everything in the moment of
deleting
> the file.
> 
> 
> Regards
> Michal
> 
> 
> 
> -----Original Message-----
> From: Leonid Satanovsky [mailto:leo...@ar...] 
> Sent: Tuesday, December 28, 2010 2:31 PM
> To: moo...@li...
> Subject: [Moosefs-users] Strange chunk of goal 0, 1 copy
> 
> Greetings, everyone!
> Recently while experementing with MooseFS 1.6.19 on FreeBSD 7.2 I came
to
> the following strange state:
> Web interface shows me that there is 1 copy of a chunk of goal 0.
> What is the chunk?...
> I have only files in the root directory of the MooseFS.
> 
> % sudo mfsfileinfo /mnt/mfs_test/* | grep chunk | wc -l
>       16
> So, these are 16 chunks of goal 2 which web interface correctly shows
me.
> Where the strage "17th chunk" comes from (All FS objects = 17)? Why has
it
> goal 0?...
> How can I figure it out?...
> -------------------------------------
> Gzipped metadata.mfs.back is in attachments.
> I'w greatly appreciate any help on the subject.
> Best regards, Leonid.

Re: [Moosefs-users] Strange chunk of goal 0, 1 copy

[<leo...@ar...>]

Interesting, ... 
More than a day have passed, there should be plenty of idle time, 
the filesystem is almost not written to or read from. The discussed chunk
is still there.
And is it normal in this situation that ...

---------------------------------
%sudo find /mnt/mfs_meta/
/mnt/mfs_meta/
/mnt/mfs_meta/trash
/mnt/mfs_meta/trash/undel
/mnt/mfs_meta/reserved
%
---------------------------------

shows no deleted files?...



On Wed, 29 Dec 2010 09:30:50 +0100, "Michal Borychowski"
<mic...@ge...> wrote:
> Hi Leonid!
> 
> This is normal. This is a way to mark chunks belonging to the
non-existing
> files (i.e. deleted ones). Deleting of files is done asynchronously in
> MooseFS. First a file is removed from metadata and its chunks are marked
as
> unnecessary (goal=0). Later the chunks are removed in an "idle" time.
This
> is much more effective than to erase everything in the moment of
deleting
> the file.
> 
> 
> Regards
> Michal
> 
> 
> 
> -----Original Message-----
> From: Leonid Satanovsky [mailto:leo...@ar...] 
> Sent: Tuesday, December 28, 2010 2:31 PM
> To: moo...@li...
> Subject: [Moosefs-users] Strange chunk of goal 0, 1 copy
> 
> Greetings, everyone!
> Recently while experementing with MooseFS 1.6.19 on FreeBSD 7.2 I came
to
> the following strange state:
> Web interface shows me that there is 1 copy of a chunk of goal 0.
> What is the chunk?...
> I have only files in the root directory of the MooseFS.
> 
> % sudo mfsfileinfo /mnt/mfs_test/* | grep chunk | wc -l
>       16
> So, these are 16 chunks of goal 2 which web interface correctly shows
me.
> Where the strage "17th chunk" comes from (All FS objects = 17)? Why has
it
> goal 0?...
> How can I figure it out?...
> -------------------------------------
> Gzipped metadata.mfs.back is in attachments.
> I'w greatly appreciate any help on the subject.
> Best regards, Leonid.

Re: [Moosefs-users] Strange chunk of goal 0, 1 copy

[Michal B. <mic...@ge...>]

Hi Leonid!

This is normal. This is a way to mark chunks belonging to the non-existing
files (i.e. deleted ones). Deleting of files is done asynchronously in
MooseFS. First a file is removed from metadata and its chunks are marked as
unnecessary (goal=0). Later the chunks are removed in an "idle" time. This
is much more effective than to erase everything in the moment of deleting
the file.


Regards
Michal



-----Original Message-----
From: Leonid Satanovsky [mailto:leo...@ar...] 
Sent: Tuesday, December 28, 2010 2:31 PM
To: moo...@li...
Subject: [Moosefs-users] Strange chunk of goal 0, 1 copy

Greetings, everyone!
Recently while experementing with MooseFS 1.6.19 on FreeBSD 7.2 I came to
the following strange state:
Web interface shows me that there is 1 copy of a chunk of goal 0.
What is the chunk?...
I have only files in the root directory of the MooseFS.

% sudo mfsfileinfo /mnt/mfs_test/* | grep chunk | wc -l
      16
So, these are 16 chunks of goal 2 which web interface correctly shows me.
Where the strage "17th chunk" comes from (All FS objects = 17)? Why has it
goal 0?...
How can I figure it out?...
-------------------------------------
Gzipped metadata.mfs.back is in attachments.
I'w greatly appreciate any help on the subject.
Best regards, Leonid.

Re: [Moosefs-users] nfs export of mfs mount point

[Jun C. P. <jun...@gm...>]

For those who are wondering how to export an MFS mount point as an NFS
mount point, I would like share my experience that uses a user space
NFS (unfs). Basically, I found a simple workaround with performance
degradation as expected (the use of NFS inherently hurts performance,
anyway).

Environment:
* CentOS 5.5
* unfs3-0.9.22
* mfs-1.6.15-2
* unfs server on 10.1.2.22

On user space NFS server:
# wget ftp://ftp.pbone.net/mirror/ftp.pramberger.at/systems/linux/contrib/rhel5/i386/unfs3-0.9.22-1.el5.pp.i386.rpm
# rpm -Uhv unfs3-0.9.22-1.el5.pp.i386.rpm
# cat /etc/exports (assuming 10.* private ip)
   /mnt/mfs 10.0.0.0/255.0.0.0(rw,no_root_squash)
# df | grep mfs
   mfs#10.1.2.22:9421   13020730816 428096384 12592634432   4% /mnt/mfs
# service nfs stop (if running)
# rpc.mountd
# portmap
# unfsd (if you want to debug, use -d)
# exportfs -a -v (if you want to add more exports)

On NFS client:
# cat /etc/fstab |grep mynfs (/mnt/mfs is an MFS mount point on the server)
10.1.2.22:/mnt/mfs /mnt/mynfs nfs rw,proto=tcp,nolock 0 0
# mount /mnt/mynfs

If you have iptables running, the following iptables rules need to be
added to have portmap and unfsd get along with firewall.

# PASS ICMP TYPE 3 PACKETS, INPUT AND OUTPUT
-A INPUT  -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A OUTPUT  -p icmp -m icmp --icmp-type 3 -j ACCEPT

# PASS PORTMAP AT 111, INPUT AND OUTPUT, TCP AND UDP
-A INPUT -p tcp -m tcp --dport 111 -j ACCEPT
-A INPUT -p udp -m udp --dport 111 -j ACCEPT
-A OUTPUT -p tcp -m tcp --sport 111 -j ACCEPT
-A OUTPUT -p udp -m udp --sport 111 -j ACCEPT

# PASS NFS AT 2049, INPUT AND OUTPUT, TCP AND UDP
-A INPUT -p tcp -m tcp --dport 2049 -j ACCEPT
-A INPUT -p udp -m udp --dport 2049 -j ACCEPT
-A OUTPUT -p tcp -m tcp --sport 2049 -j ACCEPT
-A OUTPUT -p udp -m udp --sport 2049 -j ACCEPT

I hope this can help for anyone who needs it.

-Jun

On Thu, Dec 23, 2010 at 2:25 PM, Jun Cheol Park
<jun...@gm...> wrote:
> Hi,
>
> Using nfs (v4 on CentOS 5.5) on top of mfs is definitely silly.
> However, some applications that I plan to use can run only with nfs,
> so I tried to export mfs mount point as an nfs export. For easiness,
> right now I am not using any iptables rules on any node.
>
> On the nfs server (10.1.2.22, at the same time, as a mfs client):
>
> # cat /etc/exports
> /mnt/mfs 10.*(rw,no_root_squash)
>
> # df | grep mfs
> mfs#10.1.2.22:9421 on /mnt/mfs type fuse (rw,allow_other,default_permissions)
>
> On the nfs client (10.1.2.24):
>
> I got the following error.
>
> # mount -t nfs 10.1.2.22:/mnt/mfs /mnt/nfs_mfs/
> mount: 10.1.2.22:/mnt/mfs failed, reason given by server: Permission denied
>
> # tail /var/log/messages
> Dec 23 14:23:23 host1a mountd[17279]: authenticated mount request from
> 10.1.2.24:805 for /mnt/mfs (/mnt/mfs)
> Dec 23 14:23:23 host1a mountd[17279]: Cannot export /mnt/mfs, possibly
> unsupported filesystem or fsid= required
>
> Is there anyone who knows a workaround?
>
> Thanks,
>
> -Jun
>

[Moosefs-users] Strange chunk of goal 0, 1 copy

[Leonid S. <leo...@ar...>]

Greetings, everyone!
Recently while experementing with MooseFS 1.6.19 on FreeBSD 7.2 I came to 
the following strange state:
Web interface shows me that there is 1 copy of a chunk of goal 0.
What is the chunk?...
I have only files in the root directory of the MooseFS.

% sudo mfsfileinfo /mnt/mfs_test/* | grep chunk | wc -l
      16
So, these are 16 chunks of goal 2 which web interface correctly shows me.
Where the strage "17th chunk" comes from (All FS objects = 17)? Why has it 
goal 0?...
How can I figure it out?...
-------------------------------------
Gzipped metadata.mfs.back is in attachments.
I'w greatly appreciate any help on the subject.
Best regards, Leonid.

Re: [Moosefs-users] Moosefs automated failover with ucarp

[Thomas S H. <tha...@gm...>]

The assumption here is that there is a priority, there is no priority with
this failover setup. If B becomes the master then it has priority, and when
A comes back it becomes a metalogger and starts syncing data. If B then goes
 down then the setup.sh script will remove the old metadata on A, back it
up, and only restore from the metalogger data, this ensures that the new
changes the happened on B are now running on A.

This question makes more sense as to why you asked some of your earlier
questions, the thing about this failover mechanism is that there is no
single priority master, if there is then we have issues syncing back
information, like you pointed out. Instead it operates more like
an anonymous failover cluster, all of the machines are effectively the same,
and all of them need to be configured hardware wise to act as mfsmasters.

On Mon, Dec 27, 2010 at 9:17 AM, Leonid Satanovsky <leo...@ar...>wrote:

> Thank you for answering!...
> But there are still some more questions. One of which is the fillowing:
>
> It is about carp nodes priority.
> Let me consider the following scenario:
> 1) Host A runs master.
>   Host B runs metalogger.
>
> 2) Host As' carp iface goes down, host B gets the carp managed addres and
> starts running master.
>
> 3) After a while host As' carp interface goes up, host A it becomes master
> (e.g., it has the bigger priority),
> Q:
>  a) How does A synchronyse all the changes that were made to the FS, while
> the B was the master?...
>  b) If it is doesn't isn't the only way to solve the problem is make all
> the carp nodes of equal priority
>     (so that the host coming up starts metalogger to sync the data)?
>
>
>
> ----- Original Message ----- From: "Thomas S Hatch" <tha...@gm...>
> To: "Leonid Satanovsky" <leo...@ar...>; "moosefs-users" <
> moo...@li...>
> Sent: Monday, December 27, 2010 7:05 PM
> Subject: Re: [Moosefs-users] Moosefs automated failover with ucarp
>
>
>
>  As for your first question, when the failover occurs we want it to use the
>> metalogger data. There is a chance that the machine which is the failover
>> used to be a master, and that it has old metadata. The mv command is to
>> ensure that only metalogs are being used by mfsmetarestore and not
>> metadata
>> that would be old and from a a previous run as the mfsmaster.
>>
>> As for your second question, you are correct in that the contents of the
>> setup.sh could be executed inline in the vip-up script, so long as they
>> are
>> executed in the background, but you do need everything in the setup.sh to
>> be
>> executed, otherwise many conflicts can arise!
>>
>> And finally, yes, in the vip-down could probably use a few lines to
>> shutdown
>> the mfsmaster.
>>
>> -Tom Hatch
>>
>> On Mon, Dec 27, 2010 at 8:44 AM, Leonid Satanovsky <leo...@ar...
>> >wrote:
>>
>>  Greetings!
>>> Have reviewed the HOWTO, and there are two questions:
>>> 1)
>>>  You do the following:
>>>
>>>  <...>
>>>  mv $MFS/changelog.* $MFS/metadata.* $MFS/tmp/
>>>
>>>  service mfsmetalogger stop
>>>  mfsmetarestore -a   #!# BUT, On what files will it work if all the
>>> metadata and changelog files are "gone" to the "${MFS}/tmp" folder?
>>>
>>>  if [ -e $MFS/metadata.mfs ]    #!# The file will not be there, I think,
>>> cos' you have moved it to "${MFS}/tmp" folder
>>>
>>>  <...>
>>> _______________________________________________________________________
>>> 2) The second on is for all the audience:
>>> Isn't it just enough (or is it correct?...) to do the following:
>>>
>>> ------/ucarp-down-----
>>> #!/bin/sh
>>>
>>> <....>
>>>
>>> (
>>>      mfsmaster stop
>>>      mfsmetarestore -a   # DO WE NEED THIS?
>>>      mfsmetalogger start
>>> ) &
>>>
>>> ______________________________________________________________
>>>
>>> ------/ucarp-up-----
>>>
>>>
>>> #!/bin/sh
>>>
>>> <....>
>>>
>>> (
>>>      mfsmetalogger stop
>>>      killall -9 mfsmetalogger
>>>      mfsmetarestore -a          # DO WE NEED THIS?
>>>      mfsmaster start
>>> ) &
>>>
>>>
>>>
>>>
>>>
>>> ----- Original Message ----- From: "Thomas S Hatch" <tha...@gm...>
>>> To: "moosefs-users" <moo...@li...>
>>> Sent: Friday, December 24, 2010 8:11 AM
>>>
>>> Subject: [Moosefs-users] Moosefs automated failover with ucarp
>>>
>>>
>>>  Sorry, this has taken me a while, my present service deployment has been
>>>
>>>> very demanding, and heck, it needs 1.6.19 anyway!
>>>>
>>>> I wrote it in asciidoc, enjoy!
>>>>
>>>> (Consider this file to be Copyright Thomas S Hatch under the FDL
>>>> licence,
>>>> or
>>>> whatever the MooseFS developers prefer blah blah blah...)
>>>>
>>>> -Thomas S Hatch
>>>>
>>>>
>>>>
>>>
>>>
>>> --------------------------------------------------------------------------------
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>>
>>>> Learn how Oracle Real Application Clusters (RAC) One Node allows
>>>> customers
>>>> to consolidate database storage, standardize their database environment,
>>>> and,
>>>> should the need arise, upgrade to a full multi-node Oracle RAC database
>>>> without downtime or disruption
>>>> http://p.sf.net/sfu/oracle-sfdevnl
>>>>
>>>>
>>>
>>>
>>>
>>> --------------------------------------------------------------------------------
>>>
>>>
>>>  _______________________________________________
>>>
>>>> moosefs-users mailing list
>>>> moo...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>>>
>>>>
>>>>
>>>
>>
>

Re: [Moosefs-users] Moosefs automated failover with ucarp

[Thomas S H. <tha...@gm...>]

As for your first question, when the failover occurs we want it to use the
metalogger data. There is a chance that the machine which is the failover
used to be a master, and that it has old metadata. The mv command is to
ensure that only metalogs are being used by mfsmetarestore and not metadata
that would be old and from a a previous run as the mfsmaster.

As for your second question, you are correct in that the contents of the
setup.sh could be executed inline in the vip-up script, so long as they are
executed in the background, but you do need everything in the setup.sh to be
executed, otherwise many conflicts can arise!

And finally, yes, in the vip-down could probably use a few lines to shutdown
the mfsmaster.

-Tom Hatch

On Mon, Dec 27, 2010 at 8:44 AM, Leonid Satanovsky <leo...@ar...>wrote:

> Greetings!
> Have reviewed the HOWTO, and there are two questions:
> 1)
>  You do the following:
>
>   <...>
>   mv $MFS/changelog.* $MFS/metadata.* $MFS/tmp/
>
>   service mfsmetalogger stop
>   mfsmetarestore -a   #!# BUT, On what files will it work if all the
> metadata and changelog files are "gone" to the "${MFS}/tmp" folder?
>
>   if [ -e $MFS/metadata.mfs ]    #!# The file will not be there, I think,
> cos' you have moved it to "${MFS}/tmp" folder
>
>   <...>
> _______________________________________________________________________
> 2) The second on is for all the audience:
> Isn't it just enough (or is it correct?...) to do the following:
>
> ------/ucarp-down-----
> #!/bin/sh
>
> <....>
>
> (
>       mfsmaster stop
>       mfsmetarestore -a   # DO WE NEED THIS?
>       mfsmetalogger start
> ) &
>
> ______________________________________________________________
>
> ------/ucarp-up-----
>
>
> #!/bin/sh
>
> <....>
>
> (
>       mfsmetalogger stop
>       killall -9 mfsmetalogger
>       mfsmetarestore -a          # DO WE NEED THIS?
>       mfsmaster start
> ) &
>
>
>
>
>
> ----- Original Message ----- From: "Thomas S Hatch" <tha...@gm...>
> To: "moosefs-users" <moo...@li...>
> Sent: Friday, December 24, 2010 8:11 AM
>
> Subject: [Moosefs-users] Moosefs automated failover with ucarp
>
>
>  Sorry, this has taken me a while, my present service deployment has been
>> very demanding, and heck, it needs 1.6.19 anyway!
>>
>> I wrote it in asciidoc, enjoy!
>>
>> (Consider this file to be Copyright Thomas S Hatch under the FDL licence,
>> or
>> whatever the MooseFS developers prefer blah blah blah...)
>>
>> -Thomas S Hatch
>>
>>
>
>
> --------------------------------------------------------------------------------
>
>
> ------------------------------------------------------------------------------
>> Learn how Oracle Real Application Clusters (RAC) One Node allows customers
>> to consolidate database storage, standardize their database environment,
>> and,
>> should the need arise, upgrade to a full multi-node Oracle RAC database
>> without downtime or disruption
>> http://p.sf.net/sfu/oracle-sfdevnl
>>
>
>
>
> --------------------------------------------------------------------------------
>
>
>  _______________________________________________
>> moosefs-users mailing list
>> moo...@li...
>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>
>>
>

[Moosefs-users] Moosefs automated failover with ucarp

[Thomas S H. <tha...@gm...>]

Howto Build Automatic Failover With Ucarp for MooseFS
=====================================================

Often criticized as the only serious failing of MooseFS, and the only reason I
had placed it on the bottom of my initial research list for my latest service
deployment, MooseFS is lacking a built in failover mechanism.

But the MooseFS developers are highly conscious of this, and in my opinion,
could be very close to developing a built in failover/clustering mechanism.

The redundancy of the MooseFS chunk design is unparallelled in other production
open source distributed file systems and the failure recovery of the chunk
servers is what drew me to MooseFS over the other available options.

But right now, MooseFS only supports a single metadata server, hence the
problem with failover.

Despite this the MooseFS developers have developed a viable way to distribute
the metadata to backup machines via a metalogger. This is one of the core
components of this failover design.

The other component I will be using is Ucarp. Ucarp is a network level ip
redundancy system with execution hooks which we will be using to patch together
the failover.

Step One, Set Up the Metaloggers
--------------------------------

This howto will assume that you already have a MooseFS installation and are
using the recommended mfsmaster hostname setup.

The first task is to install mfsmetaloggers on a few machines, all that needs
to be done is to install the mfs-master package for your distribution and ensure
that the mfsmetalogger service is running.

By default the mfsmetaloggers will discover the master and begin maintaining
active backups of the metadata.

As of MooseFS 1.6.19 the mfsmetaloggers work flawlessly, and are completely up
to date on all transactions.

When setting up the metaloggers remember that sending metadata over the network
in real time can cause load on the network, only maintain a few metaloggers. The
number of metaloggers you choose to setup should reflect the size of your
installation.

Step Two, Setup Ucarp
---------------------

Ucarp operates by creating a secondary ip address for a given interface and
then communicating via a network heartbeat with other ucarp daemons. When the
active ip interface goes down the backups come online and execute a startup
script.

This ucarp setup uses 4 scripts, the first is just a single line command to
start Ucarp and link into the remaining scripts:

.Ucarp Startup
[source, bash]
----
#!/bin/bash

ucarp -i storage -s eth0 -v 10 -p secret -a 172.16.0.99 -u /usr/share/ucarp/vip-up -d /usr/share/ucarp/vip-down -B -z
----

You will need to modify this script for your environment, the option after the
-s flag is the network interface to attach the ucarp ip address, the option
after the -a flag is to specify what the ip address to use share should be,
this is the address that the mfsmaster hostname needs to resolve to.

The -u and -d flags need to be followed by the paths to scripts which are used
to bring the network interface up and down respectively.

Next the vip-up script which is used to initialize the network interface and
execute the script which prepares the metadata and starts the mfsmaster.

The setup script needs to be executed in the background for reasons which will
be explained shortly:

.Vip-up script
[source, bash]
----
#!/bin/bash
exec 2> /dev/null

ip addr add "$2"/16 dev "$1"
/usr/share/ucarp/setup.sh &
exit 0
----

The vip-down script is almost identical but without calling the setup script:

.Vip-diwn script
[source, bash]
----
#! /bin/sh
exec 2> /dev/null

ip addr add "$2"/16 dev "$1"
----

Make sure to change the network mask to reflect your own deployment.

The Setup Script
----------------

In the previous section a setup script was referenced, this script is where the
real work is, everything before this has been routine ucarp.

In the vip-up script the setup script is called in the background; this is
because ucarp will hold onto the ip address until the script has exited. This
is unnecessary if there is only one failover machine, but since a file system is
a very important thing, it is wise to set up more than one failover interface.

.Setup script
[source, bash]
----
#!/bin/bash
MFS='/var/lib/mfs'
sleep 3

if ip a s eth0 | grep 'inet 172.16.0.99'
then
    mkdir -p $MFS/{bak,tmp}
    mv $MFS/changelog.* $MFS/metadata.* $MFS/tmp/

    service mfsmetalogger stop
    mfsmetarestore -a 
    
    if [ -e $MFS/metadata.mfs ]
    then
        cp -p $MFS/sessions_ml.mfs $MFS/sessions.mfs
        service mfsmaster start
        service mfscgiserv start
        service mfsmetalogger start
    else
        kill $(pidof ucarp)
    fi
    tar cvaf $MFS/bak/metabak.$(date +%s).tlz $MFS/tmp/*
    rm -rf $MFS/tmp
fi
----

The script starts by sleeping for 3 seconds, this is just long enough to wait
for all of the ucarp nodes that started up to finish arguing about who gets to
hold the ip address and then the script discovers if this is the new master or
not.

The interface named in the ucarp startup script is checked to see if it was the
winner, if so first move any possible information out of the way that may be
from a previous stint as the mfsmaster, this information will prevent the
mfsmetaresore command from creating the right metadata file.

Since the mfsmaster is down, the mfsmetalogger is not gathering any data, shut
it down and run mfsmetarestore -a to build the metadata file from the
metalogger information. There is a chance that the mfsmetaresore will fail, if
this is the case the metadata file will not be created.  If the metadata file
was not successfully created the ucarp interface gets killed and another
failover machine takes over.

Once it has been verified that the fresh metadata is ready, fire up the
mfsmaster.

Finally, with the new mfsmaster running tar up the metadata that was moved
before the procedure happened, we don't want to delete metadata unnecessarily.

Conclusion
----------

Place this setup on all of the machines you want running in your failover
cluster. Fire it all up and one of the machines will take over. At the best
of times this failover will take about 7 seconds, at the worst of times it 
will take 30-40 seconds. While the mfsmaster is down the client mounts will 
hang on IO operations, but they should all come back to life when the
failover completes.

I have tested this setup on an Ubuntu install and an ArchLinux install of
MooseFS, so far the better performance and reliability has been on ArchLinux,
although the difference has been generally nominal. This setup is distribution
agnostic and should work on any unix style system that supports ucarp and
MooseFS.



-Thomas S Hatch

[Moosefs-users] invalid copies on a chunkserver

[Jun C. P. <jun...@gm...>]

Hi,

While testing several things (e.g., multiple directories are exported
on the same physical hdd), I got the following status on the master.

# tail /var/log/messages (on the mfs master server)
Dec 23 21:07:28 ut1-qmb-rd1a mfsmaster[22409]: chunk 0000000000000BDF
has only invalid copies (1) - please repair it manually
Dec 23 21:07:28 ut1-qmb-rd1a mfsmaster[22409]: chunk
0000000000000BDF_00000002 - invalid copy on (10.1.1.66 - ver:00000001)
Dec 23 21:07:45 ut1-qmb-rd1a mfsmaster[22409]: chunk 0000000000000C80
has only invalid copies (1) - please repair it manually
Dec 23 21:07:45 ut1-qmb-rd1a mfsmaster[22409]: chunk
0000000000000C80_00000003 - invalid copy on (10.1.1.66 - ver:00000002)
Dec 23 21:07:54 ut1-qmb-rd1a mfsmaster[22409]: chunk 00000000000011AA
has only invalid copies (1) - please repair it manually
Dec 23 21:07:54 ut1-qmb-rd1a mfsmaster[22409]: chunk
00000000000011AA_00000003 - invalid copy on (10.1.1.66 - ver:00000001)

Interestingly, I could not find the reported problematic chunk files
on the chunkserver (10.1.1.66) as opposed to what the error message
said. Now I am using only one directory on the same physical hard
drive, though.

How can I manually fix this?

Thanks,

-Jun