moosefs-users — Mailing list for MooseFS Users

Re: [Moosefs-users] mmap file on moosefs error

[youngcow <you...@gm...>]

Hi
   Thanks. rpm Use BDB Library and error happened on bdb library.
Do you have any workaround for it?

> Hi
>
> We made some tests with mmap some time ago and for the moment FUSE on Linux
> doesn't support flag MAP_SHARED - it returns ENODEV error then. Only private
> mappings are supported (MAP_PRIVATE).
>
> Probably rpm or BerkeleyDB (library responsible for rpm database) tries to
> run mmap with this flag.
>
>
> Kind regards
> Michał Borychowski
> MooseFS Support Manager
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> Gemius S.A.
> ul. Wołoska 7, 02-672 Warszawa
> Budynek MARS, klatka D
> Tel.: +4822 874-41-00
> Fax : +4822 874-41-01
>
>
> -----Original Message-----
> From: youngcow [mailto:you...@gm...]
> Sent: Saturday, July 02, 2011 11:40 AM
> To: moosefs-users
> Subject: [Moosefs-users] mmap file on moosefs error
>
> Hi,
>    I found a error when I used moosefs. I used openvz(container-based
> virtualization for Linux) on RHEL 5.6 x86_64
> and put all container file into moosefs and I notice an error when I was
> running "rpm -qa " in vm:
>
> 	rpmdb: mmap: No such device
> 	error: db4 error(19) from dbenv->open: No such device
> 	error: cannot open Packages index using db3 - No such device (19)
> 	error: cannot open Packages database in /var/lib/rpm
>
> So I tested it in physical machine:
> I put the directory "/var/lib/rpm" into moosefs, link the dir to
> /var/lib and run "rpm -qa",the error message also:
> 	rpmdb: mmap: No such device
> 	error: db4 error(19) from dbenv->open: No such device
> 	error: cannot open Packages index using db3 - No such device
> 	error: cannot open Packages database in /var/lib/rpm
>
> Anyone knows the reason. Is it a bug of moosefs?
>
> Thanks.
>
> ----------------------------------------------------------------------------
> --
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>

Re: [Moosefs-users] Speed-up the server removal process

[Michal B. <mic...@ge...>]

Hi Stas!

 

Have you done restart of the master or only reload? Reload is not enough,
you need to restart it so that the new limits are set.

 

Marking a disk for removal doesn't stop updating the data. But you need to
remember that a table in CGI monitor has two "states" - one with all disks
and the other without disks marked for removal. Both tables are updated on
the fly.

 

 

Regards

-Michal

 

 

From: Stas Oskin [mailto:sta...@gm...] 
Sent: Sunday, July 03, 2011 12:16 AM
To: rxknhe
Cc: moosefs-users
Subject: Re: [Moosefs-users] Speed-up the server removal process

 

Hi.

 

I changed the params to recommended ones in FAQ, but the speed still slow.

 

What other values I can try, and what else can be done to increase the
migration speed.

 

Thanks.

On Sat, Jul 2, 2011 at 5:39 AM, Stas Oskin <sta...@gm...> wrote:

Hi.

Thanks for the hint, will give a look.

Regards.

 

On Fri, Jul 1, 2011 at 2:36 AM, rxknhe <rx...@gm...> wrote:

http://www.moosefs.org/moosefs-faq.html#rebalancing-speed

change read/write replication parameters. You need to restart master server
process after change in mfsmaster.cfg file.

 

On Thu, Jun 30, 2011 at 7:24 PM, Stas Oskin <sta...@gm...> wrote:

Hi.

We want to de-commission a server holding 4TB of data, and have set it all
it's disk to remove mode.

The problem is that it's happen too slow, with average read times of 20 - 30
Mbps. Based on this speed, it will take about a week before this server will
be done.

Is there any way to speed-up the process?

Thanks.

 

----------------------------------------------------------------------------
--
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

 

 

 

Re: [Moosefs-users] mmap file on moosefs error

[Michal B. <mic...@ge...>]

Hi

We made some tests with mmap some time ago and for the moment FUSE on Linux
doesn't support flag MAP_SHARED - it returns ENODEV error then. Only private
mappings are supported (MAP_PRIVATE).

Probably rpm or BerkeleyDB (library responsible for rpm database) tries to
run mmap with this flag.


Kind regards
Michał Borychowski 
MooseFS Support Manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Gemius S.A.
ul. Wołoska 7, 02-672 Warszawa
Budynek MARS, klatka D
Tel.: +4822 874-41-00
Fax : +4822 874-41-01


-----Original Message-----
From: youngcow [mailto:you...@gm...] 
Sent: Saturday, July 02, 2011 11:40 AM
To: moosefs-users
Subject: [Moosefs-users] mmap file on moosefs error

Hi,
  I found a error when I used moosefs. I used openvz(container-based
virtualization for Linux) on RHEL 5.6 x86_64
and put all container file into moosefs and I notice an error when I was
running "rpm -qa " in vm:

	rpmdb: mmap: No such device
	error: db4 error(19) from dbenv->open: No such device
	error: cannot open Packages index using db3 - No such device (19)
	error: cannot open Packages database in /var/lib/rpm

So I tested it in physical machine:
I put the directory "/var/lib/rpm" into moosefs, link the dir to
/var/lib and run "rpm -qa",the error message also:
	rpmdb: mmap: No such device
	error: db4 error(19) from dbenv->open: No such device
	error: cannot open Packages index using db3 - No such device
	error: cannot open Packages database in /var/lib/rpm

Anyone knows the reason. Is it a bug of moosefs?

Thanks.

----------------------------------------------------------------------------
--
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security 
threats, fraudulent activity, and more. Splunk takes this data and makes 
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] About MFS for SNAPSHOST question

[Michal B. <mic...@ge...>]

Hi!

 

We are not sure what problem you have in 1. It may happen under some conditions that “mfsappendchunks” would work exactly the same as “mfsmakesnapshot”.

 

In the second problem operation “read” talking to the master server gets broken. How often do you get this error? Is it regular or visible only while doing certain operations?

 

 

Kind regards

Michał Borychowski 

MooseFS Support Manager

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Gemius S.A.

ul. Wołoska 7, 02-672 Warszawa

Budynek MARS, klatka D

Tel.: +4822 874-41-00

Fax : +4822 874-41-01

 

From: 周韬 [mailto:zho...@16...] 
Sent: Thursday, June 30, 2011 7:58 AM
To: moo...@li...
Subject: [Moosefs-users] About MFS for SNAPSHOST question

 

Hello wirter,I'm teddy! I study MFS now,have some question want to ask you!

MooseFS  version:1.6.20-2

Fuse version:8.5

All operating system are CentOS 5.5_64

All file system are ext3

1、When I use snapshost and take a snapshost,the snapshost size is same as source directory or file,next is my step:

[root@localhost floder]# du -sh u.img 
20M     u.img
[root@localhost floder]# /usr/local/mfs/bin/mfsappendchunks u.img.bak u.img 
[root@localhost floder]# ll
total 2114278
-rw-r--r-- 1 root root     654246 Jun 30 01:14 mfs-1.6.20-2.tar.gz
-rw-r--r-- 1 root root  524288000 Jun 30 02:27 teddy500M
-rw-r--r-- 1 root root 1073741824 Jun 30 02:18 test1G
-rw-r--r-- 1 root root  524288000 Jun 30 02:19 test500M
-rw-r--r-- 1 root root   20971520 Jun 29 10:13 u.img
-rw-r--r-- 1 root root   20971520 Jun 30 18:57 u.img.bak
-rw-r--r-- 1 root root     105311 Jun 30 01:14 zlib-devel-1.2.3-4.el5.x86_64.rpm
[root@localhost floder]# du -sh u.img.bak 
20M     u.img.bak

use the mfsmakesnapshost result is same the mfsappendchunks!

2、When i mount at the client,have some wong messages at the system log,what is the mean?but i can use as normal;

[root@localhost floder]# tail -f /var/log/messages 
Jun 30 02:21:37 localhost avahi-daemon[2747]: Server startup complete. Host name is localhost-6.local. Local service cookie is 475828928.
Jun 30 02:22:16 localhost mfsmount[2864]: master: tcp recv error: EINTR (Interrupted system call) (1)
Jun 30 02:22:16 localhost mfsmount[2864]: registered to master
Jun 30 02:22:16 localhost mfsmount[2864]: master: tcp recv error: EINTR (Interrupted system call) (1)
Jun 30 02:22:16 localhost mfsmount[2864]: registered to master
Jun 30 02:22:37 localhost mfsmount[2875]: master: tcp recv error: EINTR (Interrupted system call) (1)
Jun 30 02:22:37 localhost mfsmount[2875]: registered to master
Jun 30 02:22:37 localhost mfsmount[2875]: master: tcp recv error: EINTR (Interrupted system call) (1)
Jun 30 02:22:37 localhost mfsmount[2875]: registered to master

Jun 30 02:33:33 localhost restorecond: Will not restore a file with more than one hard link (/etc/resolv.conf) Invalid argument

 

waitting for you answer on line!!!

Best wish for you!

 

 

 

Re: [Moosefs-users] Write starvation

[Michal B. <mic...@ge...>]

Hi Robert!

Parallel reading and writing to the same file indeed causes some write
delays. If it is not your case, adding 1-2 chunkservers and having a
dedicated machine for the master should help.


Kind regards
Michał Borychowski 
MooseFS Support Manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Gemius S.A.
ul. Wołoska 7, 02-672 Warszawa
Budynek MARS, klatka D
Tel.: +4822 874-41-00
Fax : +4822 874-41-01

 

-----Original Message-----
From: Robert Sandilands [mailto:rsa...@ne...] 
Sent: Wednesday, June 29, 2011 4:30 AM
To: moo...@li...
Subject: [Moosefs-users] Write starvation

I have been moving data from existing non-distributed file systems onto 
a MooseFS file system.

I am using 1.6.20 on Centos 5.6.

While moving the data I have also transfered some of the normal 
read-only traffic load to use the data already moved onto the MFS volume.

What I can see is when there is any significant read traffic then write 
traffic slows down to a crawl.

When I look at the server charts for any of the chunk servers generated 
by mfscgiserv then it seems like read and write traffic seems to alternate.

Write traffic does not stop completely, but seems to slow down to < 10 
kB per second under high read traffic conditions. When the read traffic 
decreases the write traffic will increase to normal levels.

Is this a known problem? Is there something I can do to ensure that 
write traffic is not starved by read traffic?

Robert





----------------------------------------------------------------------------
--
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security 
threats, fraudulent activity, and more. Splunk takes this data and makes 
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] Speed-up the server removal process

[Stas O. <sta...@gm...>]

Hi.

I changed the params to recommended ones in FAQ, but the speed still slow.

What other values I can try, and what else can be done to increase the
migration speed.

Thanks.

On Sat, Jul 2, 2011 at 5:39 AM, Stas Oskin <sta...@gm...> wrote:

> Hi.
>
> Thanks for the hint, will give a look.
>
> Regards.
>
>
> On Fri, Jul 1, 2011 at 2:36 AM, rxknhe <rx...@gm...> wrote:
>
>> http://www.moosefs.org/moosefs-faq.html#rebalancing-speed
>> change read/write replication parameters. You need to restart master
>> server process after change in mfsmaster.cfg file.
>>
>>
>> On Thu, Jun 30, 2011 at 7:24 PM, Stas Oskin <sta...@gm...> wrote:
>>
>>> Hi.
>>>
>>> We want to de-commission a server holding 4TB of data, and have set it
>>> all it's disk to remove mode.
>>>
>>> The problem is that it's happen too slow, with average read times of 20 -
>>> 30 Mbps. Based on this speed, it will take about a week before this server
>>> will be done.
>>>
>>> Is there any way to speed-up the process?
>>>
>>> Thanks.
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> All of the data generated in your IT infrastructure is seriously
>>> valuable.
>>> Why? It contains a definitive record of application performance, security
>>> threats, fraudulent activity, and more. Splunk takes this data and makes
>>> sense of it. IT sense. And common sense.
>>> http://p.sf.net/sfu/splunk-d2d-c2
>>> _______________________________________________
>>> moosefs-users mailing list
>>> moo...@li...
>>> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>>>
>>>
>>
>

[Moosefs-users] mmap file on moosefs error

[youngcow <you...@gm...>]

Hi,
  I found a error when I used moosefs. I used openvz(container-based
virtualization for Linux) on RHEL 5.6 x86_64
and put all container file into moosefs and I notice an error when I was
running "rpm -qa " in vm:

	rpmdb: mmap: No such device
	error: db4 error(19) from dbenv->open: No such device
	error: cannot open Packages index using db3 - No such device (19)
	error: cannot open Packages database in /var/lib/rpm

So I tested it in physical machine:
I put the directory "/var/lib/rpm" into moosefs, link the dir to
/var/lib and run "rpm -qa",the error message also:
	rpmdb: mmap: No such device
	error: db4 error(19) from dbenv->open: No such device
	error: cannot open Packages index using db3 - No such device
	error: cannot open Packages database in /var/lib/rpm

Anyone knows the reason. Is it a bug of moosefs?

Thanks.

Re: [Moosefs-users] Data migration stop

[Stas O. <sta...@gm...>]

Update - it actually seems as orange stats are lowering, but there is no
data in disks speed.

Perhaps marking a disk removal, causes it to stop updating the statistics
data?

On Sat, Jul 2, 2011 at 6:17 AM, Stas Oskin <sta...@gm...> wrote:

> Hi.
>
> After several days of migrating data off server planned to be removed, I
> see now that the migration has stopped in the CGI monitor.
>
> There is no activity what- soever on the server disks, but there is about
> of 80% of data left.
>
> How it possible to verify if this an issue or a normal behavior?
>
> Regards.
>

[Moosefs-users] Data migration stop

[Stas O. <sta...@gm...>]

Hi.

After several days of migrating data off server planned to be removed, I see
now that the migration has stopped in the CGI monitor.

There is no activity what- soever on the server disks, but there is about of
80% of data left.

How it possible to verify if this an issue or a normal behavior?

Regards.

Re: [Moosefs-users] Write starvation

[Robert S. <rsa...@ne...>]

Based on some tests I think the limit in this case is the number of 
opens per minute. I think I need to understand what happens with an open 
before I can make guesses on what can be done to get the number higher.

But then it still does not quite explain the write starvation except if 
the number of pending reads are just so much higher than the number of 
pending writes that it seems to starve the writes. Maybe this will 
resolve itself as I add more chunk servers.

Some questions:

1. Is there a limit to the number of handles that client applications 
can open per mount, per chunk server, per disk?
2. What happens when an application does fopen() on a mount? Can 
somebody give a quick overview or do I have to read some code?

Robert

On 6/30/11 11:32 AM, Ricardo J. Barberis wrote:
> El Miércoles 29 Junio 2011, Robert escribió:
>> Yes, we use Centos, but installing and using the ktune package generally
>> resolves most of the performance issues and differences I have seen with
>> Ubuntu/Debian.
> Nice to know about ktune and thank you for bringing it up, I'll take a look a
> it.
>
>> I don't understand the comment on hitting metadata a lot? What is a lot?
> A lot = reading / (re)writing / ls -l'ing / stat'ing too often.
>
> If the client can't cache the metadata but uses it often, that means it has to
> query the master every time.
>
> Network latencies might also play a role in the performance degradation.
>
>> Why would it make a difference? All the metadata is in RAM anyway? The
>> biggest limit to speed seems to be the number of IOPS that you can get out
>> of your disks you have available to you. Looking up the metadata from RAM
>> should be several orders of magnitude faster than that.
> Yep, and you have plenty of RAM, so that shouldn't be an issue in your case.
>
>> The activity reported through the CGI interface on the master is around
>> 2,400 opens per minute average. Reads and writes are also around 2400 per
>> minute alternating with each other. mknod has some peaks around 2,800 per
>> minute but is generally much lower. Lookup's are around 8,000 per minute
>> and getattr is around 700 per minute. Chunk replication and deletion is
>> around 50 per minute. The other numbers are generally very low.
> Mmm, maybe 2 chunkservers are just too litle to handle that activity but I
> would also check the network latencies.
>
> I'm also not really confident about having master and cunkserver on the same
> server but I don't have any hard evidence to support my feelings ;)
>
>> Is there a guide/hints specific to MooseFS on what IO/Net/Process
>> parameters would be good to investigate for mfsmaster?
> I'd like to know that too!
>
> Cheers,

Re: [Moosefs-users] Speed-up the server removal process

[rxknhe <rx...@gm...>]

http://www.moosefs.org/moosefs-faq.html#rebalancing-speed
change read/write replication parameters. You need to restart master server
process after change in mfsmaster.cfg file.


On Thu, Jun 30, 2011 at 7:24 PM, Stas Oskin <sta...@gm...> wrote:

> Hi.
>
> We want to de-commission a server holding 4TB of data, and have set it all
> it's disk to remove mode.
>
> The problem is that it's happen too slow, with average read times of 20 -
> 30 Mbps. Based on this speed, it will take about a week before this server
> will be done.
>
> Is there any way to speed-up the process?
>
> Thanks.
>
>
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>

[Moosefs-users] Speed-up the server removal process

[Stas O. <sta...@gm...>]

Hi.

We want to de-commission a server holding 4TB of data, and have set it all
it's disk to remove mode.

The problem is that it's happen too slow, with average read times of 20 - 30
Mbps. Based on this speed, it will take about a week before this server will
be done.

Is there any way to speed-up the process?

Thanks.

Re: [Moosefs-users] Questions and comparison

[rxknhe <rx...@gm...>]

Few drawbacks we found when testing glusterfs:
 - In our glusterfs setup, we found meta-data access was very slow. This is
 important for small file operations such as source code repository, used by
developers (svn, compiling code etc).
   In case of glusterfs, meta data is distributed and not a single point of
failure like in MooseFS. Nice feature and one of the reason why we selected
glusterFS over MooseFS in the first place. However, we decided to switch to
MooseFS later because, MooseFS offers fast meta data access and few other
reasons where glsuterfs wasn't a good fit for us.

 - glusterfs cluster totally stalled one day, when one of the disk in a
server stopped working. We had to shutdown that chunk server(gluster brick)
to bring  the whole gluster system back. It could be some tuning/config
issue that I could not deny, but  I was rather expecting a better fault
tolerance from a product.

 - glusterfs stores a complete file on a data server and replicate whole
file between data(chunk) servers. This could be slow process when writing
large file on a single disk@server and by using single disk spindle(assuming
raid is not used). MooseFS divide up files in chunks (up to 64MB max size)
and distribute across servers. So better overall i/o expected.

 - Setting up MooseFS is extremely simple,  maintenance and admin is very
easy. Overall we found MooseFS as a very smart and efficient design from
operations perspective. This white paper may also to know more about MooseFS
from operations perspective: http://contrib.meharwal.com/home/moosefs

 - Fuse vs kernel module: Although purist may argue about i/o speed between
two approaches, but as Ricardo mentioned that fuse is much straight forward
development path used by many. It is fairly beaten up and polished product.
So why re-invent another wheel. In most practical situations, speed of fuse
mounts may just be sufficient as we found in our environment.


regds,
rxknhe

On Thu, Jun 30, 2011 at 12:34 PM, Ricardo J. Barberis <
ric...@da...> wrote:

> El Miércoles 29 Junio 2011, Sébastien Morand escribió:
> > Hi,
> >
> > I'm currently interesting in the deployment of a distribuated filesystem
> > and read a paper about moosefs. I have a few questions before starting
> the
> > job:
>
> Following answers are based on personal experience, YMMV.
>
> > 1/ Why moosefs instead of glusterfs or xtreemfs?
>
> I had some reliability problems with gluster when I tested it (versions
> 2.0.8
> and some of the earlier 3.0.x). It has improved since, but I already bet on
> Moose :)
>
> XtreemFS seemed more oriented to replications via WAN, I didn't even test
> it.
>
> > 2/ glusterfs has been described in a paper I read from a french
> university
> > as really faster than moosefs, did you benchmark them too?
>
> I micro-benchmarked the versions mentioned above, for my use case Moose was
> faster than Gluster by a minimal margin.
>
> I also tested Lustre, which had even better performance but doesn't provide
> fault tolerance out of the box.
>
> > 3/ Why fuse and no kernel mode (should be faster)?
>
> There was recently a thread on lkml about this, with divided opinions:
>
>  http://thread.gmane.org/gmane.linux.kernel/1148926/
>
> Kernel-based filesystems are usually faster but more complicated to develop
> and deploy. For example, Lustre provides a kernel module for some Linux
> distros but if you don't use one of those you have to compile your own.
>
> Fuse-based are easier to develop, debug and deploy but usually not as fast
> as
> a kernel-based one.
>
> > Thanks by advance,
> > Sébastien
>
> Regards,
> --
> Ricardo J. Barberis
> Senior SysAdmin / ITI
> Dattatec.com :: Soluciones de Web Hosting
> Tu Hosting hecho Simple!
>
>
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>

Re: [Moosefs-users] Questions and comparison

[WK <wk...@bn...>]

On 6/29/2011 11:35 AM, Sébastien Morand wrote:
>
> 1/ Why moosefs instead of glusterfs or xtreemfs?

I played with GlusterFS for the past two years and lurked on their 
mailing lists. We saw lots of little issues and when we did get it 
figured out, an upgrade would create a new set of problems. Note we were 
trying to accomplish replication in addition to storage, so we had more 
translators involved, than a simple layout.

The GlusterFS devs and community are really nice and talented, but I 
think they have created a situation where they are trying to accomplish 
too many things at once. You should look at the GlusterFS mail list 
archives. The recent introduction of 3.2.0 was yet again a big change 
that broke things and once again many people had to downgrade back.

Eventually, the GlusterFS people will get it right. They seem to have a 
lot of resources and an enthusiastic community.

I don't have any experience with XtreemFS


>
> 2/ glusterfs has been described in a paper I read from a french 
> university as really faster than moosefs, did you benchmark them too?
>

I did some crude benchmarks using dd and they were equivalent.  However, 
in actual practice, for our workload MooseFS is faster. That may simply 
be that it does a better job of caching, but the users are MUCH happier 
and the admin is much easier.

Jeff Darcy works for RedHat and is developing CloudFS which is based on 
GlusterFS. He is a huge Gluster fan and is active in the Gluster 
community.  His experience was that MooseFS was faster.

 From http://cloudfs.org/2011/02/

"A less well-known project is MooseFS. It’s broadly similar to GlusterFS 
in terms of being FUSE-based and having built-in replication, but 
diverges in other ways. Their “metadata logger” approach to surviving 
metadata-server death is not quite as good as true distributed metadata 
IMO, but it’s still far better than some systems’ “one metadata server 
and if it dies or slows to a crawl then SUX2BU” approach. Built-in 
snapshots are a feature that really sets them apart. I ran it through 
some very basic tests on some of my machines, and it fared very well. I 
don’t mind saying that it was significantly better than GlusterFS for 
the workloads I tested, so it’s worth checking out."

So I think it depends upon the workload and the configuration.

> 3/ Why fuse and no kernel mode (should be faster)?
>

Thats for the devs to answer, but I would imagine Fuse is much easier to 
maintain and develop for.

WK

Re: [Moosefs-users] Questions and comparison

[Ricardo J. B. <ric...@da...>]

El Miércoles 29 Junio 2011, Sébastien Morand escribió:
> Hi,
>
> I'm currently interesting in the deployment of a distribuated filesystem
> and read a paper about moosefs. I have a few questions before starting the
> job:

Following answers are based on personal experience, YMMV.

> 1/ Why moosefs instead of glusterfs or xtreemfs?

I had some reliability problems with gluster when I tested it (versions 2.0.8 
and some of the earlier 3.0.x). It has improved since, but I already bet on 
Moose :)

XtreemFS seemed more oriented to replications via WAN, I didn't even test it.

> 2/ glusterfs has been described in a paper I read from a french university
> as really faster than moosefs, did you benchmark them too?

I micro-benchmarked the versions mentioned above, for my use case Moose was 
faster than Gluster by a minimal margin.

I also tested Lustre, which had even better performance but doesn't provide 
fault tolerance out of the box.

> 3/ Why fuse and no kernel mode (should be faster)?

There was recently a thread on lkml about this, with divided opinions:

 http://thread.gmane.org/gmane.linux.kernel/1148926/

Kernel-based filesystems are usually faster but more complicated to develop 
and deploy. For example, Lustre provides a kernel module for some Linux 
distros but if you don't use one of those you have to compile your own.

Fuse-based are easier to develop, debug and deploy but usually not as fast as 
a kernel-based one.

> Thanks by advance,
> Sébastien

Regards,
-- 
Ricardo J. Barberis
Senior SysAdmin / ITI
Dattatec.com :: Soluciones de Web Hosting
Tu Hosting hecho Simple!

[Moosefs-users] We need your help! Mastertools brainstorming

[jyc <mai...@gm...>]

i found back that you asked about new command line that would be useful :

1/ a way to know to wich file belong one chunk.
i had a broken server that gave me many bad crc. and i had to wait  the 
log tell me which file was broken with a bad crc chunk.

2/ a way to re verify a crc on a chunk, or to force the check of all the 
chunks on a chunkserver.
as for now, i changed my server, i would like to know in which state are 
the chunks on my hard-drives...

thanks for all your great job on moosefs

jyc

Re: [Moosefs-users] Retiring a chunkserver?

[Ricardo J. B. <ric...@da...>]

El Jue 30 Junio 2011, Stas Oskin escribió:
> Hi.
>
> Thanks everyone, I found that in FAQ docs (should have looked at that
> before posting :) ).
>
> That said, if there would be some command (mfsretiredisk/cs or something),
> that would be great.

A few days ago, Michał asked for suggestions for new commands for the next 
version of mfstools.

I can't find the thread right now, but this could be a good candidate for a 
new command :)

Regards,
-- 
Ricardo J. Barberis
Senior SysAdmin / ITI
Dattatec.com :: Soluciones de Web Hosting
Tu Hosting hecho Simple!

Re: [Moosefs-users] Write starvation

[Ricardo J. B. <ric...@da...>]

El Miércoles 29 Junio 2011, Robert escribió:
> Yes, we use Centos, but installing and using the ktune package generally
> resolves most of the performance issues and differences I have seen with
> Ubuntu/Debian.

Nice to know about ktune and thank you for bringing it up, I'll take a look a 
it.

> I don't understand the comment on hitting metadata a lot? What is a lot?

A lot = reading / (re)writing / ls -l'ing / stat'ing too often.

If the client can't cache the metadata but uses it often, that means it has to 
query the master every time.

Network latencies might also play a role in the performance degradation.

> Why would it make a difference? All the metadata is in RAM anyway? The
> biggest limit to speed seems to be the number of IOPS that you can get out
> of your disks you have available to you. Looking up the metadata from RAM
> should be several orders of magnitude faster than that.

Yep, and you have plenty of RAM, so that shouldn't be an issue in your case.

> The activity reported through the CGI interface on the master is around
> 2,400 opens per minute average. Reads and writes are also around 2400 per
> minute alternating with each other. mknod has some peaks around 2,800 per
> minute but is generally much lower. Lookup's are around 8,000 per minute
> and getattr is around 700 per minute. Chunk replication and deletion is
> around 50 per minute. The other numbers are generally very low.

Mmm, maybe 2 chunkservers are just too litle to handle that activity but I 
would also check the network latencies.

I'm also not really confident about having master and cunkserver on the same 
server but I don't have any hard evidence to support my feelings ;)

> Is there a guide/hints specific to MooseFS on what IO/Net/Process
> parameters would be good to investigate for mfsmaster?

I'd like to know that too!

Cheers,
-- 
Ricardo J. Barberis
Senior SysAdmin / ITI
Dattatec.com :: Soluciones de Web Hosting
Tu Hosting hecho Simple!

Re: [Moosefs-users] Retiring a chunkserver?

[Stas O. <sta...@gm...>]

Hi.

Thanks everyone, I found that in FAQ docs (should have looked at that before
posting :) ).

That said, if there would be some command (mfsretiredisk/cs or something),
that would be great.

Regards.

On Wed, Jun 29, 2011 at 4:43 PM, Laurent Wandrebeck <lw...@hy...> wrote:

> On Wed, 29 Jun 2011 16:24:37 +0300
> Stas Oskin <sta...@gm...> wrote:
>
> > Hi.
> >
> > What is the best way to retire a chunkserver?
> >
> > Is there a command that can do this? Or I just can stop mfschunkserver,
> and
> > let MFS resync the chunks to other chunkservers in cluster?
> best thing is to add a * in front of each mount point used by mfs on
> the chunkserver. Restart the CS, and let the replication do the job,
> then retire the CS. That way, you'll never be in a situation where goal
> is lower than it should be.
> HTH,
> --
> Laurent Wandrebeck
> HYGEOS, Earth Observation Department / Observation de la Terre
> Euratechnologies
> 165 Avenue de Bretagne
> 59000 Lille, France
> tel: +33 3 20 08 24 98
> http://www.hygeos.com
> GPG fingerprint/Empreinte GPG: F5CA 37A4 6D03 A90C 7A1D  2A62 54E6 EF2C
> D17C F64C
>
>
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>

Re: [Moosefs-users] geographical fun with MooseFS

[Florent B. <fl...@co...>]

Hi Michal,

Could we have an idea of when will be released the next public version 
and what improvements will be included ?

Thank you a lot for your work.

Florent

Le 30/06/2011 10:24, Michal Borychowski a écrit :
> Hi!
>
> This could be quite difficult to keep "distances" in IP addresses. We
> introduced rack maps which should be easier in the mainenance. It would be
> soon available in the public version.
>
>
> Regards
> Kind regards
> Michał Borychowski
> MooseFS Support Manager
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> Gemius S.A.
> ul. Wołoska 7, 02-672 Warszawa
> Budynek MARS, klatka D
> Tel.: +4822 874-41-00
> Fax : +4822 874-41-01
>
>
> -----Original Message-----
> From: Mike [mailto:isp...@gm...]
> Sent: Tuesday, June 21, 2011 5:02 AM
> To: moo...@li...
> Subject: [Moosefs-users] geographical fun with MooseFS
>
>
> What if someone wrote a patch for MooseFS that looked at the IP of the
> client, and the IP of the chunk servers that had the chunk the client
> wanted, and tried to pick the closest one?
>
> something like
>
>     client = 10.1.1.1
>     chunkserver with copy#1 = 10.1.1.2
>     chunkserver with copy#2 = 10.1.1.20
>     chunkserver with copy#3 = 10.1.2.2
>
> Those 3 chunk servers would be used in that order, since their IPs are
> closer to the client's IP (using a formula like a*256^3+b*256^2+c*256+d
> to calculate an integer? long int? based on an IP address).
>
> This way you could have "close" chunk servers respond to most of the
> requests from a client, but if no "close" server had the chunk you
> wanted, you could go to a "distant" one.
>
> Drop two IPs on the client's interface, and do some careful numbering,
> and you can even set preference on a machine on the same LAN.
>
> This might make for a really simple way to do "distant" archives that
> don't get used for reads unless they are the only source that's
> available, and other similar problems.
>
> writes would be a different problem.
>
> Thoughts? comments?
>
> ----------------------------------------------------------------------------
> --
> EditLive Enterprise is the world's most technically advanced content
> authoring tool. Experience the power of Track Changes, Inline Image
> Editing and ensure content is compliant with Accessibility Checking.
> http://p.sf.net/sfu/ephox-dev2dev
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] geographical fun with MooseFS

[Michal B. <mic...@ge...>]

Hi!

This could be quite difficult to keep "distances" in IP addresses. We
introduced rack maps which should be easier in the mainenance. It would be
soon available in the public version.


Regards
Kind regards
Michał Borychowski 
MooseFS Support Manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Gemius S.A.
ul. Wołoska 7, 02-672 Warszawa
Budynek MARS, klatka D
Tel.: +4822 874-41-00
Fax : +4822 874-41-01


-----Original Message-----
From: Mike [mailto:isp...@gm...] 
Sent: Tuesday, June 21, 2011 5:02 AM
To: moo...@li...
Subject: [Moosefs-users] geographical fun with MooseFS


What if someone wrote a patch for MooseFS that looked at the IP of the 
client, and the IP of the chunk servers that had the chunk the client 
wanted, and tried to pick the closest one?

something like

   client = 10.1.1.1
   chunkserver with copy#1 = 10.1.1.2
   chunkserver with copy#2 = 10.1.1.20
   chunkserver with copy#3 = 10.1.2.2

Those 3 chunk servers would be used in that order, since their IPs are 
closer to the client's IP (using a formula like a*256^3+b*256^2+c*256+d 
to calculate an integer? long int? based on an IP address).

This way you could have "close" chunk servers respond to most of the 
requests from a client, but if no "close" server had the chunk you 
wanted, you could go to a "distant" one.

Drop two IPs on the client's interface, and do some careful numbering, 
and you can even set preference on a machine on the same LAN.

This might make for a really simple way to do "distant" archives that 
don't get used for reads unless they are the only source that's 
available, and other similar problems.

writes would be a different problem.

Thoughts? comments?

----------------------------------------------------------------------------
--
EditLive Enterprise is the world's most technically advanced content
authoring tool. Experience the power of Track Changes, Inline Image
Editing and ensure content is compliant with Accessibility Checking.
http://p.sf.net/sfu/ephox-dev2dev
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] mfsmakesnapshot: symlinks dereference?

[Michal B. <mic...@ge...>]

Hi Giovanni!

We put it on our todo list, but to be honest with low priority.


Kind regards
Michal

-----Original Message-----
From: Giovanni Toraldo [mailto:gt...@li...] 
Sent: Monday, June 20, 2011 2:31 PM
To: moo...@li...
Subject: [Moosefs-users] mfsmakesnapshot: symlinks dereference?

Hi,

I'm trying to make a simple backup script for my MooseFS volume using
mfsmakesnapshot. However on my volume there are many symlinks created by the
application, and I would like to copy the referenced file and not the
symlink.

It's possible to implement in mfsmakesnapshot an option like cp -L?

Thanks.

--
Giovanni Toraldo
http://www.libersoft.it/

[Moosefs-users] About MFS for SNAPSHOST question

[周韬 <zho...@16...>]

Hello wirter,I'm teddy! I study MFS now,have some question want to ask you!
MooseFS  version:1.6.20-2
Fuse version:8.5
All operating system are CentOS 5.5_64
All file system are ext3
1、When I use snapshost and take a snapshost,the snapshost size is same as source directory or file,next is my step:
[root@localhost floder]# du -sh u.img
20M     u.img
[root@localhost floder]# /usr/local/mfs/bin/mfsappendchunks u.img.bak u.img
[root@localhost floder]# ll
total 2114278
-rw-r--r-- 1 root root     654246 Jun 30 01:14 mfs-1.6.20-2.tar.gz
-rw-r--r-- 1 root root  524288000 Jun 30 02:27 teddy500M
-rw-r--r-- 1 root root 1073741824 Jun 30 02:18 test1G
-rw-r--r-- 1 root root  524288000 Jun 30 02:19 test500M
-rw-r--r-- 1 root root   20971520 Jun 29 10:13 u.img
-rw-r--r-- 1 root root   20971520 Jun 30 18:57 u.img.bak
-rw-r--r-- 1 root root     105311 Jun 30 01:14 zlib-devel-1.2.3-4.el5.x86_64.rpm
[root@localhost floder]# du -sh u.img.bak
20M     u.img.bak
use the mfsmakesnapshost result is same the mfsappendchunks!
2、When i mount at the client,have some wong messages at the system log,what is the mean?but i can use as normal;
[root@localhost floder]# tail -f /var/log/messages
Jun 30 02:21:37 localhost avahi-daemon[2747]: Server startup complete. Host name is localhost-6.local. Local service cookie is 475828928.
Jun 30 02:22:16 localhost mfsmount[2864]: master: tcp recv error: EINTR (Interrupted system call) (1)
Jun 30 02:22:16 localhost mfsmount[2864]: registered to master
Jun 30 02:22:16 localhost mfsmount[2864]: master: tcp recv error: EINTR (Interrupted system call) (1)
Jun 30 02:22:16 localhost mfsmount[2864]: registered to master
Jun 30 02:22:37 localhost mfsmount[2875]: master: tcp recv error: EINTR (Interrupted system call) (1)
Jun 30 02:22:37 localhost mfsmount[2875]: registered to master
Jun 30 02:22:37 localhost mfsmount[2875]: master: tcp recv error: EINTR (Interrupted system call) (1)
Jun 30 02:22:37 localhost mfsmount[2875]: registered to master
Jun 30 02:33:33 localhost restorecond: Will not restore a file with more than one hard link (/etc/resolv.conf) Invalid argument
 
waitting for you answer on line!!!
Best wish for you!
 
 

Re: [Moosefs-users] Write starvation

[Robert S. <rsa...@ne...>]

The one chunk server is also the master. As we free up space on other servers we will add more chunk servers. We currently only have two. The other chunk server has 36 x 2 TB drives and 64 GB of RAM.

I agree with your comments on multiple chunk servers. That is one of the major reasons why we chose MooseFS.

Robert

Incomprehensible email compliments of my iPad

On Jun 29, 2011, at 8:32 PM, WK <wk...@bn...> wrote:

> On 6/29/2011 11:29 AM, Robert wrote:
>> 
>> Yes, the master server is working hard. But I would still expect a somewhat fair distribution of load between read and write.
>> 
>> The specs:
>> 
>> 2 x quad core Xeon E5405 @ 2GHz.
>> 64 GB of RAM
>> 32 x 2 TB 7200 RPM SATA disks
>> 68 million file system objects
>> 65.4 million files
>> No swap is being used
>> mfsmaster is using 23 GB of RAM.
>> 
> 
> OK, I am sorry if this seem silly and I'm completely missing something.
> 
> But I am confused as to the above specs.
> 
> Is your Master and Chunkserver on the same machine?
> 
> or are those 32 x 2TB drives spread out on how many chunkservers?
> 
> Our experience has been the more chunkservers we have, the better the performance because the read/writes are spread out among the chunkservers  (in addition to individual spindles on each chunkserver).
> 
> WK
> 
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security 
> threats, fraudulent activity, and more. Splunk takes this data and makes 
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] Write starvation

[WK <wk...@bn...>]

On 6/29/2011 11:29 AM, Robert wrote:
> Yes, the master server is working hard. But I would still expect a 
> somewhat fair distribution of load between read and write.
>
> The specs:
>
> 2 x quad core Xeon E5405 @ 2GHz.
> 64 GB of RAM
> 32 x 2 TB 7200 RPM SATA disks
> 68 million file system objects
> 65.4 million files
> No swap is being used
> mfsmaster is using 23 GB of RAM.
>

OK, I am sorry if this seem silly and I'm completely missing something.

But I am confused as to the above specs.

Is your Master and Chunkserver on the same machine?

or are those 32 x 2TB drives spread out on how many chunkservers?

Our experience has been the more chunkservers we have, the better the 
performance because the read/writes are spread out among the 
chunkservers  (in addition to individual spindles on each chunkserver).

WK