moosefs-users — Mailing list for MooseFS Users

[Moosefs-users] My Project with MooseFS

[Sollix - A. C. <acu...@so...>]

Hello everybody, 

I'm a new french user (sorry for my bad english) of the Moose FS. I have installed Moose on a development infrastructure to test it. 

I will try to explain you what i want to do : 
I'm a reseller IT and i sell Linux server to my customer. The aim : do backup of all server. 
They are all connected to my VPN (server 100Mbits in datacenter). The Master are setup on it. 
I want to install chunkserver on all server. Near 300To will be available . 

Each server will copy data on this FS with a goal = X (for example 4). All server are connect to internet thanks ADSL . 

After some test, i have a little problem with network performance. 

Example : 
If i copy file1.txt on the MooseFS with a goal of 5, the node will sent the file to the other node to have the replication. 
So the server upload 5x the size of the file. It's not very optimized. 
I search a solution to upload the file to the VPN Server (i can setup a chunkserver on it) and only the VPN Server will deploy the file on the another chunkserver to use the big bandwidth of this server -> 100Mbits 

I hope i was clear. 
Thanks you 

Cordialement, 

Alexandre CUVELIER 

Re: [Moosefs-users] Some new FAQ entries on the website

[Jun C. P. <jun...@gm...>]

Awesome!

Clear explanations for questions/issues that I have been really wondering about.

Thanks a lot,

-Jun

2011/2/7 Michal Borychowski <mic...@ge...>:
> Hi!
>
>
>
> I've just added 11 new Frequently Asked Questions (with answers! :)) which
> were coming from you during several last months.
>
>
>
> The new part starts here:
> http://www.moosefs.org/moosefs-faq.html#file-locking. I hope they would
> clarify some points with MooseFS!
>
>
>
> And again many many thanks to Travis for his proofreading! :)
>
>
>
>
>
> Kind regards
>
> Michal
>
>
>
> MooseFS Support Manager
>
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Gemius S.A.
>
> ul. Wołoska 7, 02-672 Warszawa
>
> Budynek MARS, klatka D
>
> Tel.: +4822 874-41-00
>
> Fax : +4822 874-41-01
>
>
>
> ------------------------------------------------------------------------------
> The modern datacenter depends on network connectivity to access resources
> and provide services. The best practices for maximizing a physical server's
> connectivity to a physical network are well understood - see how these
> rules translate into the virtual world?
> http://p.sf.net/sfu/oracle-sfdevnlfb
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>

[Moosefs-users] Some new FAQ entries on the website

[Michal B. <mic...@ge...>]

Hi!

 

I've just added 11 new Frequently Asked Questions (with answers! :)) which
were coming from you during several last months.

 

The new part starts here:
http://www.moosefs.org/moosefs-faq.html#file-locking. I hope they would
clarify some points with MooseFS!

 

And again many many thanks to Travis for his proofreading! :)

 

 

Kind regards

Michal 

 

MooseFS Support Manager

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Gemius S.A.

ul. Wołoska 7, 02-672 Warszawa

Budynek MARS, klatka D

Tel.: +4822 874-41-00

Fax : +4822 874-41-01

 

[Moosefs-users] Non-persistent TCP connections (port 9422) for accessing chunkserver?

[Jun C. P. <jun...@gm...>]

Hi,

I am using KVM (kernel based virtualization machine) on MFS. Some
tools given by KVM such as qemu-img use a lot of separate I/O requests
for creating a single file. I found it seems like that, upon different
I/O requests even for the same file, MFS re-establishes many TCP
connections with all the required chunkservers. Then, it ends up that
the write performance is significantly reduced from 60M/s (with 'cp'
command for large files) to 1M/s. So this kind of non-persistent
connection management on MFS appears to be the main root cause of the
slow I/O write performance. Could somebody please confirm my guess?

Is there any way to work around it? Or is there any plan to make MFS
use persistent connections? Or even is there any reason why MFS uses
non-persistent connections for the same file requests?

Thanks,

-Jun

Re: [Moosefs-users] what's mean CHUNKS_LOOP_TIME

[Michal B. <mic...@ge...>]

This time in seconds means how long the all chunks tests should take. To be
honest you should not touch this parameter. You may slightly increase it
but it’s better not to exceed one hour (3600). 

 

Master every second tests “number of all chunks / CHUNKS_LOOP_TIME” of
chunks. For each chunk it tests if there is some operation to do (delete,
replicate). So if you increase this value, master CPU load will decrease
but at a cost of bigger latency in delete or replicate operations. 

 

It is also possible that this CHUNKS_LOOP_TIME parameter may be removed by
us completely in the future.

 

 

Kind regards

Michał Borychowski 

MooseFS Support Manager

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Gemius S.A.

ul. Wołoska 7, 02-672 Warszawa

Budynek MARS, klatka D

Tel.: +4822 874-41-00

 

 

From: 妖狐 [mailto:myt...@gm...] 
Sent: Wednesday, January 19, 2011 4:17 PM
To: moo...@li...
Subject: [Moosefs-users] what's mean CHUNKS_LOOP_TIME

 

hi all:

      CHUNKS_LOOP_TIME ：（Chunks loop frequency in seconds）chunks. what
’s mean the chunks loop ？ 

 

[Moosefs-users] MooseFS packages in Arch Linux

[Thomas S H. <tha...@gm...>]

Arch Linux makes packages prove that they are "worthy" so to speak before
adding them to the binary repositories. the MooseFS packages have passed the
test and have been moved into the main Arch Linux package repository. So now
you can install MooseFS on Arch Linux with pacman -Sy mfs-master mfs-client
mfs-chunkserver

The packages can be found at the following new links:
http://www.archlinux.org/packages/community/x86_64/mfs-client/
http://www.archlinux.org/packages/community/i686/mfs-client/
http://www.archlinux.org/packages/community/x86_64/mfs-chunkserver/
http://www.archlinux.org/packages/community/i686/mfs-chunkserver/
http://www.archlinux.org/packages/community/x86_64/mfs-master/
http://www.archlinux.org/packages/community/i686/mfs-master/

Michal, please update the download page, as the link that is there now has
been deleted

Enjoy MooseFS on Arch!

Thanks again to the MooseFS team, MooseFS is amazing!

-Thomas S Hatch

[Moosefs-users] mfsmetalogger shutdown issues

[Thomas S H. <tha...@gm...>]

I am working on the moosefs failover, and I have cleaned up a number of
bugs, but I did discover that if the mfsmetalogger files are not present
when you try to shut down the metalogger you need to pass it a kill -9. Any
way this could be changed to detect conditions which will cause the
mfsmetalogger stop to deadlock and kill it differently?

[Moosefs-users] Optimal writes for MooseFS

[Thomas S H. <tha...@gm...>]

We already know that vm image writes are not the optimal way to write to
MooseFS, and randomIO is obviously not the best, so let me ask, what it the
optimal way to write to a MooseFS filesystem? just sequentially? Is an
append to an existing file faster?

Thanks!

-Thomas S Hatch

[Moosefs-users] currently unavailable chunk and reserved files

[Giovanni T. <gt...@li...>]

Hi,

I was testing master recovery after failure: after switching master, 127
missing chunks appeared.

currently unavailable chunk 0000000000001244 (inode: 193 ; index: 0)
currently unavailable chunk 0000000000001245 (inode: 193 ; index: 1)
[..]
currently unavailable chunk 00000000000012C1 (inode: 193 ; index: 125)
currently unavailable chunk 00000000000012C2 (inode: 193 ; index: 126)
+ currently unavailable reserved file 193:
images/60a81e97504d92ce5238400baba858954bce2ead
unavailable chunks: 127
unavailable reserved files: 1

I didn't found any data corruption on my services (I use mfs as
OpenNebula storage backend).

What to do when unavailable chunk appears? There is a way to tell Moose
to forget them after sometime?
And what about unavailable reserved file? I have no problems accessing
that file (I did a cat file > /dev/null).

Thanks for any reply.

-- 
Giovanni Toraldo
http://www.libersoft.it/

Re: [Moosefs-users] how compile moosefs for windows???

[Alexander A. <akh...@ri...>]

Sorry for brief... ;--)

In my same case I use chunk servers virtual machines running on VMWare Server 1.0.10.
VMWare Server runs on Windows XP workstation.
Chunk servers VMs are FreeBSD 8.
VM's network adapters are configured as bridget network in VMWare settings.
so thay can directly communicate with MFS Master and MFS Clients.

wbr
Alexander Akhobadze

Re: [Moosefs-users] how compile moosefs for windows???

[Alexander A. <akh...@ri...>]

Hi!

In my same case I use chunk servers virtual machines running on VMWare Server 1.0.10.
VMWare Server runs on Windows XP workstation.

wbr
Alexander Akhobadze

======================================================
Вы писали 26 января 2011 г., 22:03:15:
======================================================

how compile moosefs for windows???
i want to use my local company network of over 500 personal computers as distributed data storage for vast amounts of data with low access available

Re: [Moosefs-users] how compile moosefs for windows???

[Anh K. H. <ky...@vi...>]

On Thu, 27 Jan 2011 00:03:15 +0500
Vadim Gluk <ro...@gm...> wrote:

> how compile moosefs for windows???
> i want to use my local company network of over 500 personal
> computers as distributed data storage for vast amounts of data with
> low access available

Quoted from MFS homepage: "The master server, metalogger server and chunkservers can also be run on Solaris or Windows with Cygwin. Unfortunately without FUSE it won't be possible to mount the filesystem within these operating system"

In your network, you can use some PCs as chunkservers; 500 is a great number :P

-- 
Anh Ky Huynh @ UTC+7

[Moosefs-users] how compile moosefs for windows???

[Vadim G. <ro...@gm...>]

how compile moosefs for windows???
i want to use my local company network of over 500 personal computers as
distributed data storage for vast amounts of data with low access available

[Moosefs-users] Performance issue with symlinks

[Flow J. <fl...@gm...>]

Hi,

I setup 2 Dell PowerEdge 1950 servers with moosefs-1.6.17 recently. 
Everything looks better than our original NFS mounted FS but when 
compiling source codes with gcc, the performance is bad. It takes 2~3 
times longer than NFS to generate dependency files. I did many tests to 
find out the reason, including changing the cache parameters for 
mfsmount, but 1Gbps mounted MFS is still not as good as 100Mbps mounted NFS.

Today I happened to see the report generated by mfscgiserv and one 
interesting finding is the "readlink" count to the source code folder is 
huge. And our source code is linked with symlinks for compiling so I 
think it's caused by the activity of reading source files.

I googled a while and found there's an old thread talked about caching 
for symlinks with fuse: 
http://sourceforge.net/mailarchive/forum.php?thread_name=E1K8yNn-00017E-PD%40pomaz-ex.szeredi.hu&forum_name=fuse-devel. 
So would that be the root cause of my problem? Or any other suggestions?

Many Thanks
Flow

Re: [Moosefs-users] What hardware do you use to keep costs low when using MooseFS?

[Steve <st...@bo...>]

Hi,



The low power supermicro atom X7SLA-L with ide for silicon os drive and 4
sata ports worked great for a chunkserver for me.

I intended to use the H dual core version with dual LAN for master but
budget ran dry.



The other chunks are on my 12 UKP each recycled dells you cant get cheaper
than that! 

 

 

 

-------Original Message------- 

 

From: Bán Miklós 

Date: 25/01/2011 22:16:37 

To: moo...@li... 

Subject: Re: [Moosefs-users] What hardware do you use to keep costs low when
using MooseFS? 

 

Hi, it's a good question! Recently I'm planning to create a low cost 

cluster with MooseFs. I would like to save energy using 

special hardware for chunk servers. Within a short time I will test 2.5" 

HDs in some hackable NAS boxes for this purpose. According to my idea, 

I can install mfs chunk server on NAS in a minimal Linux environment. 

For metalogger and master servers I'm thinking about to use SSD drives, 

however they are quite expensive, but fast and the power consumption is 

very low. So, on the whole, I prefer the lower operational costs than 

lower procurement cost. 

 

Miklos 

 

 

On Tue, 25 Jan 2011 21:09:23 +0100 

Ioannis Aslanidis <ias...@fl...> wrote: 

 

> Hello, 

> 

> I am wondering what every MooseFS user is currently purchasing to set 

> up their clusters. I want to set up something as cheap as possible 

> and I would like to know from the experiences of all of you. Can you 

> please share your knowledge? 

> 

> Thanks. 

> 

 

 

-----------------------------------------------------------------------------
 

Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)! 

Finally, a world-class log management solution at an even better price-free!


Download using promo code Free_Logger_4_Dev2Dev. Offer expires 

February 28th, so secure your free ArcSight Logger TODAY! 

http://p.sf.net/sfu/arcsight-sfd2d 

_______________________________________________ 

moosefs-users mailing list 

moo...@li... 

https://lists.sourceforge.net/lists/listinfo/moosefs-users 

 

Re: [Moosefs-users] What hardware do you use to keep costs low when using MooseFS?

[Bán M. <ba...@vo...>]

Hi, it's a good question! Recently I'm planning to create a low cost
cluster with MooseFs. I would like to save energy using
special hardware for chunk servers. Within a short time I will test 2.5"
HDs in some hackable NAS boxes for this purpose. According to my idea,
I can install mfs chunk server on NAS in a minimal Linux environment.
For metalogger and master servers I'm thinking about to use SSD drives,
however they are quite expensive, but fast and the power consumption is
very low. So, on the whole, I prefer the lower operational costs than
lower procurement cost.

Miklos


On Tue, 25 Jan 2011 21:09:23 +0100
Ioannis Aslanidis <ias...@fl...> wrote:

> Hello,
> 
> I am wondering what every MooseFS user is currently purchasing to set
> up their clusters. I want to set up something as cheap as possible
> and I would like to know from the experiences of all of you. Can you
> please share your knowledge?
> 
> Thanks.
> 

[Moosefs-users] What hardware do you use to keep costs low when using MooseFS?

[Ioannis A. <ias...@fl...>]

Hello,

I am wondering what every MooseFS user is currently purchasing to set up
their clusters. I want to set up something as cheap as possible and I would
like to know from the experiences of all of you. Can you please share your
knowledge?

Thanks.

-- 
Ioannis Aslanidis
System and Network Administrator
Flumotion Services, S.A.
E-Mail: iaslanidis at flumotion dot com
Office Phone: +34 93 508 63 59
Mobile Phone: +34 672 20 45 75

[Moosefs-users] Moosefs with VMWARE 4 ESX

[Bruce M. <bma...@gm...>]

We would like to use Moosefs as the primary storage for our VMWare ESX hosts
Clusters.  Can

someone please inform me how I would mount moose on vmware esx. I have been
trying with no luck.

It is my understanding that VmWware uses fuse as well.

Re: [Moosefs-users] problems with mfsmount under debian

[Josef <pe...@p-...>]

  Thanks for help, the device was missing, I needed to install udev package

Dne 1/21/11 10:08 AM, Michal Borychowski napsal(a):
> And do you have /dev/fuse? What are the privileges?
>
>
> Regards
> Michał
>
>
> -----Original Message-----
> From: Josef [mailto:pe...@p-...]
> Sent: Thursday, January 20, 2011 8:04 PM
> Cc: moo...@li...
> Subject: Re: [Moosefs-users] problems with mfsmount under debian
>
>    fuse is alive...
>
> postak:/mnt# lsmod | grep fuse
> fuse                   40596  1
>
>
> Dne 1/20/11 7:55 PM, Ricardo J. Barberis napsal(a):
>> El Jue 20 Enero 2011, Josef escribió:
>>>     Hello,
>>>        I'm having problems with mfsmount on my Debian system. If I do
>>> mfsmount /mnt/mfs/ -H 10.0.0.1
>>> I get:
>>> mfsmaster accepted connection with parameters: read-write,restricted_ip
>>> ; root mapped to root:root
>>> fuse: device not found, try 'modprobe fuse' first
>>> error in fuse_mount
>>>
>>> But fuse seems to be installed:
>>> fusermount --version
>>> fusermount version: 2.7.4
>>>
>>> Any suggestions?
>> Firts check if it really is loaded:
>>
>> lsmod | grep fuse
>>
>> If not:
>>
>> modprobe -v fuse
>>
>>> Thanks!
>>>
>>> Josef
>> Cheers,
> ----------------------------------------------------------------------------
> --
> Protect Your Site and Customers from Malware Attacks
> Learn about various malware tactics and how to avoid them. Understand
> malware threats, the impact they can have on your business, and how you
> can protect your company and customers by using code signing.
> http://p.sf.net/sfu/oracle-sfdevnl
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>

Re: [Moosefs-users] How big can MooseFS really go?

[Michal B. <mic...@ge...>]

Hi Thomas!

 

 

> As I understand it, when changes are made to the /etc/mfsexports.cfg file
you need to restart the mfsmaster daemon. We would like to be able to modify
this file and then have the new exports applied without having to restart
the mfsmaster daemon.

[MB] You should run "mfsmaster reload" or "killall -HUP mfsmaster"

 

  

> let me know if there is a way to do this already, I was going to see if it
would just take a HUP signal once out QA environment is rebuilt, but I don't
think it hurts to ask.

[MB] For the moment after HUP signal only "mfsexports.cfg" is reloaded - but
this is just what you need

 

Regards

Michal

 

 

From: Thomas S Hatch [mailto:tha...@gm...] 
Sent: Friday, January 21, 2011 4:04 PM
To: Michal Borychowski
Cc: moosefs-users
Subject: Re: [Moosefs-users] How big can MooseFS really go?

 

Thanks Michal! I really appreciate your efforts for us, I hope I can give
back more to moosefs in the future.

 

we are planning on giving it 128 MB of ram, minimum, and then we are going
to toss a fusionio drive into the master and set it up as swap space <just
in case>

 

Right now we are serving files via apache, and the web front-end is entirely
in django using wsgi as the python interface. We are looking into moving it
to nginx, as most of our frontends use nginx, but an apache module would
encourage us to stick with apache!

 

But one of the big epiphanies came yesterday when we realized that we can
use fusionio drives as swap space, because they perform at almost the speed
of ram. We are probably not looking at this for a while, but I am serious
about giving you guys a fusionio drive in the future if it will help moosefs
development.

 

We did have one question, what are the chances that the master servers
exports could be dynamically loaded?

 

This and the client caching we are really counting on! We have been
crunching the numbers and it looks like we will be resting somewhere around
3.2 Petabytes when we are done with the build out. We will keep you posted
and we deeply appreciate your efforts, not only with MooseFS, but also on
our behalf!

 

-Tom

2011/1/21 Michal Borychowski <mic...@ge...>

Hi Thomas!

 

Sorry for late reply but the last two days were very busy and full of
meetings.

 

We consulted your installation among our devs and admins and to be honest we
cannot give you any more tips to what Jose and Reinis have already said. For
a start you should have about 64GB RAM available in the master server
(probably 48GB would be also fine). Performance is not that affected by the
space in the cluster as by the number of files (objects) in the system and
number of current operations (recorded in changelogs).

 

As we already talked your files don't change often (if ever) so if you
prepare client machines with much RAM quite a lot of files would get cached
in RAM on them (see
http://www.moosefs.org/news-reader/items/moose-file-system-v-1617-released.h
tml). 

 

And what server would you use to serve the files? Lighttpd / Apache or any
other? We had some plans to implement a direct module in Apache for MooseFS
- it should also speed up the communication.

 

 

Regards

Michal

 

 

 

 

From: Thomas S Hatch [mailto:tha...@gm...] 

Sent: Tuesday, January 18, 2011 6:56 PM
To: Michal Borychowski
Cc: moosefs-users

Subject: Re: [Moosefs-users] How big can MooseFS really go?

 

FusionIO in all the chunkservers, thats a little too rich for my blood :)

 

one of the problems we are seeing is that we need to have the failover work
faster, and the bottleneck for mfsmetarestore looks like it is IO, and the
mfsmetarestore in the failover process is what takes up the most time. Thats
why we want the fusionio drives.

 

As for number of files, we have about 13 million right now, but we have only
imported a small percentage of the total number of files we are contracted
to get (we are making a music "cloud", and we are getting all the music in
the world, a lot is still coming). By the end of the year we should have
about 100-150 million files.

 

Right now the idea is to have two types of storage, the moose for large
scale storage where write speed is not a major issue, and then a "cluster"
of ssd pci cards to handle high speed storage needs, like databases, and the
master and metalogger to speed up restores and to make sure the changelogs
can be written fast enough when the activity is so high.

 

Paint a better picture?

 

-Tom

2011/1/18 Michal Borychowski <mic...@ge...>

WOW!!! 

 

And what about FusionIO in all the chunkservers? ;) But as you are talking
about FusionIO in mfsmaster so probably you are also going to use them in
metaloggers? I just think if it is necessary. Metadata is cached in RAM in
mfsmaster, but there are changelogs. If the system will be busy (and for
sure it will be) there would be lots of operations logged to the files and
transmitted to metaloggers.

 

Please tell us how many files do you plan to store?

 

 

Regards

Michal

 

 

From: Thomas S Hatch [mailto:tha...@gm...] 
Sent: Tuesday, January 18, 2011 6:06 PM
To: moosefs-users
Subject: [Moosefs-users] How big can MooseFS really go?

 

I am architecting a potential 3P MooseFS install.... 3 Peta-bytes... and
maybe even 4P (assuming we will be moving to 4T disks when they come out).

 

My question is this, can MooseFS handle that kind of load? Are there any
additional considerations that I will need to take as I approach such a high
volume.

 

As I am seeing I will have over 100 chunkservers attached to one master. I
am going to change out the mfsmaster metadata store with fusionio
(http://www.fusionio.com/) drives to maintain the disk speed that metadata
operations will need.

 

This deployment will also require well over 100 million chunks.

 

So my question again is, what, if any, special considerations should I take
as I roll this out?

 

-Thomas S Hatch

 

 

Re: [Moosefs-users] How big can MooseFS really go?

[jose m. <let...@us...>]

El vie, 21-01-2011 a las 08:30 +0100, Michal Borychowski escribió:
> [MB] Hi Jose!
> 
> [MB] We consulted your tuning suggestions and here are some of our thoughts
> 
> [...]
> litte sysctl tunning
> 


* The parameters applied to the options are "mere examples" and
dependent on characteristics on the cards of network and switch's, etc.,
they can be increased or diminished according to the observed yield and
the characteristics of the hardware, in my experience in two main
clusters, I have preferred not to look for the transference limits since
they are connected between them and it was causing the loss of constant
and random connection of chunkservers, to fit the reconnection times and
and beginning in the connections has caused an entire stability in the
connections, of course at the cost of a less yield in the tranferencias,
but in my case this was not priority. 

Re: [Moosefs-users] How big can MooseFS really go?

[Thomas S H. <tha...@gm...>]

Thanks Michal! I really appreciate your efforts for us, I hope I can give
back more to moosefs in the future.

we are planning on giving it 128 MB of ram, minimum, and then we are going
to toss a fusionio drive into the master and set it up as swap space <just
in case>

Right now we are serving files via apache, and the web front-end is entirely
in django using wsgi as the python interface. We are looking into moving it
to nginx, as most of our frontends use nginx, but an apache module would
encourage us to stick with apache!

But one of the big epiphanies came yesterday when we realized that we can
use fusionio drives as swap space, because they perform at almost the speed
of ram. We are probably not looking at this for a while, but I am serious
about giving you guys a fusionio drive in the future if it will help moosefs
development.

We did have one question, what are the chances that the master servers
exports could be dynamically loaded?

This and the client caching we are really counting on! We have been
crunching the numbers and it looks like we will be resting somewhere around
3.2 Petabytes when we are done with the build out. We will keep you posted
and we deeply appreciate your efforts, not only with MooseFS, but also on
our behalf!

-Tom

2011/1/21 Michal Borychowski <mic...@ge...>

>  Hi Thomas!
>
>
>
> Sorry for late reply but the last two days were very busy and full of
> meetings.
>
>
>
> We consulted your installation among our devs and admins and to be honest
> we cannot give you any more tips to what Jose and Reinis have already said.
> For a start you should have about 64GB RAM available in the master server
> (probably 48GB would be also fine). Performance is not that affected by the
> space in the cluster as by the number of files (objects) in the system and
> number of current operations (recorded in changelogs).
>
>
>
> As we already talked your files don’t change often (if ever) so if you
> prepare client machines with much RAM quite a lot of files would get cached
> in RAM on them (see
> http://www.moosefs.org/news-reader/items/moose-file-system-v-1617-released.html).
>
>
>
>
> And what server would you use to serve the files? Lighttpd / Apache or any
> other? We had some plans to implement a direct module in Apache for MooseFS
> – it should also speed up the communication.
>
>
>
>
>
> Regards
>
> Michal
>
>
>
>
>
>
>
>
>
> *From:* Thomas S Hatch [mailto:tha...@gm...]
> *Sent:* Tuesday, January 18, 2011 6:56 PM
> *To:* Michal Borychowski
> *Cc:* moosefs-users
> *Subject:* Re: [Moosefs-users] How big can MooseFS really go?
>
>
>
> FusionIO in all the chunkservers, thats a little too rich for my blood :)
>
>
>
> one of the problems we are seeing is that we need to have the failover work
> faster, and the bottleneck for mfsmetarestore looks like it is IO, and the
> mfsmetarestore in the failover process is what takes up the most time. Thats
> why we want the fusionio drives.
>
>
>
> As for number of files, we have about 13 million right now, but we have
> only imported a small percentage of the total number of files we are
> contracted to get (we are making a music "cloud", and we are getting all the
> music in the world, a lot is still coming). By the end of the year we should
> have about 100-150 million files.
>
>
>
> Right now the idea is to have two types of storage, the moose for large
> scale storage where write speed is not a major issue, and then a "cluster"
> of ssd pci cards to handle high speed storage needs, like databases, and the
> master and metalogger to speed up restores and to make sure the changelogs
> can be written fast enough when the activity is so high.
>
>
>
> Paint a better picture?
>
>
>
> -Tom
>
> 2011/1/18 Michal Borychowski <mic...@ge...>
>
> WOW!!!
>
>
>
> And what about FusionIO in all the chunkservers? ;) But as you are talking
> about FusionIO in mfsmaster so probably you are also going to use them in
> metaloggers? I just think if it is necessary… Metadata is cached in RAM in
> mfsmaster, but there are changelogs… If the system will be busy (and for
> sure it will be) there would be lots of operations logged to the files and
> transmitted to metaloggers…
>
>
>
> Please tell us how many files do you plan to store?
>
>
>
>
>
> Regards
>
> Michal
>
>
>
>
>
> *From:* Thomas S Hatch [mailto:tha...@gm...]
> *Sent:* Tuesday, January 18, 2011 6:06 PM
> *To:* moosefs-users
> *Subject:* [Moosefs-users] How big can MooseFS really go?
>
>
>
> I am architecting a potential 3P MooseFS install.... 3 Peta-bytes... and
> maybe even 4P (assuming we will be moving to 4T disks when they come out).
>
>
>
> My question is this, can MooseFS handle that kind of load? Are there any
> additional considerations that I will need to take as I approach such a high
> volume.
>
>
>
> As I am seeing I will have over 100 chunkservers attached to one master. I
> am going to change out the mfsmaster metadata store with fusionio (
> http://www.fusionio.com/) drives to maintain the disk speed that
> metadata operations will need.
>
>
>
> This deployment will also require well over 100 million chunks.
>
>
>
> So my question again is, what, if any, special considerations should I take
> as I roll this out?
>
>
>
> -Thomas S Hatch
>
>
>

Re: [Moosefs-users] problems with mfsmount under debian

[Michal B. <mic...@ge...>]

And do you have /dev/fuse? What are the privileges?


Regards
Michał 


-----Original Message-----
From: Josef [mailto:pe...@p-...] 
Sent: Thursday, January 20, 2011 8:04 PM
Cc: moo...@li...
Subject: Re: [Moosefs-users] problems with mfsmount under debian

  fuse is alive...

postak:/mnt# lsmod | grep fuse
fuse                   40596  1


Dne 1/20/11 7:55 PM, Ricardo J. Barberis napsal(a):
> El Jue 20 Enero 2011, Josef escribió:
>>    Hello,
>>       I'm having problems with mfsmount on my Debian system. If I do
>> mfsmount /mnt/mfs/ -H 10.0.0.1
>> I get:
>> mfsmaster accepted connection with parameters: read-write,restricted_ip
>> ; root mapped to root:root
>> fuse: device not found, try 'modprobe fuse' first
>> error in fuse_mount
>>
>> But fuse seems to be installed:
>> fusermount --version
>> fusermount version: 2.7.4
>>
>> Any suggestions?
> Firts check if it really is loaded:
>
> lsmod | grep fuse
>
> If not:
>
> modprobe -v fuse
>
>> Thanks!
>>
>> Josef
> Cheers,

----------------------------------------------------------------------------
--
Protect Your Site and Customers from Malware Attacks
Learn about various malware tactics and how to avoid them. Understand 
malware threats, the impact they can have on your business, and how you 
can protect your company and customers by using code signing.
http://p.sf.net/sfu/oracle-sfdevnl
_______________________________________________
moosefs-users mailing list
moo...@li...
https://lists.sourceforge.net/lists/listinfo/moosefs-users

Re: [Moosefs-users] How big can MooseFS really go?

[Michal B. <mic...@ge...>]

[MB] Hi Jose!

[MB] We consulted your tuning suggestions and here are some of our thoughts

[...]
litte sysctl tunning

# increase TCP max buffer size
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# increase Linux autotuning TCP buffer limits 
# min, default, and max number of bytes to use 
net.ipv4.tcp_rmem = 4096 87380 16777216 
net.ipv4.tcp_wmem = 4096 65536 16777216 
[MB] As far as we know it should be already tuned in new kernels (ca 2.6.20+)


# don't cache ssthresh from previous connection 
net.ipv4.tcp_no_metrics_save = 1 
[MB] We are not sure about this. Doesn't it have a contrary result as slowing down new TCP connections at the beginning? We found out that this function is more for benchmarking and tests.

net.ipv4.tcp_moderate_rcvbuf = 1 
[MB] As far as we know this is on by default.

# recommended to increase this for 1000 BT or higher 
net.core.netdev_max_backlog = 2500 
# for 10 GigE, use this 
# net.core.netdev_max_backlog = 30000 
This would be useful only if you see timeouts in connections. In our environment by 50k connection attempts / sec we do not observe any timeouts.


# cubic on my system 
net.ipv4.tcp_congestion_control = cubic 
[MB] Similarly on new systems it is by default (eg. Ubuntu 10.X)


# probe swappiness 0 #
vm.swappiness=2
vm.vfs_cache_pressure = 10000
[MB] If the server has lots of RAM probably it won’t have any bigger effect.

We found some more information here:
http://www.linuxweblog.com/blogs/sandip/20080331/tuning-tcp-sysctlconf


And one simple thing to speed up the chunks is to specify 'noatime' flag in fstab. By default every time a file is accessed the file's inode information is updated to reflect the last access time which incurs a write to the file system metadata. When we set noatime flag there would be no unnecessary write while reading.


Regards
Michal

Re: [Moosefs-users] How big can MooseFS really go?

[Michal B. <mic...@ge...>]

Hi Thomas!

 

Sorry for late reply but the last two days were very busy and full of
meetings.

 

We consulted your installation among our devs and admins and to be honest we
cannot give you any more tips to what Jose and Reinis have already said. For
a start you should have about 64GB RAM available in the master server
(probably 48GB would be also fine). Performance is not that affected by the
space in the cluster as by the number of files (objects) in the system and
number of current operations (recorded in changelogs).

 

As we already talked your files don't change often (if ever) so if you
prepare client machines with much RAM quite a lot of files would get cached
in RAM on them (see
http://www.moosefs.org/news-reader/items/moose-file-system-v-1617-released.h
tml). 

 

And what server would you use to serve the files? Lighttpd / Apache or any
other? We had some plans to implement a direct module in Apache for MooseFS
- it should also speed up the communication.

 

 

Regards

Michal

 

 

 

 

From: Thomas S Hatch [mailto:tha...@gm...] 
Sent: Tuesday, January 18, 2011 6:56 PM
To: Michal Borychowski
Cc: moosefs-users
Subject: Re: [Moosefs-users] How big can MooseFS really go?

 

FusionIO in all the chunkservers, thats a little too rich for my blood :)

 

one of the problems we are seeing is that we need to have the failover work
faster, and the bottleneck for mfsmetarestore looks like it is IO, and the
mfsmetarestore in the failover process is what takes up the most time. Thats
why we want the fusionio drives.

 

As for number of files, we have about 13 million right now, but we have only
imported a small percentage of the total number of files we are contracted
to get (we are making a music "cloud", and we are getting all the music in
the world, a lot is still coming). By the end of the year we should have
about 100-150 million files.

 

Right now the idea is to have two types of storage, the moose for large
scale storage where write speed is not a major issue, and then a "cluster"
of ssd pci cards to handle high speed storage needs, like databases, and the
master and metalogger to speed up restores and to make sure the changelogs
can be written fast enough when the activity is so high.

 

Paint a better picture?

 

-Tom

2011/1/18 Michal Borychowski <mic...@ge...>

WOW!!! 

 

And what about FusionIO in all the chunkservers? ;) But as you are talking
about FusionIO in mfsmaster so probably you are also going to use them in
metaloggers? I just think if it is necessary. Metadata is cached in RAM in
mfsmaster, but there are changelogs. If the system will be busy (and for
sure it will be) there would be lots of operations logged to the files and
transmitted to metaloggers.

 

Please tell us how many files do you plan to store?

 

 

Regards

Michal

 

 

From: Thomas S Hatch [mailto:tha...@gm...] 
Sent: Tuesday, January 18, 2011 6:06 PM
To: moosefs-users
Subject: [Moosefs-users] How big can MooseFS really go?

 

I am architecting a potential 3P MooseFS install.... 3 Peta-bytes... and
maybe even 4P (assuming we will be moving to 4T disks when they come out).

 

My question is this, can MooseFS handle that kind of load? Are there any
additional considerations that I will need to take as I approach such a high
volume.

 

As I am seeing I will have over 100 chunkservers attached to one master. I
am going to change out the mfsmaster metadata store with fusionio
(http://www.fusionio.com/) drives to maintain the disk speed that metadata
operations will need.

 

This deployment will also require well over 100 million chunks.

 

So my question again is, what, if any, special considerations should I take
as I roll this out?

 

-Thomas S Hatch