Re: [Moosefs-users] How big can MooseFS really go?

[Thomas S H. <tha...@gm...>]

Don't worry about the English jose, this data is great, you have a few more
kernel tuning changes that I had not thought of.

Just like my environment, hardly any load on the cpu, your swappiness is
lower that I have made mine (10) but 2 may yet be a good way to go.

Thanks jose! This is good stuff!

-Tom

On Tue, Jan 18, 2011 at 1:52 PM, jose maria <let...@us...> wrote:

> El mar, 18-01-2011 a las 11:58 -0700, Thomas S Hatch escribió:
> > Thanks jose, yes, we are on 10G networks, it sounds like the primary
> > cap to worry about might be the ram usage on the mfsmaster.
> >
>
> * standard kernel opensuse 11.3, X86_64 x 2 processors
> Intel(R) Xeon(R) CPU           E5504  @ 2.00GHz
>
> top - 20:56:26 up 18 days, 10 min,  1 user,  load average: 0.00, 0.00,
> 0.00
> Tasks: 141 total,   1 running, 140 sleeping,   0 stopped,   0 zombie
> Cpu0  :  0.3%us,  0.3%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.3%hi,  0.0%si,
> 0.0%st
> Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu3  :  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu4  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu6  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu7  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
>
> Mem:   16 GiB total , 22.8 %  memory used for mfsmaster process
> +-20.000.000 chunks, +-6.000.000 files. , litte files, goal 3
>
> MTU 9000 , NIC's 2x1GE, bonding mode 5.
>
> 20 chunkservers 2x1GE , bonding mode 6.
>
> litte sysctl tunning
>
> # increase TCP max buffer size
> net.core.rmem_max = 16777216
> net.core.wmem_max = 16777216
> # increase Linux autotuning TCP buffer limits
> # min, default, and max number of bytes to use
> net.ipv4.tcp_rmem = 4096 87380 16777216
> net.ipv4.tcp_wmem = 4096 65536 16777216
> # don't cache ssthresh from previous connection
> net.ipv4.tcp_no_metrics_save = 1
> net.ipv4.tcp_moderate_rcvbuf = 1
> # recommended to increase this for 1000 BT or higher
> net.core.netdev_max_backlog = 2500
> # for 10 GigE, use this
> # net.core.netdev_max_backlog = 30000
> # cubic on my system
> net.ipv4.tcp_congestion_control = cubic
> # probe swappiness 0
> #
> vm.swappiness=2
> vm.vfs_cache_pressure = 10000
>
>
> * Another, 83 chunkservers 1x10GE,
> 2x10GE in mfsmaster + 1x1GE
>
> opensuse 11.3 recompiled, numa, hig memory.
>
> 64GiB Ram, +-40% memory used por mfsmaster process
> +-100.000.000 chunks, +-30.000.000 litte files, goal 3
>
> * For test purposes
> http://control.seycob.es:9425
> http://control.seycob.es:9426
>
> * in upgrade to 1.2.19
> * problems with cpu comsuption on mfs-1.6.19, solved, fine in 1.6.20
> * problems with files goal 0, ¿valid copies 1? 4 days not solved.
> * problems with files goal 1, valid copies 1, apply goal 2 to all
> filesystem and not reflected. 4 days not solved, in test cluster:
>
> http://control.seycob.es:9426/mCZ RevoDfs.cgi<http://control.seycob.es:9426/mfs.cgi>
>
> * poor english, sorry
>
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Protect Your Site and Customers from Malware Attacks
> Learn about various malware tactics and how to avoid them. Understand
> malware threats, the impact they can have on your business, and how you
> can protect your company and customers by using code signing.
> http://p.sf.net/sfu/oracle-sfdevnl
> _______________________________________________
> moosefs-users mailing list
> moo...@li...
> https://lists.sourceforge.net/lists/listinfo/moosefs-users
>
>