Re: [Moosefs-users] How big can MooseFS really go?

[jose m. <let...@us...>]

El mar, 18-01-2011 a las 11:58 -0700, Thomas S Hatch escribió:
> Thanks jose, yes, we are on 10G networks, it sounds like the primary
> cap to worry about might be the ram usage on the mfsmaster.
>

* standard kernel opensuse 11.3, X86_64 x 2 processors
Intel(R) Xeon(R) CPU           E5504  @ 2.00GHz

top - 20:56:26 up 18 days, 10 min,  1 user,  load average: 0.00, 0.00,
0.00
Tasks: 141 total,   1 running, 140 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.3%us,  0.3%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.3%hi,  0.0%si,
0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu3  :  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu4  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu6  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu7  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st

Mem:   16 GiB total , 22.8 %  memory used for mfsmaster process
+-20.000.000 chunks, +-6.000.000 files. , litte files, goal 3

MTU 9000 , NIC's 2x1GE, bonding mode 5.

20 chunkservers 2x1GE , bonding mode 6.

litte sysctl tunning

# increase TCP max buffer size
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# increase Linux autotuning TCP buffer limits
# min, default, and max number of bytes to use
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
# don't cache ssthresh from previous connection
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
# recommended to increase this for 1000 BT or higher
net.core.netdev_max_backlog = 2500
# for 10 GigE, use this
# net.core.netdev_max_backlog = 30000
# cubic on my system
net.ipv4.tcp_congestion_control = cubic
# probe swappiness 0
#
vm.swappiness=2
vm.vfs_cache_pressure = 10000


* Another, 83 chunkservers 1x10GE, 
2x10GE in mfsmaster + 1x1GE

opensuse 11.3 recompiled, numa, hig memory.

64GiB Ram, +-40% memory used por mfsmaster process
+-100.000.000 chunks, +-30.000.000 litte files, goal 3

* For test purposes
http://control.seycob.es:9425
http://control.seycob.es:9426

* in upgrade to 1.2.19
* problems with cpu comsuption on mfs-1.6.19, solved, fine in 1.6.20
* problems with files goal 0, ¿valid copies 1? 4 days not solved.
* problems with files goal 1, valid copies 1, apply goal 2 to all
filesystem and not reflected. 4 days not solved, in test cluster:

http://control.seycob.es:9426/mfs.cgi

* poor english, sorry