Re: [Moosefs-users] How big can MooseFS really go?

[Michal B. <mic...@ge...>]

[MB] Hi Jose!

[MB] We consulted your tuning suggestions and here are some of our thoughts

[...]
litte sysctl tunning

# increase TCP max buffer size
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# increase Linux autotuning TCP buffer limits 
# min, default, and max number of bytes to use 
net.ipv4.tcp_rmem = 4096 87380 16777216 
net.ipv4.tcp_wmem = 4096 65536 16777216 
[MB] As far as we know it should be already tuned in new kernels (ca 2.6.20+)


# don't cache ssthresh from previous connection 
net.ipv4.tcp_no_metrics_save = 1 
[MB] We are not sure about this. Doesn't it have a contrary result as slowing down new TCP connections at the beginning? We found out that this function is more for benchmarking and tests.

net.ipv4.tcp_moderate_rcvbuf = 1 
[MB] As far as we know this is on by default.

# recommended to increase this for 1000 BT or higher 
net.core.netdev_max_backlog = 2500 
# for 10 GigE, use this 
# net.core.netdev_max_backlog = 30000 
This would be useful only if you see timeouts in connections. In our environment by 50k connection attempts / sec we do not observe any timeouts.


# cubic on my system 
net.ipv4.tcp_congestion_control = cubic 
[MB] Similarly on new systems it is by default (eg. Ubuntu 10.X)


# probe swappiness 0 #
vm.swappiness=2
vm.vfs_cache_pressure = 10000
[MB] If the server has lots of RAM probably it won’t have any bigger effect.

We found some more information here:
http://www.linuxweblog.com/blogs/sandip/20080331/tuning-tcp-sysctlconf


And one simple thing to speed up the chunks is to specify 'noatime' flag in fstab. By default every time a file is accessed the file's inode information is updated to reflect the last access time which incurs a write to the file system metadata. When we set noatime flag there would be no unnecessary write while reading.


Regards
Michal