[Moosefs-users] MooseFS and Generic Segmentation Offload in CentOS

[Ólafur Ó. <osv...@ne...>]

Hi,
I just wanted to give a heads up if anyone gets into the same problems as we did here.

Our new setup is like this:

1 x Master (Ubuntu 10.04)
1 x Metalogger (Ubuntu 10.04)
10 x Chunkservers (Ubuntu 10.04)

And then we have for testing 8 CentOS 5.5 servers using the MFS.

All Ubuntu servers have Broadcom BCM5708 network cards and the CentOS blades have BCM5709 and this is all connected through Cisco 3560 and 2960 switches.

Everything was working fine after the initial install and then I set Jumbo-Frames on the switches and changed the MTU on all the servers to 9000, and by that point I could only get a listing of files but we were unable to read the contents of large files on MFS mounts from the CentOS machines, but if the filesystem was mounted from the Master it worked fine.

Direct communication seemed to work fine, ping with different size packets, ssh and other services did not fail.

After a couple of days looking at this I found that the operating systems have different driver settings for atleast Broadcom cards, while Ubuntu has a switch called "generic segmentation offload" set by default to on, CentOS has the switch set to off, by changing this setting on the CentOS machines it now runs fine just like before.

To set it you use "ethtool -K gso on"

/Oli

--
Ólafur Osvaldsson
System Administrator
Nethonnun ehf.
e-mail: osv...@ne...
phone: +354 517 3418