[MooseFS-Users] Performance degradation on Moosefs 3.x

[Ricardo J. B. <ric...@do...>]

Hello,

I'm having a strange issue with Moosefs 3.x, my servers (1 metadata, 2 
chunkservers) and clients (webhosting servers) are all CentOS 7 with Moosefs 
3.0.84.

My clients also have a Moosefs 2.x cluster mounted so I compiled 
moosefs-client-3.0.84 (cause mfsmount 3.x can't mount from moosefs 2.x 
master).

The day after we started using moosefs 3.x we noticed some processes (tar xf, 
rm -f) got stuck for a long time, we killed them, remounted moosefs 3.x and 
started the processes again and they completed quickly and without errors.


Here's an example showing how performance degrades over time for writing:

[root@centos7 ~] # /usr/local/moosefs-3/bin/mfsmount --version
MFS version 3.0.84-1
FUSE library version: 2.9.2
fusermount version: 2.9.2

[root@centos7 ~] # time tar xf joomla.tar -C /mfs/joomla
real    0m17.145s
user    0m0.273s
sys     0m0.361s

[root@centos7 ~] # time tar xf joomla.tar -C /mfs/joomla
real    0m35.582s
user    0m0.298s
sys     0m0.521s

[root@centos7 ~] # time tar xf joomla.tar -C /mfs/joomla
real    0m59.039s
user    0m0.358s
sys     0m0.605s

[root@centos7 ~] # time tar xf joomla.tar -C /mfs/joomla
real    1m24.846s
user    0m0.350s
sys     0m0.627s

[root@centos7 ~] # time tar xf joomla.tar -C /mfs/joomla
real    1m37.159s
user    0m0.376s
sys     0m0.686s



Another example:

[root@centos7 ~] # for N in {0..4} ; do echo -e "\nUncompressing 
in /mfs/joomla - ${N}" ; mkdir -p /mfs/joomla ; sleep 2 ; time tar 
xf /usr/src/joomla.tar -C /mfs/joomla ; echo "Sleeping 20..." ; sleep 20 ; 
echo -e "\nRemoving /mfs/joomla - ${N}" ; time rm -fr /mfs/joomla ; 
echo "Sleeping 5..." ; sleep 5 ; done

Uncompressing in /mfs/joomla - 0
real    0m12.545s
user    0m0.255s
sys     0m0.272s
Sleeping 20...

Removing /mfs/joomla - 0
real    0m3.524s
user    0m0.022s
sys     0m0.127s
Sleeping 5...

Uncompressing in /mfs/joomla - 1
real    0m15.138s
user    0m0.289s
sys     0m0.289s
Sleeping 20...
Removing /mfs/joomla - 1

real    0m6.321s
user    0m0.025s
sys     0m0.166s
Sleeping 5...

Uncompressing in /mfs/joomla - 2
real    0m14.181s
user    0m0.262s
sys     0m0.266s
Sleeping 20...

Removing /mfs/joomla - 2
real    0m6.539s
user    0m0.025s
sys     0m0.134s
Sleeping 5...

Uncompressing in /mfs/joomla - 3
real    0m15.442s
user    0m0.253s
sys     0m0.253s
Sleeping 20...

Removing /mfs/joomla - 3
real    0m8.541s
user    0m0.033s
sys     0m0.171s
Sleeping 5...

Uncompressing in /mfs/joomla - 4
real    0m17.717s
user    0m0.257s
sys     0m0.280s
Sleeping 20...

Removing /mfs/joomla - 4
real    0m11.533s
user    0m0.031s
sys     0m0.172s
Sleeping 5...



If I mount the same cluster (mfsmaster 3.x) with mfsmount 2.x it works as 
expected (but notice that moosefs 3.x works way faster initially!), eg:

[root@centos7 ~] # for N in {0..4} ; do echo -e "\nUncompressing 
in /mfs/joomla - ${N}" ; mkdir -p /mfs/joomla ; sleep 2 ; time tar 
xf /usr/src/joomla.tar -C /mfs/joomla ; echo "Sleeping 20..." ; sleep 20 ; 
echo -e "\nRemoving /mfs/joomla - ${N}" ; time rm -fr /mfs/joomla ; 
echo "Sleeping 5..." ; sleep 5 ; done

Uncompressing in /mfs/joomla - 0
real    0m31.878s
user    0m0.311s
sys     0m0.394s
Sleeping 20...

Removing /mfs/joomla - 0
real    0m3.495s
user    0m0.026s
sys     0m0.118s
Sleeping 5...

Uncompressing in /mfs/joomla - 1
real    0m25.544s
user    0m0.283s
sys     0m0.394s
Sleeping 20...

Removing /mfs/joomla - 1
real    0m3.435s
user    0m0.017s
sys     0m0.127s
Sleeping 5...

Uncompressing in /mfs/joomla - 2
real    0m26.498s
user    0m0.258s
sys     0m0.314s
Sleeping 20...

Removing /mfs/joomla - 2
real    0m2.725s
user    0m0.022s
sys     0m0.138s
Sleeping 5...

Uncompressing in /mfs/joomla - 3
real    0m21.142s
user    0m0.276s
sys     0m0.344s
Sleeping 20...

Removing /mfs/joomla - 3
real    0m2.718s
user    0m0.020s
sys     0m0.110s
Sleeping 5...

Uncompressing in /mfs/joomla - 4
real    0m22.301s
user    0m0.259s
sys     0m0.292s
Sleeping 20...

Removing /mfs/joomla - 4
real    0m2.942s
user    0m0.025s
sys     0m0.106s
Sleeping 5...


I tried with several combinations of mfsnoxattrs, mfsnoposixlocks and 
mfsnobsdlocks.

I also set 'ATIME_MODE = 4' on the master but it made no difference.

Any hints, optimization tips, etc are welcome!

Regards,
-- 
Ricardo J. Barberis
Senior SysAdmin / IT Architect
DonWeb
La Actitud Es Todo
www.DonWeb.com