Menu
▾
▴
Re: [Moosefs-users] Performance with 40 Clients
|
From: Kai K. <kk...@we...> - 2010-10-29 13:42:58
|
Hi, we've been able to reduce getattr and read operations from constantly 1.5 million per minute to 100000 per minute by using 'mfsmount -o mfsdirentrycacheto=30 -s -f /mnt/mfs'. lookup operations were in the 20000/minute range and write operations differ widely between zero and 1500 per minute. Everyting else barely noticeable. But in the end it didn't help to reduce cpu usage, dtrace helped us to confirm that the high system call rate of 250000 per second is attributed to high select() count. Contex switches were at the rate of 25000 per second. Every box is using version 1.6.17. Unfortunately we're unable to exactly simulate the situation offline but running "find /mnt/mfs >/dev/null" on 20 clients easily gets cpu usage up to 90% at 10MiB/s network traffic whereas running "rsync -a /usr/ports /mnt/mfs" on 20 clients runs pretty slow at 5 to 15% percent cpu usage. Our mfs.cgi website is not generally accessible from the internet but I guess we're happy to enable access if you are so kind to tell us your static client IP-address :-) Thanks, Tino Am Friday 29 October 2010 14:16:21 schrieb Michał Borychowski: > Hi! > > Could you send some screens / charts with the master server usage? We would > be mainly interested what operations and how many of them take place. > > At our environment the main cause of CPU increase are "getattr", "lookup" > and "readdir" operations. The options used in mfsmount are correct and > should decrease number of these operations. But in your case probably > something else slows down the server. > > We have two master processes on a FreeBSD 7.1 machine: > > 791 nobody 1 80 -19 10665M 10047M CPU0 0 659.8H 16.70% mfsmaster > 786 nobody 1 69 -19 238M 225M select 1 127.0H 11.28% mfsmaster > > The first has 23 000 000 files and the second 5 000 > > systat gives: > > em0 in 808.354 KB/s 1010.191 KB/s 3.687 TB > out 1.328 MB/s 9.395 MB/s 11.022 TB > > so the transfer is similar. > > You can have a look at our charts and compare: http://80.48.16.122/ > > > > If you need any further assistance please let us know. > > Kind regards > Michał Borychowski > MooseFS Support Manager > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > Gemius S.A. > ul. Wołoska 7, 02-672 Warszawa > Budynek MARS, klatka D > Tel.: +4822 874-41-00 > Fax : +4822 874-41-01 > > > > -----Original Message----- > From: Kai Kockro [mailto:kk...@we...] > Sent: Thursday, October 28, 2010 10:32 AM > To: moo...@li... > Subject: [Moosefs-users] Performance with 40 Clients > > Hi, > > we have one mfsmaster running on FreeBSD 8.1-STABLE. Filesystem is > completly > > ZFS. The Server has 16G Ram. The mfsmaster prozess has a res size of 173M. > CPU > is a Xeon L3426 @ 1.87GHz. > > Our 40 Clients are running on FreeBSD 7.2. There comes massive file > uploading > ( filesize from 10kb - 10mb ). > > The args for mfsmount are "mfsmount -o mfscachefiles -o mfsentrycacheto=30 > -o > mfsattrcacheto=30 -o mfsdirentrycacheto=30 -s -f /mnt/" Without these > settings > we had massive gettatr requests. > > If we use this mounts for live system ( massive traffic, systat -if 1 shows > 1-10MB/sec on mfsmaster server ), the mfsmaster runs to 100% cpu use. The > php > / httpd prozesses on the clients stuck in fu_ans state. Are there any tips > or > tricks we can try? If you need more infos, please let us know. > > Thanks, > Kai > > --------------------------------------------------------------------------- > - -- > Nokia and AT&T present the 2010 Calling All Innovators-North America > contest Create new apps & games for the Nokia N8 for consumers in U.S. > and Canada $10 million total in prizes - $4M cash, 500 devices, nearly $6M > in marketing Develop with Nokia Qt SDK, Web Runtime, or Java and Publish > to Ovi Store http://p.sf.net/sfu/nokia-dev2dev > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users |
