Menu
▾
▴
Re: [Moosefs-users] mfsmaster performance and hardware
|
From: Davies L. <dav...@gm...> - 2011-08-31 02:42:54
|
The bottle neck is FUSE and mfsmount, you should try to use native API ( borrowed from mfsmount) of MFS to re-implement a HTTP server, one socket per thread , or sockets pool. I just want do it in Go, may by python is easier. Davies On Wed, Aug 31, 2011 at 8:54 AM, Robert Sandilands <rsa...@ne...>wrote: > Further on this subject. > > I wrote a dedicated http server to serve the files instead of using Apache. > It allowed me to gain a few extra percent of performance and decreased the > memory usage of the web servers. The web server also gave me some > interesting timings: > > File open average 405.3732 ms > File read average 238.7784 ms > File close average 286.8376 ms > File size average 0.0026 ms > Net read average 2.536 ms > Net write average 2.2148 ms > Log to access log average 0.2526 ms > Log to error log average 0.2234 ms > > Average time to process a file 936.2186 ms > Total files processed 1,503,610 > > What I really find scary is that to open a file takes nearly half a second. > To close a file a quarter of a second. The time to open() and close() is > nearly 3 times more than the time to read the data. The server always reads > in multiples of 64 kB except if there are less data available. Although it > uses posix_fadvise() to try and do some read-ahead. This is the average over > 5 machines running mfsmount and my custom web server running for about 18 > hours. > > On a machine that only serves a low number of clients the times for open > and close are negligible. open() and close() seems to scale very badly with > an increase in clients using mfsmount. > > From looking at the code for mfsmount it seems like all communication to > the master happens over a single TCP socket with a global handle and mutex > to protect it. This may be the bottle neck? If there are multiple open()'s > at the same time they may end up waiting for the mutex to get an opportunity > to communicate with the master? The same handle and mutex is also used to > read replies and this may also not help the situation? > > What prevents multiple sockets to the master? > > It also seems to indicate that the only way to get the open() average down > is to introduce more web servers and that a single web server can only serve > a very low number of clients. Is that a correct assumption? > > > Robert > > On 8/26/11 3:25 AM, Davies Liu wrote: > > Hi Robert, > > Another hint to make mfsmaster more responsive is to locate the > metadata.mfs > on a separated disk with change logs, such as SAS array, then you should > modify > the source code of mfsmaster to do this. > > PS: what is the average size of you files? MooseFS (like GFS) is designed > for > large file (100M+), it can not serve well for amount of small files. > Haystack from > Facebook may be the better choice. We (douban.com) use MooseFS to serve > 200+T(1M files) offline data and beansdb [1] to serve 500 million online > small > files, it performs very well. > > [1]: http://code.google.com/p/*beansdb*/ > > Davies > > On Fri, Aug 26, 2011 at 9:08 AM, Robert Sandilands <rsa...@ne...>wrote: > >> Hi Elliot, >> >> There is nothing in the code to change the priority. >> >> Taking virtually all other load from the chunk and master servers seems >> to have improved this significantly. I still see timeouts from mfsmount, >> but not enough to be problematic. >> >> To try and optimize the performance I am experimenting with accessing >> the data using different APIs and block sizes but this has been >> inconclusive. I have tried the effect of posix_fadvise(), sendfile() and >> different sized buffers for read(). I still want to try mmap(). >> Sendfile() did seem to be slightly slower than read(). >> >> Robert >> >> On 8/24/11 11:05 AM, Elliot Finley wrote: >> > On Tue, Aug 9, 2011 at 6:46 PM, Robert Sandilands<rsa...@ne...> >> wrote: >> >> Increasing the swap space fixed the fork() issue. It seems that you >> have to >> >> ensure that memory available is always double the memory needed by >> >> mfsmaster. None of the swap space was used over the last 24 hours. >> >> >> >> This did solve the extreme comb-like behavior of mfsmaster. It still >> does >> >> not resolve its sensitivity to load on the server. I am still seeing >> >> timeouts on the chunkservers and mounts on the hour due to the high CPU >> and >> >> I/O load when the meta data is dumped to disk. It did however decrease >> >> significantly. >> > Here is another thought on this... >> > >> > The process is niced to -19 (very high priority) so that it has good >> > performance. It forks once per hour to write out the metadata. I >> > haven't checked the code for this, but is the forked process lowering >> > it's priority so it doesn't compete with the original process? >> > >> > If it's not, it should be an easy code change to lower the priority in >> > the child process (metadata writer) so that it doesn't compete with >> > the original process at the same priority. >> > >> > If you check into this, I'm sure the list would appreciate an update. :) >> > >> > Elliot >> >> >> >> ------------------------------------------------------------------------------ >> EMC VNX: the world's simplest storage, starting under $10K >> The only unified storage solution that offers unified management >> Up to 160% more powerful than alternatives and 25% more efficient. >> Guaranteed. http://p.sf.net/sfu/emc-vnx-dev2dev >> _______________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> > > > > -- > - Davies > > > > > ------------------------------------------------------------------------------ > Special Offer -- Download ArcSight Logger for FREE! > Finally, a world-class log management solution at an even better > price-free! And you'll get a free "Love Thy Logs" t-shirt when you > download Logger. Secure your free ArcSight Logger TODAY! > http://p.sf.net/sfu/arcsisghtdev2dev > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > > -- - Davies |
