Menu
▾
▴
Re: [Moosefs-users] mfsmaster performance and hardware
|
From: Davies L. <dav...@gm...> - 2011-08-31 06:19:38
|
Not yet, but we can export parts of mfsmount, then create Python or Go binding of it. Davies On Wed, Aug 31, 2011 at 11:18 AM, Robert Sandilands <rsa...@ne...>wrote: > There is a native API? Where can I find information about it? Or do you > have to reverse it from the code? > > Robert > > > On 8/30/11 10:42 PM, Davies Liu wrote: > > The bottle neck is FUSE and mfsmount, you should try to use native API ( > borrowed from mfsmount) > of MFS to re-implement a HTTP server, one socket per thread , or sockets > pool. > > I just want do it in Go, may by python is easier. > > Davies > > On Wed, Aug 31, 2011 at 8:54 AM, Robert Sandilands <rsa...@ne...>wrote: > >> Further on this subject. >> >> I wrote a dedicated http server to serve the files instead of using >> Apache. It allowed me to gain a few extra percent of performance and >> decreased the memory usage of the web servers. The web server also gave me >> some interesting timings: >> >> File open average 405.3732 ms >> File read average 238.7784 ms >> File close average 286.8376 ms >> File size average 0.0026 ms >> Net read average 2.536 ms >> Net write average 2.2148 ms >> Log to access log average 0.2526 ms >> Log to error log average 0.2234 ms >> >> Average time to process a file 936.2186 ms >> Total files processed 1,503,610 >> >> What I really find scary is that to open a file takes nearly half a >> second. To close a file a quarter of a second. The time to open() and >> close() is nearly 3 times more than the time to read the data. The server >> always reads in multiples of 64 kB except if there are less data available. >> Although it uses posix_fadvise() to try and do some read-ahead. This is the >> average over 5 machines running mfsmount and my custom web server running >> for about 18 hours. >> >> On a machine that only serves a low number of clients the times for open >> and close are negligible. open() and close() seems to scale very badly with >> an increase in clients using mfsmount. >> >> From looking at the code for mfsmount it seems like all communication to >> the master happens over a single TCP socket with a global handle and mutex >> to protect it. This may be the bottle neck? If there are multiple open()'s >> at the same time they may end up waiting for the mutex to get an opportunity >> to communicate with the master? The same handle and mutex is also used to >> read replies and this may also not help the situation? >> >> What prevents multiple sockets to the master? >> >> It also seems to indicate that the only way to get the open() average down >> is to introduce more web servers and that a single web server can only serve >> a very low number of clients. Is that a correct assumption? >> >> >> Robert >> >> On 8/26/11 3:25 AM, Davies Liu wrote: >> >> Hi Robert, >> >> Another hint to make mfsmaster more responsive is to locate the >> metadata.mfs >> on a separated disk with change logs, such as SAS array, then you should >> modify >> the source code of mfsmaster to do this. >> >> PS: what is the average size of you files? MooseFS (like GFS) is >> designed for >> large file (100M+), it can not serve well for amount of small files. >> Haystack from >> Facebook may be the better choice. We (douban.com) use MooseFS to serve >> 200+T(1M files) offline data and beansdb [1] to serve 500 million online >> small >> files, it performs very well. >> >> [1]: http://code.google.com/p/*beansdb*/ >> >> Davies >> >> On Fri, Aug 26, 2011 at 9:08 AM, Robert Sandilands <rsa...@ne... >> > wrote: >> >>> Hi Elliot, >>> >>> There is nothing in the code to change the priority. >>> >>> Taking virtually all other load from the chunk and master servers seems >>> to have improved this significantly. I still see timeouts from mfsmount, >>> but not enough to be problematic. >>> >>> To try and optimize the performance I am experimenting with accessing >>> the data using different APIs and block sizes but this has been >>> inconclusive. I have tried the effect of posix_fadvise(), sendfile() and >>> different sized buffers for read(). I still want to try mmap(). >>> Sendfile() did seem to be slightly slower than read(). >>> >>> Robert >>> >>> On 8/24/11 11:05 AM, Elliot Finley wrote: >>> > On Tue, Aug 9, 2011 at 6:46 PM, Robert Sandilands< >>> rsa...@ne...> wrote: >>> >> Increasing the swap space fixed the fork() issue. It seems that you >>> have to >>> >> ensure that memory available is always double the memory needed by >>> >> mfsmaster. None of the swap space was used over the last 24 hours. >>> >> >>> >> This did solve the extreme comb-like behavior of mfsmaster. It still >>> does >>> >> not resolve its sensitivity to load on the server. I am still seeing >>> >> timeouts on the chunkservers and mounts on the hour due to the high >>> CPU and >>> >> I/O load when the meta data is dumped to disk. It did however decrease >>> >> significantly. >>> > Here is another thought on this... >>> > >>> > The process is niced to -19 (very high priority) so that it has good >>> > performance. It forks once per hour to write out the metadata. I >>> > haven't checked the code for this, but is the forked process lowering >>> > it's priority so it doesn't compete with the original process? >>> > >>> > If it's not, it should be an easy code change to lower the priority in >>> > the child process (metadata writer) so that it doesn't compete with >>> > the original process at the same priority. >>> > >>> > If you check into this, I'm sure the list would appreciate an update. >>> :) >>> > >>> > Elliot >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> EMC VNX: the world's simplest storage, starting under $10K >>> The only unified storage solution that offers unified management >>> Up to 160% more powerful than alternatives and 25% more efficient. >>> Guaranteed. http://p.sf.net/sfu/emc-vnx-dev2dev >>> _______________________________________________ >>> moosefs-users mailing list >>> moo...@li... >>> https://lists.sourceforge.net/lists/listinfo/moosefs-users >>> >> >> >> >> -- >> - Davies >> >> >> >> >> ------------------------------------------------------------------------------ >> Special Offer -- Download ArcSight Logger for FREE! >> Finally, a world-class log management solution at an even better >> price-free! And you'll get a free "Love Thy Logs" t-shirt when you >> download Logger. Secure your free ArcSight Logger TODAY! >> http://p.sf.net/sfu/arcsisghtdev2dev >> _______________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> >> > > > -- > - Davies > > > -- - Davies |
