Menu
▾
▴
Re: [Moosefs-users] Write starvation
|
From: Robert <rsa...@ne...> - 2011-06-29 20:33:14
|
Yes, it is rather disappointing how Linux slows down on large folders.Not only that but some file systems like ext2/3/4 limits the number of files in a folder to a low number (16k/16k/64k). Generally we store files in a hash based structure where the first n characters of the files hash is the first folder name. The next m characters of the files hash is the second folder name and then the filename is the hash of the file. This prevents us from having a too large number of files in a folder, avoids the number of files per folder limits and slowdowns and prevents a too deep directory structure. It also gives us some deduplication. Yes, we use Centos, but installing and using the ktune package generally resolves most of the performance issues and differences I have seen with Ubuntu/Debian. It also helps to disable the cpuspeed daemon if you know the server does not have much down time. I don't understand the comment on hitting metadata a lot? What is a lot? Why would it make a difference? All the metadata is in RAM anyway? The biggest limit to speed seems to be the number of IOPS that you can get out of your disks you have available to you. Looking up the metadata from RAM should be several orders of magnitude faster than that. The activity reported through the CGI interface on the master is around 2,400 opens per minute average. Reads and writes are also around 2400 per minute alternating with each other. mknod has some peaks around 2,800 per minute but is generally much lower. Lookup's are around 8,000 per minute and getattr is around 700 per minute. Chunk replication and deletion is around 50 per minute. The other numbers are generally very low. Is there a guide/hints specific to MooseFS on what IO/Net/Process parameters would be good to investigate for mfsmaster? Robert -----Original Message----- From: Robert Dye <ro...@in...> To: 'Robert' <rsa...@ne...>; moosefs-users <moo...@li...> Sent: Wed, Jun 29, 2011 3:00 pm Subject: RE: [Moosefs-users] Write starvation As Ricardo mentioned below, you could behitting the metadata, a lot. When I ran benchmarks for enumerations in adirectory with more than 10,000 files, it was an obvious slow down incomparison to a directory with only a few files. Not sure which OS you are running, but Iwould tweak the many IO/Net/Process parameters. From personal experience, Ihad a huge slow down with mfsmasterserver when running on CentOS. I have sincemoved to Ubuntu, which appears to be faster out of the box. -Rob From: Robert[mailto:rsa...@ne...] Sent: Wednesday, June 29, 201111:29 AM To: ric...@da...;moo...@li... Subject: Re: [Moosefs-users] Writestarvation Yes,the master server is working hard. But I would still expect a somewhat fairdistribution of load between read and write. The specs: 2 x quad core Xeon E5405 @ 2GHz. 64 GB of RAM 32 x 2 TB 7200 RPM SATA disks 68 million file system objects 65.4 million files No swap is being used mfsmaster is using 23 GB of RAM. Robert -----OriginalMessage----- From: Ricardo J. Barberis <ric...@da...> To: moosefs-users <moo...@li...> Sent: Wed, Jun 29, 2011 12:08 pm Subject: Re: [Moosefs-users] Write starvation El Martes 28 Junio 2011, Robert Sandilands escribió: > Write traffic does not stop completely, but seems to slow down to < 10 > kB per second under high read traffic conditions. When the read traffic > decreases the write traffic will increase to normal levels. > > Is this a known problem? Is there something I can do to ensure that > write traffic is not starved by read traffic? > > Robert What are the master machine specs, and how many files do you have already on MFS? You might be hiting the master too hard with metadata. Regards, -- Ricardo J. Barberis Senior SysAdmin / ITI Dattatec.com :: Soluciones de Web Hosting Tu Hosting hecho Simple! ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
