Menu
▾
▴
Re: [MooseFS-Users] Dealing with millions of files
|
From: Casper L. <cas...@pr...> - 2017-09-13 09:37:37
|
"There is some swapping taking place" What does that mean? A few MB's of swapped out, or is your chunkserver actively swapping? I've configured my chunk servers to a memory size that allows about 85% of unused memory for processes. My mfschunkserver processes use about 10% of available memory. Actual memory as there is almost no swap being used <0.1%. This 80% gets used by the kernels disk cache. Increasing memory available for disk caches could speed up your cluster even more if the size of your 'hot' zone of popular files is bigger and your read/write ratio is higher (more reads). Any less than 80% for disk cache will probably have a significant impact on disks, especially on the old rotating ones. -- Casper 2017-09-12 21:42 GMT+02:00 Wilson, Steven M <st...@pu...>: > The chunk servers have at least 64GB of memory in each one. There is some > swapping taking place but I set the LOCK_MEMORY option in > mfschunkserver.cfg to avoid swapping out the chunk server process. A quick > check shows that the most memory being used by any mfschunkserver is about > 55% on one of the 64GB chunk servers. > > > The ls and cat (both to /dev/null) seem fairly responsive: > > ls takes 0.169s on a directory of 57K files > > cat takes 0.363s on a 308KB file > > > It looks like the write speed for small files may be what's killing me. I > tried copying 900 small files (308KB) to another directory on the same > MooseFS filesystem and it took 273 seconds or about 1.1MB/s. Ouch! > > > Three of the four chunk servers are doing an internal rebalance of > chunks. Perhaps that's having a larger impact on my overall > I/O performance than I expected. > > > Steve > > > ------------------------------ > *From:* est...@gm... <est...@gm...> on behalf of Matt > Welland <mat...@gm...> > *Sent:* Tuesday, September 12, 2017 2:06 PM > *To:* Wilson, Steven M > *Cc:* moo...@li... > *Subject:* Re: [MooseFS-Users] Dealing with millions of files > > To ask the obvious ... you don't explicitly state that the chunk servers > have plenty of memory and are not hitting swap, are they fine? > > Can you tell from the logs or cgi web view if the slowness comes from > metadata queries or from retrieving the chunks? Are other operations > equally slow? E.g. if you change to one of the directories that you are > tarring up, do an ls and then cat a file to /dev/null are both the ls and > cat slow or just one or the other? > > On Tue, Sep 12, 2017 at 9:48 AM, Wilson, Steven M <st...@pu...> > wrote: > >> Hi, >> >> >> We have eight different MooseFS installations at our site and they are >> all doing terrific (thanks, MooseFS developers!) except for the one that >> hosts around 180 million files. The performance on this installation is >> quite dismal. I expect to see some decrease in performance due to the >> sheer number of files but not to the degree that I'm experiencing. For >> example, I am in the process of tarring/gzipping 176 directories that >> together contain 15M files and occupy 5.5TB of space. I am about 78% done >> after 25 days. >> >> >> Are there any users on this list who have MooseFS installations with >> large numbers of files and have some hints they can share about how to >> improve performance? >> >> >> Here is some basic information about this installation: >> >> * MooseFS master is running on a Xeon E5-1630v3 (3.7GHz) and 132GB of >> memory. Its system disk (where /var/lib/mfs is located) is a 256GB Samsung >> 850 Pro SSD. >> >> * Four chunk servers, two with 16 SATA disks ranging from 3TB to 10TB >> each. And two chunk servers with two 10TB SATA disks each. All disks are >> JBOD and most are formatted using XFS (some are ext4). >> >> * Disk schedulers on chunk servers are set to use deadline. Tried >> different settings for read_ahead_kb and nr_requests without much >> performance gain. >> >> * Network is gigabit Ethernet >> >> * Master and chunk servers are running a mix of MooseFS >> versions 3.0.90 and 3.0.92. >> >> >> Thanks in advance for any suggestions! >> >> >> Steve >> >> ------------------------------------------------------------ >> ------------------ >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> <http://secure-web.cisco.com/1HJ9vXYy_0YCboBJl7cH9g4OKnzybhwhU5E-KN6mKbzvJTv_0EFZoTbu57nNPgOyGXdiBI7Ebib354Zs5lp-DeQjKWiaM1BVgwizyNKZ4igyNhSqyPHbGJM_6dOqp4CGSi-r8RR9rF-Mujd7bs-SrArr-JrGjSfuC8h5MZYDExbEorjdK_0MRbDP-2ELiGVwCtRtA_FTYr9VwrcoBJ_t5RBF-ic3-HaKwB5KgVcZUbp5iaLIRu87Ql4ezlf7SQCgc5PH1v5sbazeuVyNpybOA3Pn6UAac0KvRIq9cmTGCClj-Pz1ph5SLaxVMqbj1MZHbY64h-Cwp9EAMw_0qk3pvPD8S6LE4uWguvaszNlInxoQ/http%3A%2F%2Fsdm.link%2Fslashdot> >> _________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> <https://secure-web.cisco.com/1nhR7evg0-W5aGax886YeNrvlUfVHI5tJY16u--TYA7btVPw6zu7QElZxalaz7TojfSM7i7jPnAZks1Lq3MfUXyMP71hFwoB9HGHfRBHj30KJUpaMptBTruXnIswidqiqxPomN_7qPIUUBNhXNHIh0NaCmf3SkwoVfru5MU8yJ1RJoG-5QL3q7LabhYLOqKsuhXsXW_RMFcRpfGVLsIOC-xS0-ahznXLJ8_jBLuJc33osnPzwSJgPejlaMgkfuDLtIUsJti9fzlDpBapBw4RcixpGbsncoBZm94djVyikbrp5C7fMgQh9ZeIGZ5pxxr_YgnVt180JS1yRdxZiGn8IZVgI1eLr2NsIZyWsUQDxGWU/https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fmoosefs-users> >> >> > > ------------------------------------------------------------ > ------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > > |
