Menu
▾
▴
moosefs-users
|
From: Robert S. <rsa...@ne...> - 2011-06-29 02:30:32
|
I have been moving data from existing non-distributed file systems onto a MooseFS file system. I am using 1.6.20 on Centos 5.6. While moving the data I have also transfered some of the normal read-only traffic load to use the data already moved onto the MFS volume. What I can see is when there is any significant read traffic then write traffic slows down to a crawl. When I look at the server charts for any of the chunk servers generated by mfscgiserv then it seems like read and write traffic seems to alternate. Write traffic does not stop completely, but seems to slow down to < 10 kB per second under high read traffic conditions. When the read traffic decreases the write traffic will increase to normal levels. Is this a known problem? Is there something I can do to ensure that write traffic is not starved by read traffic? Robert |
|
From: Ricardo J. B. <ric...@da...> - 2011-06-29 16:08:48
|
El Martes 28 Junio 2011, Robert Sandilands escribió: > Write traffic does not stop completely, but seems to slow down to < 10 > kB per second under high read traffic conditions. When the read traffic > decreases the write traffic will increase to normal levels. > > Is this a known problem? Is there something I can do to ensure that > write traffic is not starved by read traffic? > > Robert What are the master machine specs, and how many files do you have already on MFS? You might be hiting the master too hard with metadata. Regards, -- Ricardo J. Barberis Senior SysAdmin / ITI Dattatec.com :: Soluciones de Web Hosting Tu Hosting hecho Simple! |
|
From: Robert <rsa...@ne...> - 2011-06-29 18:45:25
|
Yes, the master server is working hard. But I would still expect a somewhat fair distribution of load between read and write. The specs: 2 x quad core Xeon E5405 @ 2GHz. 64 GB of RAM 32 x 2 TB 7200 RPM SATA disks 68 million file system objects 65.4 million files No swap is being used mfsmaster is using 23 GB of RAM. Robert -----Original Message----- From: Ricardo J. Barberis <ric...@da...> To: moosefs-users <moo...@li...> Sent: Wed, Jun 29, 2011 12:08 pm Subject: Re: [Moosefs-users] Write starvation El Martes 28 Junio 2011, Robert Sandilands escribió: > Write traffic does not stop completely, but seems to slow down to < 10 > kB per second under high read traffic conditions. When the read traffic > decreases the write traffic will increase to normal levels. > > Is this a known problem? Is there something I can do to ensure that > write traffic is not starved by read traffic? > > Robert What are the master machine specs, and how many files do you have already on MFS? You might be hiting the master too hard with metadata. Regards, -- Ricardo J. Barberis Senior SysAdmin / ITI Dattatec.com :: Soluciones de Web Hosting Tu Hosting hecho Simple! ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Robert D. <ro...@in...> - 2011-06-29 19:00:33
|
As Ricardo mentioned below, you could be hitting the metadata, a lot. When I ran benchmarks for enumerations in a directory with more than 10,000 files, it was an obvious slow down in comparison to a directory with only a few files. Not sure which OS you are running, but I would tweak the many IO/Net/Process parameters. From personal experience, I had a huge slow down with mfsmasterserver when running on CentOS. I have since moved to Ubuntu, which appears to be faster out of the box. -Rob _____ From: Robert [mailto:rsa...@ne...] Sent: Wednesday, June 29, 2011 11:29 AM To: ric...@da...; moo...@li... Subject: Re: [Moosefs-users] Write starvation Yes, the master server is working hard. But I would still expect a somewhat fair distribution of load between read and write. The specs: 2 x quad core Xeon E5405 @ 2GHz. 64 GB of RAM 32 x 2 TB 7200 RPM SATA disks 68 million file system objects 65.4 million files No swap is being used mfsmaster is using 23 GB of RAM. Robert -----Original Message----- From: Ricardo J. Barberis <ric...@da...> To: moosefs-users <moo...@li...> Sent: Wed, Jun 29, 2011 12:08 pm Subject: Re: [Moosefs-users] Write starvation El Martes 28 Junio 2011, Robert Sandilands escribió: > Write traffic does not stop completely, but seems to slow down to < 10 > kB per second under high read traffic conditions. When the read traffic > decreases the write traffic will increase to normal levels. > > Is this a known problem? Is there something I can do to ensure that > write traffic is not starved by read traffic? > > Robert What are the master machine specs, and how many files do you have already on MFS? You might be hiting the master too hard with metadata. Regards, -- Ricardo J. Barberis Senior SysAdmin / ITI Dattatec.com :: Soluciones de Web Hosting Tu Hosting hecho Simple! ---------------------------------------------------------------------------- -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Robert <rsa...@ne...> - 2011-06-29 20:33:14
|
Yes, it is rather disappointing how Linux slows down on large folders.Not only that but some file systems like ext2/3/4 limits the number of files in a folder to a low number (16k/16k/64k). Generally we store files in a hash based structure where the first n characters of the files hash is the first folder name. The next m characters of the files hash is the second folder name and then the filename is the hash of the file. This prevents us from having a too large number of files in a folder, avoids the number of files per folder limits and slowdowns and prevents a too deep directory structure. It also gives us some deduplication. Yes, we use Centos, but installing and using the ktune package generally resolves most of the performance issues and differences I have seen with Ubuntu/Debian. It also helps to disable the cpuspeed daemon if you know the server does not have much down time. I don't understand the comment on hitting metadata a lot? What is a lot? Why would it make a difference? All the metadata is in RAM anyway? The biggest limit to speed seems to be the number of IOPS that you can get out of your disks you have available to you. Looking up the metadata from RAM should be several orders of magnitude faster than that. The activity reported through the CGI interface on the master is around 2,400 opens per minute average. Reads and writes are also around 2400 per minute alternating with each other. mknod has some peaks around 2,800 per minute but is generally much lower. Lookup's are around 8,000 per minute and getattr is around 700 per minute. Chunk replication and deletion is around 50 per minute. The other numbers are generally very low. Is there a guide/hints specific to MooseFS on what IO/Net/Process parameters would be good to investigate for mfsmaster? Robert -----Original Message----- From: Robert Dye <ro...@in...> To: 'Robert' <rsa...@ne...>; moosefs-users <moo...@li...> Sent: Wed, Jun 29, 2011 3:00 pm Subject: RE: [Moosefs-users] Write starvation As Ricardo mentioned below, you could behitting the metadata, a lot. When I ran benchmarks for enumerations in adirectory with more than 10,000 files, it was an obvious slow down incomparison to a directory with only a few files. Not sure which OS you are running, but Iwould tweak the many IO/Net/Process parameters. From personal experience, Ihad a huge slow down with mfsmasterserver when running on CentOS. I have sincemoved to Ubuntu, which appears to be faster out of the box. -Rob From: Robert[mailto:rsa...@ne...] Sent: Wednesday, June 29, 201111:29 AM To: ric...@da...;moo...@li... Subject: Re: [Moosefs-users] Writestarvation Yes,the master server is working hard. But I would still expect a somewhat fairdistribution of load between read and write. The specs: 2 x quad core Xeon E5405 @ 2GHz. 64 GB of RAM 32 x 2 TB 7200 RPM SATA disks 68 million file system objects 65.4 million files No swap is being used mfsmaster is using 23 GB of RAM. Robert -----OriginalMessage----- From: Ricardo J. Barberis <ric...@da...> To: moosefs-users <moo...@li...> Sent: Wed, Jun 29, 2011 12:08 pm Subject: Re: [Moosefs-users] Write starvation El Martes 28 Junio 2011, Robert Sandilands escribió: > Write traffic does not stop completely, but seems to slow down to < 10 > kB per second under high read traffic conditions. When the read traffic > decreases the write traffic will increase to normal levels. > > Is this a known problem? Is there something I can do to ensure that > write traffic is not starved by read traffic? > > Robert What are the master machine specs, and how many files do you have already on MFS? You might be hiting the master too hard with metadata. Regards, -- Ricardo J. Barberis Senior SysAdmin / ITI Dattatec.com :: Soluciones de Web Hosting Tu Hosting hecho Simple! ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Ricardo J. B. <ric...@da...> - 2011-06-30 15:33:11
|
El Miércoles 29 Junio 2011, Robert escribió: > Yes, we use Centos, but installing and using the ktune package generally > resolves most of the performance issues and differences I have seen with > Ubuntu/Debian. Nice to know about ktune and thank you for bringing it up, I'll take a look a it. > I don't understand the comment on hitting metadata a lot? What is a lot? A lot = reading / (re)writing / ls -l'ing / stat'ing too often. If the client can't cache the metadata but uses it often, that means it has to query the master every time. Network latencies might also play a role in the performance degradation. > Why would it make a difference? All the metadata is in RAM anyway? The > biggest limit to speed seems to be the number of IOPS that you can get out > of your disks you have available to you. Looking up the metadata from RAM > should be several orders of magnitude faster than that. Yep, and you have plenty of RAM, so that shouldn't be an issue in your case. > The activity reported through the CGI interface on the master is around > 2,400 opens per minute average. Reads and writes are also around 2400 per > minute alternating with each other. mknod has some peaks around 2,800 per > minute but is generally much lower. Lookup's are around 8,000 per minute > and getattr is around 700 per minute. Chunk replication and deletion is > around 50 per minute. The other numbers are generally very low. Mmm, maybe 2 chunkservers are just too litle to handle that activity but I would also check the network latencies. I'm also not really confident about having master and cunkserver on the same server but I don't have any hard evidence to support my feelings ;) > Is there a guide/hints specific to MooseFS on what IO/Net/Process > parameters would be good to investigate for mfsmaster? I'd like to know that too! Cheers, -- Ricardo J. Barberis Senior SysAdmin / ITI Dattatec.com :: Soluciones de Web Hosting Tu Hosting hecho Simple! |
|
From: Robert S. <rsa...@ne...> - 2011-07-02 01:09:53
|
Based on some tests I think the limit in this case is the number of opens per minute. I think I need to understand what happens with an open before I can make guesses on what can be done to get the number higher. But then it still does not quite explain the write starvation except if the number of pending reads are just so much higher than the number of pending writes that it seems to starve the writes. Maybe this will resolve itself as I add more chunk servers. Some questions: 1. Is there a limit to the number of handles that client applications can open per mount, per chunk server, per disk? 2. What happens when an application does fopen() on a mount? Can somebody give a quick overview or do I have to read some code? Robert On 6/30/11 11:32 AM, Ricardo J. Barberis wrote: > El Miércoles 29 Junio 2011, Robert escribió: >> Yes, we use Centos, but installing and using the ktune package generally >> resolves most of the performance issues and differences I have seen with >> Ubuntu/Debian. > Nice to know about ktune and thank you for bringing it up, I'll take a look a > it. > >> I don't understand the comment on hitting metadata a lot? What is a lot? > A lot = reading / (re)writing / ls -l'ing / stat'ing too often. > > If the client can't cache the metadata but uses it often, that means it has to > query the master every time. > > Network latencies might also play a role in the performance degradation. > >> Why would it make a difference? All the metadata is in RAM anyway? The >> biggest limit to speed seems to be the number of IOPS that you can get out >> of your disks you have available to you. Looking up the metadata from RAM >> should be several orders of magnitude faster than that. > Yep, and you have plenty of RAM, so that shouldn't be an issue in your case. > >> The activity reported through the CGI interface on the master is around >> 2,400 opens per minute average. Reads and writes are also around 2400 per >> minute alternating with each other. mknod has some peaks around 2,800 per >> minute but is generally much lower. Lookup's are around 8,000 per minute >> and getattr is around 700 per minute. Chunk replication and deletion is >> around 50 per minute. The other numbers are generally very low. > Mmm, maybe 2 chunkservers are just too litle to handle that activity but I > would also check the network latencies. > > I'm also not really confident about having master and cunkserver on the same > server but I don't have any hard evidence to support my feelings ;) > >> Is there a guide/hints specific to MooseFS on what IO/Net/Process >> parameters would be good to investigate for mfsmaster? > I'd like to know that too! > > Cheers, |
|
From: Michal B. <mic...@ge...> - 2011-07-05 07:15:35
|
Hi Robert! Ad. 1. There is no limit in mfsmount itself, but there are some limits in the operating system. Generally speaking it is wise not to open more than several thousands files in parallel. Ad. 2. Fopen invokes open, and open invokes (through kernel and FUSE) functions mfs_lookup and mfs_open. Mfs_lookup function changes consequtive path elements into i-node number. While mfs_open makes the target file opening. It sends a packet to the master in order to receive information about possibility to keep the file in the cache. It also marks the file in the master as opened - in cases it is deleted, it is sustained to the moment of closing. BTW. Why do you need this? Kind regards Michał Borychowski MooseFS Support Manager _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Gemius S.A. ul. Wołoska 7, 02-672 Warszawa Budynek MARS, klatka D Tel.: +4822 874-41-00 Fax : +4822 874-41-01 -----Original Message----- From: Robert Sandilands [mailto:rsa...@ne...] Sent: Saturday, July 02, 2011 2:54 AM To: moo...@li... Subject: Re: [Moosefs-users] Write starvation Based on some tests I think the limit in this case is the number of opens per minute. I think I need to understand what happens with an open before I can make guesses on what can be done to get the number higher. But then it still does not quite explain the write starvation except if the number of pending reads are just so much higher than the number of pending writes that it seems to starve the writes. Maybe this will resolve itself as I add more chunk servers. Some questions: 1. Is there a limit to the number of handles that client applications can open per mount, per chunk server, per disk? 2. What happens when an application does fopen() on a mount? Can somebody give a quick overview or do I have to read some code? Robert On 6/30/11 11:32 AM, Ricardo J. Barberis wrote: > El Miércoles 29 Junio 2011, Robert escribió: >> Yes, we use Centos, but installing and using the ktune package generally >> resolves most of the performance issues and differences I have seen with >> Ubuntu/Debian. > Nice to know about ktune and thank you for bringing it up, I'll take a look a > it. > >> I don't understand the comment on hitting metadata a lot? What is a lot? > A lot = reading / (re)writing / ls -l'ing / stat'ing too often. > > If the client can't cache the metadata but uses it often, that means it has to > query the master every time. > > Network latencies might also play a role in the performance degradation. > >> Why would it make a difference? All the metadata is in RAM anyway? The >> biggest limit to speed seems to be the number of IOPS that you can get out >> of your disks you have available to you. Looking up the metadata from RAM >> should be several orders of magnitude faster than that. > Yep, and you have plenty of RAM, so that shouldn't be an issue in your case. > >> The activity reported through the CGI interface on the master is around >> 2,400 opens per minute average. Reads and writes are also around 2400 per >> minute alternating with each other. mknod has some peaks around 2,800 per >> minute but is generally much lower. Lookup's are around 8,000 per minute >> and getattr is around 700 per minute. Chunk replication and deletion is >> around 50 per minute. The other numbers are generally very low. > Mmm, maybe 2 chunkservers are just too litle to handle that activity but I > would also check the network latencies. > > I'm also not really confident about having master and cunkserver on the same > server but I don't have any hard evidence to support my feelings ;) > >> Is there a guide/hints specific to MooseFS on what IO/Net/Process >> parameters would be good to investigate for mfsmaster? > I'd like to know that too! > > Cheers, ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Robert S. <rsa...@ne...> - 2011-07-05 12:47:37
|
Hi Michal, I need this to see if there is a way I can optimize the system to open more files per minute. At this stage our systems can open a few hundred files in parallel. I am not yet at the point where I can do thousands. What I think I am seeing is that the writes are starved because there are too many pending opens and most of them are for reading files. There seems to be a limit of around 2,400 opens per minute on the hardware I have and I am looking at what needs to be done to improve that. Based on your answer it sounds like the network traffic from the machine running mfsmount() to the master may be the biggest delay? Short of converting to 10 GB/s or trying to get all the servers on the same switch I don't know if there is much to be done about it? Robert On 7/5/11 3:15 AM, Michal Borychowski wrote: > Hi Robert! > > Ad. 1. There is no limit in mfsmount itself, but there are some limits in the operating system. Generally speaking it is wise not to open more than several thousands files in parallel. > > Ad. 2. Fopen invokes open, and open invokes (through kernel and FUSE) functions mfs_lookup and mfs_open. Mfs_lookup function changes consequtive path elements into i-node number. While mfs_open makes the target file opening. It sends a packet to the master in order to receive information about possibility to keep the file in the cache. It also marks the file in the master as opened - in cases it is deleted, it is sustained to the moment of closing. > > BTW. Why do you need this? > > > Kind regards > Michał Borychowski > MooseFS Support Manager > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > Gemius S.A. > ul. Wołoska 7, 02-672 Warszawa > Budynek MARS, klatka D > Tel.: +4822 874-41-00 > Fax : +4822 874-41-01 > > > > -----Original Message----- > From: Robert Sandilands [mailto:rsa...@ne...] > Sent: Saturday, July 02, 2011 2:54 AM > To: moo...@li... > Subject: Re: [Moosefs-users] Write starvation > > Based on some tests I think the limit in this case is the number of > opens per minute. I think I need to understand what happens with an open > before I can make guesses on what can be done to get the number higher. > > But then it still does not quite explain the write starvation except if > the number of pending reads are just so much higher than the number of > pending writes that it seems to starve the writes. Maybe this will > resolve itself as I add more chunk servers. > > Some questions: > > 1. Is there a limit to the number of handles that client applications > can open per mount, per chunk server, per disk? > 2. What happens when an application does fopen() on a mount? Can > somebody give a quick overview or do I have to read some code? > > Robert > > On 6/30/11 11:32 AM, Ricardo J. Barberis wrote: >> El Miércoles 29 Junio 2011, Robert escribió: >>> Yes, we use Centos, but installing and using the ktune package generally >>> resolves most of the performance issues and differences I have seen with >>> Ubuntu/Debian. >> Nice to know about ktune and thank you for bringing it up, I'll take a look a >> it. >> >>> I don't understand the comment on hitting metadata a lot? What is a lot? >> A lot = reading / (re)writing / ls -l'ing / stat'ing too often. >> >> If the client can't cache the metadata but uses it often, that means it has to >> query the master every time. >> >> Network latencies might also play a role in the performance degradation. >> >>> Why would it make a difference? All the metadata is in RAM anyway? The >>> biggest limit to speed seems to be the number of IOPS that you can get out >>> of your disks you have available to you. Looking up the metadata from RAM >>> should be several orders of magnitude faster than that. >> Yep, and you have plenty of RAM, so that shouldn't be an issue in your case. >> >>> The activity reported through the CGI interface on the master is around >>> 2,400 opens per minute average. Reads and writes are also around 2400 per >>> minute alternating with each other. mknod has some peaks around 2,800 per >>> minute but is generally much lower. Lookup's are around 8,000 per minute >>> and getattr is around 700 per minute. Chunk replication and deletion is >>> around 50 per minute. The other numbers are generally very low. >> Mmm, maybe 2 chunkservers are just too litle to handle that activity but I >> would also check the network latencies. >> >> I'm also not really confident about having master and cunkserver on the same >> server but I don't have any hard evidence to support my feelings ;) >> >>> Is there a guide/hints specific to MooseFS on what IO/Net/Process >>> parameters would be good to investigate for mfsmaster? >> I'd like to know that too! >> >> Cheers, > > ------------------------------------------------------------------------------ > All of the data generated in your IT infrastructure is seriously valuable. > Why? It contains a definitive record of application performance, security > threats, fraudulent activity, and more. Splunk takes this data and makes > sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-d2d-c2 > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > |
|
From: Michal B. <mic...@ge...> - 2011-07-11 13:29:49
|
Hi Robert! We are not sure that switching to 10Gb/s network can help much. Still, all open files will be opened network connections in mfsmount... I would recommend to check the need of opening so many files at one moment... Is it really necessary? Or maybe you can use some mechanisms like memcache, etc.? We don't see any other way to improve the situation of opening so many files. Regards -Michał -----Original Message----- From: Robert Sandilands [mailto:rsa...@ne...] Sent: Tuesday, July 05, 2011 2:32 PM To: Michal Borychowski Cc: moo...@li... Subject: Re: [Moosefs-users] Write starvation Hi Michal, I need this to see if there is a way I can optimize the system to open more files per minute. At this stage our systems can open a few hundred files in parallel. I am not yet at the point where I can do thousands. What I think I am seeing is that the writes are starved because there are too many pending opens and most of them are for reading files. There seems to be a limit of around 2,400 opens per minute on the hardware I have and I am looking at what needs to be done to improve that. Based on your answer it sounds like the network traffic from the machine running mfsmount() to the master may be the biggest delay? Short of converting to 10 GB/s or trying to get all the servers on the same switch I don't know if there is much to be done about it? Robert On 7/5/11 3:15 AM, Michal Borychowski wrote: > Hi Robert! > > Ad. 1. There is no limit in mfsmount itself, but there are some limits in the operating system. Generally speaking it is wise not to open more than several thousands files in parallel. > > Ad. 2. Fopen invokes open, and open invokes (through kernel and FUSE) functions mfs_lookup and mfs_open. Mfs_lookup function changes consequtive path elements into i-node number. While mfs_open makes the target file opening. It sends a packet to the master in order to receive information about possibility to keep the file in the cache. It also marks the file in the master as opened - in cases it is deleted, it is sustained to the moment of closing. > > BTW. Why do you need this? > > > Kind regards > Michał Borychowski > MooseFS Support Manager > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > Gemius S.A. > ul. Wołoska 7, 02-672 Warszawa > Budynek MARS, klatka D > Tel.: +4822 874-41-00 > Fax : +4822 874-41-01 > > > > -----Original Message----- > From: Robert Sandilands [mailto:rsa...@ne...] > Sent: Saturday, July 02, 2011 2:54 AM > To: moo...@li... > Subject: Re: [Moosefs-users] Write starvation > > Based on some tests I think the limit in this case is the number of > opens per minute. I think I need to understand what happens with an open > before I can make guesses on what can be done to get the number higher. > > But then it still does not quite explain the write starvation except if > the number of pending reads are just so much higher than the number of > pending writes that it seems to starve the writes. Maybe this will > resolve itself as I add more chunk servers. > > Some questions: > > 1. Is there a limit to the number of handles that client applications > can open per mount, per chunk server, per disk? > 2. What happens when an application does fopen() on a mount? Can > somebody give a quick overview or do I have to read some code? > > Robert > > On 6/30/11 11:32 AM, Ricardo J. Barberis wrote: >> El Miércoles 29 Junio 2011, Robert escribió: >>> Yes, we use Centos, but installing and using the ktune package generally >>> resolves most of the performance issues and differences I have seen with >>> Ubuntu/Debian. >> Nice to know about ktune and thank you for bringing it up, I'll take a look a >> it. >> >>> I don't understand the comment on hitting metadata a lot? What is a lot? >> A lot = reading / (re)writing / ls -l'ing / stat'ing too often. >> >> If the client can't cache the metadata but uses it often, that means it has to >> query the master every time. >> >> Network latencies might also play a role in the performance degradation. >> >>> Why would it make a difference? All the metadata is in RAM anyway? The >>> biggest limit to speed seems to be the number of IOPS that you can get out >>> of your disks you have available to you. Looking up the metadata from RAM >>> should be several orders of magnitude faster than that. >> Yep, and you have plenty of RAM, so that shouldn't be an issue in your case. >> >>> The activity reported through the CGI interface on the master is around >>> 2,400 opens per minute average. Reads and writes are also around 2400 per >>> minute alternating with each other. mknod has some peaks around 2,800 per >>> minute but is generally much lower. Lookup's are around 8,000 per minute >>> and getattr is around 700 per minute. Chunk replication and deletion is >>> around 50 per minute. The other numbers are generally very low. >> Mmm, maybe 2 chunkservers are just too litle to handle that activity but I >> would also check the network latencies. >> >> I'm also not really confident about having master and cunkserver on the same >> server but I don't have any hard evidence to support my feelings ;) >> >>> Is there a guide/hints specific to MooseFS on what IO/Net/Process >>> parameters would be good to investigate for mfsmaster? >> I'd like to know that too! >> >> Cheers, > > ------------------------------------------------------------------------------ > All of the data generated in your IT infrastructure is seriously valuable. > Why? It contains a definitive record of application performance, security > threats, fraudulent activity, and more. Splunk takes this data and makes > sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-d2d-c2 > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > ------------------------------------------------------------------------------ All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Robert S. <rsa...@ne...> - 2011-07-11 22:18:15
|
Hi Michal, Caching has not been a very successful path for us. With a 2 TB Squid cache we got < 35% cache hit rates. One of the things we need to do is to re-organize some of our processes to distribute the load a bit better. Our load do come in bursts and it would lead to other improvements if we can distribute the load better through time. But do I understand correctly that if I could distribute the opens to more mounts on more machines that I could expect to open more files simultaneously? That assumes the same mfsmaster and the same number of chunk servers. Is the limit/slowdown in mfsmount or mfsmaster? I know there is a limit on the disk speed and the number of spindles etc. But our disk utilization is currently < 50% according to "iostat -x". Even if I could distribute the load better, the total load will just increase with time and if there is a hard limit on the scalability of MooseFS then it may become a bit problematic in a year or two. Robert On 7/11/11 9:29 AM, Michal Borychowski wrote: > Hi Robert! > > We are not sure that switching to 10Gb/s network can help much. Still, all open files will be opened network connections in mfsmount... > > I would recommend to check the need of opening so many files at one moment... Is it really necessary? Or maybe you can use some mechanisms like memcache, etc.? > > We don't see any other way to improve the situation of opening so many files. > > > Regards > -Michał > > > -----Original Message----- > From: Robert Sandilands [mailto:rsa...@ne...] > Sent: Tuesday, July 05, 2011 2:32 PM > To: Michal Borychowski > Cc: moo...@li... > Subject: Re: [Moosefs-users] Write starvation > > Hi Michal, > > I need this to see if there is a way I can optimize the system to open > more files per minute. At this stage our systems can open a few hundred > files in parallel. I am not yet at the point where I can do thousands. > What I think I am seeing is that the writes are starved because there > are too many pending opens and most of them are for reading files. There > seems to be a limit of around 2,400 opens per minute on the hardware I > have and I am looking at what needs to be done to improve that. Based on > your answer it sounds like the network traffic from the machine running > mfsmount() to the master may be the biggest delay? Short of converting > to 10 GB/s or trying to get all the servers on the same switch I don't > know if there is much to be done about it? > > Robert > > On 7/5/11 3:15 AM, Michal Borychowski wrote: >> Hi Robert! >> >> Ad. 1. There is no limit in mfsmount itself, but there are some limits in the operating system. Generally speaking it is wise not to open more than several thousands files in parallel. >> >> Ad. 2. Fopen invokes open, and open invokes (through kernel and FUSE) functions mfs_lookup and mfs_open. Mfs_lookup function changes consequtive path elements into i-node number. While mfs_open makes the target file opening. It sends a packet to the master in order to receive information about possibility to keep the file in the cache. It also marks the file in the master as opened - in cases it is deleted, it is sustained to the moment of closing. >> >> BTW. Why do you need this? >> >> >> Kind regards >> Michał Borychowski >> MooseFS Support Manager >> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >> Gemius S.A. >> ul. Wołoska 7, 02-672 Warszawa >> Budynek MARS, klatka D >> Tel.: +4822 874-41-00 >> Fax : +4822 874-41-01 >> >> >> >> -----Original Message----- >> From: Robert Sandilands [mailto:rsa...@ne...] >> Sent: Saturday, July 02, 2011 2:54 AM >> To: moo...@li... >> Subject: Re: [Moosefs-users] Write starvation >> >> Based on some tests I think the limit in this case is the number of >> opens per minute. I think I need to understand what happens with an open >> before I can make guesses on what can be done to get the number higher. >> >> But then it still does not quite explain the write starvation except if >> the number of pending reads are just so much higher than the number of >> pending writes that it seems to starve the writes. Maybe this will >> resolve itself as I add more chunk servers. >> >> Some questions: >> >> 1. Is there a limit to the number of handles that client applications >> can open per mount, per chunk server, per disk? >> 2. What happens when an application does fopen() on a mount? Can >> somebody give a quick overview or do I have to read some code? >> >> Robert >> >> On 6/30/11 11:32 AM, Ricardo J. Barberis wrote: >>> El Miércoles 29 Junio 2011, Robert escribió: >>>> Yes, we use Centos, but installing and using the ktune package generally >>>> resolves most of the performance issues and differences I have seen with >>>> Ubuntu/Debian. >>> Nice to know about ktune and thank you for bringing it up, I'll take a look a >>> it. >>> >>>> I don't understand the comment on hitting metadata a lot? What is a lot? >>> A lot = reading / (re)writing / ls -l'ing / stat'ing too often. >>> >>> If the client can't cache the metadata but uses it often, that means it has to >>> query the master every time. >>> >>> Network latencies might also play a role in the performance degradation. >>> >>>> Why would it make a difference? All the metadata is in RAM anyway? The >>>> biggest limit to speed seems to be the number of IOPS that you can get out >>>> of your disks you have available to you. Looking up the metadata from RAM >>>> should be several orders of magnitude faster than that. >>> Yep, and you have plenty of RAM, so that shouldn't be an issue in your case. >>> >>>> The activity reported through the CGI interface on the master is around >>>> 2,400 opens per minute average. Reads and writes are also around 2400 per >>>> minute alternating with each other. mknod has some peaks around 2,800 per >>>> minute but is generally much lower. Lookup's are around 8,000 per minute >>>> and getattr is around 700 per minute. Chunk replication and deletion is >>>> around 50 per minute. The other numbers are generally very low. >>> Mmm, maybe 2 chunkservers are just too litle to handle that activity but I >>> would also check the network latencies. >>> >>> I'm also not really confident about having master and cunkserver on the same >>> server but I don't have any hard evidence to support my feelings ;) >>> >>>> Is there a guide/hints specific to MooseFS on what IO/Net/Process >>>> parameters would be good to investigate for mfsmaster? >>> I'd like to know that too! >>> >>> Cheers, >> ------------------------------------------------------------------------------ >> All of the data generated in your IT infrastructure is seriously valuable. >> Why? It contains a definitive record of application performance, security >> threats, fraudulent activity, and more. Splunk takes this data and makes >> sense of it. IT sense. And common sense. >> http://p.sf.net/sfu/splunk-d2d-c2 >> _______________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> > > ------------------------------------------------------------------------------ > All of the data generated in your IT infrastructure is seriously valuable. > Why? It contains a definitive record of application performance, security > threats, fraudulent activity, and more. Splunk takes this data and makes > sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-d2d-c2 > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > |
|
From: Robert S. <rsa...@ne...> - 2011-08-02 03:09:03
|
Has anybody looked at the effect on the performance and stability of MooseFS when you modify FUSE_MAX_BACKGROUND in the fuse kernel module? I notice that newer Linux kernels make this a tunable parameter but I don't want to go to a new kernel due to the khugepage problems reported with newer kernels in Centos 6 & Debian with fuse. This may contribute to the write starvation I have been seeing on our system. Adding more chunkservers does not seem to resolve the write starvation problem. Robert On 7/11/11 6:18 PM, Robert Sandilands wrote: > Hi Michal, > > Caching has not been a very successful path for us. With a 2 TB Squid > cache we got< 35% cache hit rates. One of the things we need to do is > to re-organize some of our processes to distribute the load a bit > better. Our load do come in bursts and it would lead to other > improvements if we can distribute the load better through time. > > But do I understand correctly that if I could distribute the opens to > more mounts on more machines that I could expect to open more files > simultaneously? That assumes the same mfsmaster and the same number of > chunk servers. Is the limit/slowdown in mfsmount or mfsmaster? I know > there is a limit on the disk speed and the number of spindles etc. But > our disk utilization is currently< 50% according to "iostat -x". > > Even if I could distribute the load better, the total load will just > increase with time and if there is a hard limit on the scalability of > MooseFS then it may become a bit problematic in a year or two. > > Robert > > On 7/11/11 9:29 AM, Michal Borychowski wrote: >> Hi Robert! >> >> We are not sure that switching to 10Gb/s network can help much. Still, all open files will be opened network connections in mfsmount... >> >> I would recommend to check the need of opening so many files at one moment... Is it really necessary? Or maybe you can use some mechanisms like memcache, etc.? >> >> We don't see any other way to improve the situation of opening so many files. >> >> >> Regards >> -Michał >> >> >> -----Original Message----- >> From: Robert Sandilands [mailto:rsa...@ne...] >> Sent: Tuesday, July 05, 2011 2:32 PM >> To: Michal Borychowski >> Cc: moo...@li... >> Subject: Re: [Moosefs-users] Write starvation >> >> Hi Michal, >> >> I need this to see if there is a way I can optimize the system to open >> more files per minute. At this stage our systems can open a few hundred >> files in parallel. I am not yet at the point where I can do thousands. >> What I think I am seeing is that the writes are starved because there >> are too many pending opens and most of them are for reading files. There >> seems to be a limit of around 2,400 opens per minute on the hardware I >> have and I am looking at what needs to be done to improve that. Based on >> your answer it sounds like the network traffic from the machine running >> mfsmount() to the master may be the biggest delay? Short of converting >> to 10 GB/s or trying to get all the servers on the same switch I don't >> know if there is much to be done about it? >> >> Robert >> >> On 7/5/11 3:15 AM, Michal Borychowski wrote: >>> Hi Robert! >>> >>> Ad. 1. There is no limit in mfsmount itself, but there are some limits in the operating system. Generally speaking it is wise not to open more than several thousands files in parallel. >>> >>> Ad. 2. Fopen invokes open, and open invokes (through kernel and FUSE) functions mfs_lookup and mfs_open. Mfs_lookup function changes consequtive path elements into i-node number. While mfs_open makes the target file opening. It sends a packet to the master in order to receive information about possibility to keep the file in the cache. It also marks the file in the master as opened - in cases it is deleted, it is sustained to the moment of closing. >>> >>> BTW. Why do you need this? >>> >>> >>> Kind regards >>> Michał Borychowski >>> MooseFS Support Manager >>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >>> Gemius S.A. >>> ul. Wołoska 7, 02-672 Warszawa >>> Budynek MARS, klatka D >>> Tel.: +4822 874-41-00 >>> Fax : +4822 874-41-01 >>> >>> >>> >>> -----Original Message----- >>> From: Robert Sandilands [mailto:rsa...@ne...] >>> Sent: Saturday, July 02, 2011 2:54 AM >>> To: moo...@li... >>> Subject: Re: [Moosefs-users] Write starvation >>> >>> Based on some tests I think the limit in this case is the number of >>> opens per minute. I think I need to understand what happens with an open >>> before I can make guesses on what can be done to get the number higher. >>> >>> But then it still does not quite explain the write starvation except if >>> the number of pending reads are just so much higher than the number of >>> pending writes that it seems to starve the writes. Maybe this will >>> resolve itself as I add more chunk servers. >>> >>> Some questions: >>> >>> 1. Is there a limit to the number of handles that client applications >>> can open per mount, per chunk server, per disk? >>> 2. What happens when an application does fopen() on a mount? Can >>> somebody give a quick overview or do I have to read some code? >>> >>> Robert >>> >>> On 6/30/11 11:32 AM, Ricardo J. Barberis wrote: >>>> El Miércoles 29 Junio 2011, Robert escribió: >>>>> Yes, we use Centos, but installing and using the ktune package generally >>>>> resolves most of the performance issues and differences I have seen with >>>>> Ubuntu/Debian. >>>> Nice to know about ktune and thank you for bringing it up, I'll take a look a >>>> it. >>>> >>>>> I don't understand the comment on hitting metadata a lot? What is a lot? >>>> A lot = reading / (re)writing / ls -l'ing / stat'ing too often. >>>> >>>> If the client can't cache the metadata but uses it often, that means it has to >>>> query the master every time. >>>> >>>> Network latencies might also play a role in the performance degradation. >>>> >>>>> Why would it make a difference? All the metadata is in RAM anyway? The >>>>> biggest limit to speed seems to be the number of IOPS that you can get out >>>>> of your disks you have available to you. Looking up the metadata from RAM >>>>> should be several orders of magnitude faster than that. >>>> Yep, and you have plenty of RAM, so that shouldn't be an issue in your case. >>>> >>>>> The activity reported through the CGI interface on the master is around >>>>> 2,400 opens per minute average. Reads and writes are also around 2400 per >>>>> minute alternating with each other. mknod has some peaks around 2,800 per >>>>> minute but is generally much lower. Lookup's are around 8,000 per minute >>>>> and getattr is around 700 per minute. Chunk replication and deletion is >>>>> around 50 per minute. The other numbers are generally very low. >>>> Mmm, maybe 2 chunkservers are just too litle to handle that activity but I >>>> would also check the network latencies. >>>> >>>> I'm also not really confident about having master and cunkserver on the same >>>> server but I don't have any hard evidence to support my feelings ;) >>>> >>>>> Is there a guide/hints specific to MooseFS on what IO/Net/Process >>>>> parameters would be good to investigate for mfsmaster? >>>> I'd like to know that too! >>>> >>>> Cheers, >>> ------------------------------------------------------------------------------ >>> All of the data generated in your IT infrastructure is seriously valuable. >>> Why? It contains a definitive record of application performance, security >>> threats, fraudulent activity, and more. Splunk takes this data and makes >>> sense of it. IT sense. And common sense. >>> http://p.sf.net/sfu/splunk-d2d-c2 >>> _______________________________________________ >>> moosefs-users mailing list >>> moo...@li... >>> https://lists.sourceforge.net/lists/listinfo/moosefs-users >>> >> ------------------------------------------------------------------------------ >> All of the data generated in your IT infrastructure is seriously valuable. >> Why? It contains a definitive record of application performance, security >> threats, fraudulent activity, and more. Splunk takes this data and makes >> sense of it. IT sense. And common sense. >> http://p.sf.net/sfu/splunk-d2d-c2 >> _______________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> > > ------------------------------------------------------------------------------ > All of the data generated in your IT infrastructure is seriously valuable. > Why? It contains a definitive record of application performance, security > threats, fraudulent activity, and more. Splunk takes this data and makes > sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-d2d-c2 > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: WK <wk...@bn...> - 2011-06-30 00:32:54
|
On 6/29/2011 11:29 AM, Robert wrote: > Yes, the master server is working hard. But I would still expect a > somewhat fair distribution of load between read and write. > > The specs: > > 2 x quad core Xeon E5405 @ 2GHz. > 64 GB of RAM > 32 x 2 TB 7200 RPM SATA disks > 68 million file system objects > 65.4 million files > No swap is being used > mfsmaster is using 23 GB of RAM. > OK, I am sorry if this seem silly and I'm completely missing something. But I am confused as to the above specs. Is your Master and Chunkserver on the same machine? or are those 32 x 2TB drives spread out on how many chunkservers? Our experience has been the more chunkservers we have, the better the performance because the read/writes are spread out among the chunkservers (in addition to individual spindles on each chunkserver). WK |
|
From: Robert S. <rsa...@ne...> - 2011-06-30 01:37:55
|
The one chunk server is also the master. As we free up space on other servers we will add more chunk servers. We currently only have two. The other chunk server has 36 x 2 TB drives and 64 GB of RAM. I agree with your comments on multiple chunk servers. That is one of the major reasons why we chose MooseFS. Robert Incomprehensible email compliments of my iPad On Jun 29, 2011, at 8:32 PM, WK <wk...@bn...> wrote: > On 6/29/2011 11:29 AM, Robert wrote: >> >> Yes, the master server is working hard. But I would still expect a somewhat fair distribution of load between read and write. >> >> The specs: >> >> 2 x quad core Xeon E5405 @ 2GHz. >> 64 GB of RAM >> 32 x 2 TB 7200 RPM SATA disks >> 68 million file system objects >> 65.4 million files >> No swap is being used >> mfsmaster is using 23 GB of RAM. >> > > OK, I am sorry if this seem silly and I'm completely missing something. > > But I am confused as to the above specs. > > Is your Master and Chunkserver on the same machine? > > or are those 32 x 2TB drives spread out on how many chunkservers? > > Our experience has been the more chunkservers we have, the better the performance because the read/writes are spread out among the chunkservers (in addition to individual spindles on each chunkserver). > > WK > > ------------------------------------------------------------------------------ > All of the data generated in your IT infrastructure is seriously valuable. > Why? It contains a definitive record of application performance, security > threats, fraudulent activity, and more. Splunk takes this data and makes > sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-d2d-c2 > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Michal B. <mic...@ge...> - 2011-07-04 07:31:41
|
Hi Robert! Parallel reading and writing to the same file indeed causes some write delays. If it is not your case, adding 1-2 chunkservers and having a dedicated machine for the master should help. Kind regards Michał Borychowski MooseFS Support Manager _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Gemius S.A. ul. Wołoska 7, 02-672 Warszawa Budynek MARS, klatka D Tel.: +4822 874-41-00 Fax : +4822 874-41-01 -----Original Message----- From: Robert Sandilands [mailto:rsa...@ne...] Sent: Wednesday, June 29, 2011 4:30 AM To: moo...@li... Subject: [Moosefs-users] Write starvation I have been moving data from existing non-distributed file systems onto a MooseFS file system. I am using 1.6.20 on Centos 5.6. While moving the data I have also transfered some of the normal read-only traffic load to use the data already moved onto the MFS volume. What I can see is when there is any significant read traffic then write traffic slows down to a crawl. When I look at the server charts for any of the chunk servers generated by mfscgiserv then it seems like read and write traffic seems to alternate. Write traffic does not stop completely, but seems to slow down to < 10 kB per second under high read traffic conditions. When the read traffic decreases the write traffic will increase to normal levels. Is this a known problem? Is there something I can do to ensure that write traffic is not starved by read traffic? Robert ---------------------------------------------------------------------------- -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Robert S. <rsa...@ne...> - 2011-07-04 17:13:21
|
Hi Michal, In our case we never modify the files once we have received them. It is also very rare for multiple processes to read the same file at the same time. What is not rare is hundreds of simultaneous reads on hundreds of unique files randomly distributed through the file system. We plan on adding more chunkservers as we move content from traditional file systems to MFS. A dedicated master may be some time away. Getting a machine with 64+ GB of RAM is not always easy to get past the budget monsters. Robert On 7/4/11 3:30 AM, Michal Borychowski wrote: > Hi Robert! > > Parallel reading and writing to the same file indeed causes some write > delays. If it is not your case, adding 1-2 chunkservers and having a > dedicated machine for the master should help. > > > Kind regards > Michał Borychowski > MooseFS Support Manager > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > Gemius S.A. > ul. Wołoska 7, 02-672 Warszawa > Budynek MARS, klatka D > Tel.: +4822 874-41-00 > Fax : +4822 874-41-01 > > > > -----Original Message----- > From: Robert Sandilands [mailto:rsa...@ne...] > Sent: Wednesday, June 29, 2011 4:30 AM > To: moo...@li... > Subject: [Moosefs-users] Write starvation > > I have been moving data from existing non-distributed file systems onto > a MooseFS file system. > > I am using 1.6.20 on Centos 5.6. > > While moving the data I have also transfered some of the normal > read-only traffic load to use the data already moved onto the MFS volume. > > What I can see is when there is any significant read traffic then write > traffic slows down to a crawl. > > When I look at the server charts for any of the chunk servers generated > by mfscgiserv then it seems like read and write traffic seems to alternate. > > Write traffic does not stop completely, but seems to slow down to< 10 > kB per second under high read traffic conditions. When the read traffic > decreases the write traffic will increase to normal levels. > > Is this a known problem? Is there something I can do to ensure that > write traffic is not starved by read traffic? > > Robert > > > > > > ---------------------------------------------------------------------------- > -- > All of the data generated in your IT infrastructure is seriously valuable. > Why? It contains a definitive record of application performance, security > threats, fraudulent activity, and more. Splunk takes this data and makes > sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-d2d-c2 > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > |
|
From: Ricardo J. B. <ric...@da...> - 2011-07-04 21:59:05
|
El Lunes 04 Julio 2011, Robert Sandilands escribió: > We plan on adding more chunkservers as we move content from traditional > file systems to MFS. A dedicated master may be some time away. Getting a > machine with 64+ GB of RAM is not always easy to get past the budget > monsters. If the problem is RAM, you can take some from your chunkservers, as they don't use it much (only for disk cache I would guess). Or, once you add another chunkserver you might free the master disks and keep it as master only. I don't have so many files (600.000) but my chunkservers have 2 or 4 GB and are only using ~200 MB and my master is using 580 MB from a total of 4 GB. The rest of the memory is all used for disk cache and buffers, at least acording to "free -m". Regards, -- Ricardo J. Barberis Senior SysAdmin / ITI Dattatec.com :: Soluciones de Web Hosting Tu Hosting hecho Simple! |
|
From: Robert S. <rsa...@ne...> - 2011-07-05 12:47:21
|
Hi Ricardo, My guess is that the number of chunks that the chunkserver knows about has an influence on the memory usage. In my case mfschunkserver uses a bit more memory. As reported by top: USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND daemon 0 -19 3622m 3.4g 756 S 16.9 5.5 188:10.92 mfschunkserver daemon 0 -19 8763m 8.5g 676 D 22.6 13.5 7766:43 mfschunkserver daemon 0 -19 24.3g 24g 768 S 10.0 39.2 248:44.71 mfsmaster I have seen mfsmount use around 1 GB of RAM too. This is the current usage on two servers where only read traffic happens. root 4 -19 745m 51m 496 S 3.3 0.1 453:04.87 mfsmount root 3 -19 684m 52m 748 S 2.7 0.1 55:44.67 mfsmount This is mfsmount on a server where only write traffic happens: root 0 -19 298m 39m 528 S 4.0 2.0 2017:43 mfsmount On another machine with a mix of read and write: root 6 -19 364m 12m 492 S 0.0 0.2 57:15.53 mfsmount But, yes. We will have to see how we can juggle hardware to get the best performance within the other constraints we have. Robert On 7/4/11 5:58 PM, Ricardo J. Barberis wrote: > El Lunes 04 Julio 2011, Robert Sandilands escribió: >> We plan on adding more chunkservers as we move content from traditional >> file systems to MFS. A dedicated master may be some time away. Getting a >> machine with 64+ GB of RAM is not always easy to get past the budget >> monsters. > If the problem is RAM, you can take some from your chunkservers, as they don't > use it much (only for disk cache I would guess). > > Or, once you add another chunkserver you might free the master disks and keep > it as master only. > > I don't have so many files (600.000) but my chunkservers have 2 or 4 GB and > are only using ~200 MB and my master is using 580 MB from a total of 4 GB. > > The rest of the memory is all used for disk cache and buffers, at least > acording to "free -m". > > Regards, |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X