Menu
▾
▴
moosefs-users
|
From: Eduard I. <edu...@di...> - 2010-01-21 10:29:55
|
Hi. I have a webserver that has the files (~1.1mil files and 280gb) mounted over mfs 1.6.11. Although the server generates about 2-3mbit of http trafic, it retrieves from chunk servers between 35 and 70mbit (5sec average). The mount has: mfscachefiles,mfsattrcacheto=10,mfsentrycacheto=10,mfsdirentrycacheto=20 Is there anything I can do to lower the amount of traffic from chunks? Thanks, -- Eduard |
|
From: Steve <st...@bo...> - 2010-01-21 14:27:54
|
Hi, Is there any extra docs on the cache pro's and con's, guidelines for usage ? more explicit than the man page. -------Original Message------- From: Eduard Iacoboaia Date: 21/01/2010 10:30:01 To: moo...@li... Subject: [Moosefs-users] mfs mount generates high traffic from chunks Hi. I have a webserver that has the files (~1.1mil files and 280gb) mounted over mfs 1.6.11. Although the server generates about 2-3mbit of http traffic, it retrieves from chunk servers between 35 and 70mbit (5sec average). The mount has: mfscachefiles,mfsattrcacheto=10,mfsentrycacheto=10,mfsdirentrycacheto=20 Is there anything I can do to lower the amount of traffic from chunks? Thanks, -- Eduard ----------------------------------------------------------------------------- Throughout its 18-year history, RSA Conference consistently attracts the world's best and brightest in the field, creating opportunities for Conference attendees to learn about information security's most important issues through interactions with peers, luminaries and emerging and established companies. http://p.sf.net/sfu/rsaconf-dev2dev _______________________________________________ moosefs-users mailing list moo...@li... https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Michał B. <mic...@ge...> - 2010-01-28 10:04:04
|
> -O mfscachefiles
> preserve file data in cache (default is not to use data cache)
Now this option should be used:
-o mfscachemode=X
where X can be:
YES - all files are stored in a local cache
NO - no files are stored in a local cache
AUTO - storing in a local cache depends on 'allowdatacache' flag which can
be set individually for every file
MooseFS is a network file system and generally speaking files should not be
kept in cache because from the moment of putting a file in cache another
computer may change it. So cache should be used only there where you are
sure about this.
> -O mfsattrcacheto=SEC
> set attributes cache timeout in seconds (default: 1.0)
This is time of keeping file attributes in cache which are used by kernel by
simple operations like "rm", "mv", etc. The system tests many times
attributes of the same i-nodes and that's why we should allow for using
cache here. The time should be relevantly short but not shorter than time
needed for an individual operation like "rm" or "mv". We suggest leaving the
default value.
> -O mfsentrycacheto=SEC
> set file entry cache timeout in seconds (default: 0.0, I.e. no
> cache)
> -O mfsdirentrycacheto=SEC
> set directory entry cache timeout in seconds (default: 1.0)
Both options affect keeping entry in a folder ("lookup" operation - mapping
file or folder name to its i-node). This situation is similar to the above.
Every operation in which we refer to the full path causes that the system
every time scans file system in order to locate the given object. Most of
these operations concerns folders and folder's attributes do not change
often. So we decided to store in cache subfolders entries. That's why there
are two options: "mfsdirentrycacheto" for subfolders and "mfsentrycacheto"
for other objects. We also suggest leaving the default values here.
> -O mfswritecachesize=N
> specify write cache size in MiB (in range: 16..2048 - default:
> 250)
In order to increase performance writing in MooseFS is doen by "mfsmount" in
background. That's why some part of RAM has to be used as a write buffer.
This option allows to change this buffer. Lower value means lower memory
usage but also potentially slower writing speed. Greater value means higher
memory usage and in some cases can increase write operations. If there is no
need to keep this value smaller, we recommend not to change it. In pratice
increasing this buffer value does not influence a lot.
So the most important is possibility to control cache of file content (which
is important for the real system performance) with these options:
mfsseteattr -f allowdatacache FILE_1 [ FILE_2 ... ]
mfsdeleattr -f allowdatacache FILE_1 [ FILE_2 ... ]
Kind regards
Michał Borychowski
MooseFS Support Manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Gemius S.A.
ul. Wołoska 7, 02-672 Warszawa
Budynek MARS, klatka D
Tel.: +4822 874-41-00
Fax : +4822 874-41-01
> -----Original Message-----
> From: Steve [mailto:st...@bo...]
> Sent: Monday, January 25, 2010 3:17 PM
> To: Michał Borychowski
> Subject: RE: [Moosefs-users] mfs mount generates high trafic from
> chunks
>
> I understand the outline of what a cache does but under what situation
> would
> it be useful ? What is the downside ?
>
> Any guidance on choosing parameters or is a case of trying values and
> measuring results ?
>
>
>
>
>
>
>
> -------Original Message-------
>
>
>
> From: Michał Borychowski
>
> Date: 25/01/2010 10:48:36
>
> To: 'Steve'
>
> Subject: RE: [Moosefs-users] mfs mount generates high traffic from
> chunks
>
>
>
> Steve, let me know more precisely what did you mean by your question
> below??
>
>
>
>
>
>
> Regards
>
> Michał
>
>
>
> -----Original Message-----
>
> From: Steve [mailto:st...@bo...]
>
> Sent: Thursday, January 21, 2010 3:27 PM
>
> To: moo...@li...
>
> Subject: Re: [Moosefs-users] mfs mount generates high traffic from
> chunks
>
>
>
> Hi,
>
>
>
>
>
>
>
> Is there any extra docs on the cache pro's and con's, guidelines for
> usage ?
>
>
> more explicit than the man page.
|
|
From: Eduard I. <edu...@di...> - 2010-01-22 11:31:48
Attachments:
mfscgi-1.jpg
|
Hi. This is the traffic incoming on port 9422 in the mount server: -------------------------------------------------------- | Rate | Average for | -------------------------------------------------------- |########| 3 sec | 15 sec | 30 sec | -------------------------------------------------------- | bits | 154.2M | 142.4M | 145.7M | | pkts | 13.1K | 12.1K | 12.3K | -------------------------------------------------------- This is the outgoing port 80 from the same server: -------------------------------------------------------- | Rate | Average for | -------------------------------------------------------- |########| 3 sec | 15 sec | 30 sec | -------------------------------------------------------- | bits | 1.6M | 1.2M | 1.2M | | pkts | 171 | 129 | 127 | -------------------------------------------------------- I also attached a screenshot. If it's not what you needed, please let me know. Thanks, -- Eduard Michał Borychowski wrote: > And what do you have on graphs from chunkservers? Could you send us the > screenshots from the cgi monitor? > > Is there really 35-70Mb transfer between mfsmount and chunkservers? > > > Kind regards > Michal > > -----Original Message----- > From: Eduard Iacoboaia [mailto:edu...@di...] > Sent: Thursday, January 21, 2010 11:29 AM > To: moo...@li... > Subject: [Moosefs-users] mfs mount generates high trafic from chunks > > Hi. > > I have a webserver that has the files (~1.1mil files and 280gb) mounted > over mfs 1.6.11. Although the server generates about 2-3mbit of http > trafic, it retrieves from chunk servers between 35 and 70mbit (5sec > average). The mount has: > > mfscachefiles,mfsattrcacheto=10,mfsentrycacheto=10,mfsdirentrycacheto=20 > > Is there anything I can do to lower the amount of traffic from chunks? > > Thanks, > -- > Eduard > > > ---------------------------------------------------------------------------- > -- > Throughout its 18-year history, RSA Conference consistently attracts the > world's best and brightest in the field, creating opportunities for > Conference > attendees to learn about information security's most important issues > through > interactions with peers, luminaries and emerging and established companies. > http://p.sf.net/sfu/rsaconf-dev2dev > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > > |
|
From: Michał B. <mic...@ge...> - 2010-01-22 14:00:23
|
Don't you have any additional replications between chunkservers? Does the webserver and one of chunkservers use the same machine? Kind regards Michał Borychowski MooseFS Support Manager -----Original Message----- From: Eduard Iacoboaia [mailto:edu...@di...] Sent: Friday, January 22, 2010 12:31 PM To: moo...@li... Subject: Re: [Moosefs-users] mfs mount generates high trafic from chunks Hi. This is the traffic incoming on port 9422 in the mount server: -------------------------------------------------------- | Rate | Average for | -------------------------------------------------------- |########| 3 sec | 15 sec | 30 sec | -------------------------------------------------------- | bits | 154.2M | 142.4M | 145.7M | | pkts | 13.1K | 12.1K | 12.3K | -------------------------------------------------------- This is the outgoing port 80 from the same server: -------------------------------------------------------- | Rate | Average for | -------------------------------------------------------- |########| 3 sec | 15 sec | 30 sec | -------------------------------------------------------- | bits | 1.6M | 1.2M | 1.2M | | pkts | 171 | 129 | 127 | -------------------------------------------------------- I also attached a screenshot. If it's not what you needed, please let me know. Thanks, -- Eduard Michał Borychowski wrote: > And what do you have on graphs from chunkservers? Could you send us > the screenshots from the cgi monitor? > > Is there really 35-70Mb transfer between mfsmount and chunkservers? > > > Kind regards > Michal > > -----Original Message----- > From: Eduard Iacoboaia [mailto:edu...@di...] > Sent: Thursday, January 21, 2010 11:29 AM > To: moo...@li... > Subject: [Moosefs-users] mfs mount generates high trafic from chunks > > Hi. > > I have a webserver that has the files (~1.1mil files and 280gb) > mounted over mfs 1.6.11. Although the server generates about 2-3mbit > of http trafic, it retrieves from chunk servers between 35 and 70mbit > (5sec average). The mount has: > > mfscachefiles,mfsattrcacheto=10,mfsentrycacheto=10,mfsdirentrycacheto= > 20 > > Is there anything I can do to lower the amount of traffic from chunks? > > Thanks, > -- > Eduard > > > ---------------------------------------------------------------------- > ------ > -- > Throughout its 18-year history, RSA Conference consistently attracts > the world's best and brightest in the field, creating opportunities > for Conference attendees to learn about information security's most > important issues through interactions with peers, luminaries and > emerging and established companies. > http://p.sf.net/sfu/rsaconf-dev2dev > _______________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > > |
|
From: Eduard I. <edu...@di...> - 2010-01-22 14:26:28
|
There are 2 chunks, goal 2 on all files, one active mount (the webserver): 3 virtual machines, each hosted on separate servers, nothing else on those servers. The master, metaloger and mfscgi are 3 virtual machines hosted on a 4th server. I attached some screenshots. I noticed the getattr chart resembles a lot those from traffic in chunk servers. -- Eduard Michał Borychowski wrote: > Don't you have any additional replications between chunkservers? Does the > webserver and one of chunkservers use the same machine? > > > Kind regards > Michał Borychowski > MooseFS Support Manager > > > > -----Original Message----- > From: Eduard Iacoboaia [mailto:edu...@di...] > Sent: Friday, January 22, 2010 12:31 PM > To: moo...@li... > Subject: Re: [Moosefs-users] mfs mount generates high trafic from chunks > > Hi. > > This is the traffic incoming on port 9422 in the mount server: > -------------------------------------------------------- > | Rate | Average for | > -------------------------------------------------------- > |########| 3 sec | 15 sec | 30 sec | > -------------------------------------------------------- > | bits | 154.2M | 142.4M | 145.7M | > | pkts | 13.1K | 12.1K | 12.3K | > -------------------------------------------------------- > > > This is the outgoing port 80 from the same server: > -------------------------------------------------------- > | Rate | Average for | > -------------------------------------------------------- > |########| 3 sec | 15 sec | 30 sec | > -------------------------------------------------------- > | bits | 1.6M | 1.2M | 1.2M | > | pkts | 171 | 129 | 127 | > -------------------------------------------------------- > > > I also attached a screenshot. If it's not what you needed, please let me > know. > > Thanks, > -- > Eduard > > Michał Borychowski wrote: > >> And what do you have on graphs from chunkservers? Could you send us >> the screenshots from the cgi monitor? >> >> Is there really 35-70Mb transfer between mfsmount and chunkservers? >> >> >> Kind regards >> Michal >> >> -----Original Message----- >> From: Eduard Iacoboaia [mailto:edu...@di...] >> Sent: Thursday, January 21, 2010 11:29 AM >> To: moo...@li... >> Subject: [Moosefs-users] mfs mount generates high trafic from chunks >> >> Hi. >> >> I have a webserver that has the files (~1.1mil files and 280gb) >> mounted over mfs 1.6.11. Although the server generates about 2-3mbit >> of http trafic, it retrieves from chunk servers between 35 and 70mbit >> (5sec average). The mount has: >> >> mfscachefiles,mfsattrcacheto=10,mfsentrycacheto=10,mfsdirentrycacheto= >> 20 >> >> Is there anything I can do to lower the amount of traffic from chunks? >> >> Thanks, >> -- >> Eduard >> >> >> ---------------------------------------------------------------------- >> ------ >> -- >> Throughout its 18-year history, RSA Conference consistently attracts >> the world's best and brightest in the field, creating opportunities >> for Conference attendees to learn about information security's most >> important issues through interactions with peers, luminaries and >> emerging and established companies. >> http://p.sf.net/sfu/rsaconf-dev2dev >> _______________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> >> >> > > |
|
From: Michał B. <mic...@ge...> - 2010-01-25 11:34:59
|
Hi Eduard! So... From the number of 'getattr', 'open' i 'read' operations we conclude that there are about 4500 files read in one minute. Given three chunkservers it takes 1500 reads per minute per one server, which is 25 reads per second on one server. The system shows there are 24MB read per second which would mean an average file size of about 950kB and transfer about 190Mb/s. On the other hand there are about 1 160 000 files which with an average file size of 950kB would occupy about 1 TB. From the Info tab we see the system has 14 TB of space with 13 TB free - everything sums up properly. So the question would rather be - why the server requests so many reads though it does not send that amount of data? Please check your webserver and its configuration. What webserver do you use? Thereotically none webserver should make unnecessary reads. You can try connecting to the webserver by a tool like "strace" and see what operations are made. Kind regards Michał Borychowski MooseFS Support Manager _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Gemius S.A. ul. Wołoska 7, 02-672 Warszawa Budynek MARS, klatka D Tel.: +4822 874-41-00 Fax : +4822 874-41-01 -----Original Message----- From: Eduard Iacoboaia [mailto:edu...@di...] Sent: Friday, January 22, 2010 3:26 PM To: moo...@li... Subject: Re: [Moosefs-users] mfs mount generates high trafic from chunks There are 2 chunks, goal 2 on all files, one active mount (the webserver): 3 virtual machines, each hosted on separate servers, nothing else on those servers. The master, metaloger and mfscgi are 3 virtual machines hosted on a 4th server. I attached some screenshots. I noticed the getattr chart resembles a lot those from traffic in chunk servers. -- Eduard Michał Borychowski wrote: > Don't you have any additional replications between chunkservers? Does > the webserver and one of chunkservers use the same machine? > > > Kind regards > Michał Borychowski > MooseFS Support Manager > > > > -----Original Message----- > From: Eduard Iacoboaia [mailto:edu...@di...] > Sent: Friday, January 22, 2010 12:31 PM > To: moo...@li... > Subject: Re: [Moosefs-users] mfs mount generates high trafic from > chunks > > Hi. > > This is the traffic incoming on port 9422 in the mount server: > -------------------------------------------------------- > | Rate | Average for | > -------------------------------------------------------- > |########| 3 sec | 15 sec | 30 sec | > -------------------------------------------------------- > | bits | 154.2M | 142.4M | 145.7M | > | pkts | 13.1K | 12.1K | 12.3K | > -------------------------------------------------------- > > > This is the outgoing port 80 from the same server: > -------------------------------------------------------- > | Rate | Average for | > -------------------------------------------------------- > |########| 3 sec | 15 sec | 30 sec | > -------------------------------------------------------- > | bits | 1.6M | 1.2M | 1.2M | > | pkts | 171 | 129 | 127 | > -------------------------------------------------------- > > > I also attached a screenshot. If it's not what you needed, please let > me know. > > Thanks, > -- > Eduard > > Michał Borychowski wrote: > >> And what do you have on graphs from chunkservers? Could you send us >> the screenshots from the cgi monitor? >> >> Is there really 35-70Mb transfer between mfsmount and chunkservers? >> >> >> Kind regards >> Michal >> >> -----Original Message----- >> From: Eduard Iacoboaia [mailto:edu...@di...] >> Sent: Thursday, January 21, 2010 11:29 AM >> To: moo...@li... >> Subject: [Moosefs-users] mfs mount generates high trafic from chunks >> >> Hi. >> >> I have a webserver that has the files (~1.1mil files and 280gb) >> mounted over mfs 1.6.11. Although the server generates about 2-3mbit >> of http trafic, it retrieves from chunk servers between 35 and 70mbit >> (5sec average). The mount has: >> >> mfscachefiles,mfsattrcacheto=10,mfsentrycacheto=10,mfsdirentrycacheto >> = >> 20 >> >> Is there anything I can do to lower the amount of traffic from chunks? >> >> Thanks, >> -- >> Eduard >> >> >> --------------------------------------------------------------------- >> - >> ------ >> -- >> Throughout its 18-year history, RSA Conference consistently attracts >> the world's best and brightest in the field, creating opportunities >> for Conference attendees to learn about information security's most >> important issues through interactions with peers, luminaries and >> emerging and established companies. >> http://p.sf.net/sfu/rsaconf-dev2dev >> _______________________________________________ >> moosefs-users mailing list >> moo...@li... >> https://lists.sourceforge.net/lists/listinfo/moosefs-users >> >> >> > > |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X