Menu
▾
▴
moosefs-users
|
From: Wilson, S. M <st...@pu...> - 2018-05-07 14:36:55
|
Hi,
I'm considering implementing a dedicated network for our chunk servers to use soley for replication among themselves. By doing this, I hope to separate the chunk traffic from the clients from the replication traffic that takes place among the chunk servers. If my understanding is correct, this is not achieved by using the REMAP_* options in mfsmaster.cfg which only separates out the traffic to/from the master.
If anyone else has done this, I'd be grateful to hear about your experience, especially in these two areas:
1) what level of performance improvement was seen
2) what needed to be done in the MooseFS configuration and OS networking to implement it
?Thanks!
Steve
|
|
From: Casper L. <cas...@pr...> - 2018-05-12 16:15:29
|
Hi Steve, As an alternative I'd like to suggest Linux Network Bonding to you. You should at least read up on it if you don't know it already. https://wiki.linuxfoundation.org/networking/bonding Bonding allows you to use the extra network bandwidth effectively, for *both* replication and client traffic. This also possibly eliminates the network as single point of failure in your setup. Choose your bonding mode carefully. Greetings, Casper Op ma 7 mei 2018 om 16:37 schreef Wilson, Steven M <st...@pu...>: > Hi, > > > I'm considering implementing a dedicated network for our chunk servers to > use soley for replication among themselves. By doing this, I hope to > separate the chunk traffic from the clients from the replication traffic > that takes place among the chunk servers. If my understanding is correct, > this is not achieved by using the REMAP_* options in mfsmaster.cfg which > only separates out the traffic to/from the master. > > > If anyone else has done this, I'd be grateful to hear about your > experience, especially in these two areas: > > 1) what level of performance improvement was seen > > 2) what needed to be done in the MooseFS configuration and OS > networking to implement it > > > Thanks! > > Steve > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users > |
|
From: R.C. <mil...@gm...> - 2018-05-12 16:43:23
|
Hi Steve, if your concern is about reducing network traffic on your switches and you plan to install MooseFS (master and chuncks) on dedicated HW in a dedicated rack or so, just place a switch between MooseFS units and the rest of your network. The switch will keep the MooseFS internal traffic right behind the switch. You can then connect this switch to the rest of your network with SFP port or a dedicated uplink connection (if your current switches have one of these) instead of a standard cat6 cable, which would of course become in this case a bottleneck. Once MooseFS is "behind" the switch, you can furtherly improve its bandwidth by implementing Bonding (as suggested by Casper) or Teaming. See here for a complete comparison: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-comparison_of_network_teaming_to_bonding Hope it helps Bye Raf Il 12/05/2018 17:51, Casper Langemeijer ha scritto: > Hi Steve, > > As an alternative I'd like to suggest Linux Network Bonding to you. > You should at least read up on it if you don't know it already. > https://wiki.linuxfoundation.org/networking/bonding Bonding allows you > to use the extra network bandwidth effectively, for *both* replication > and client traffic. This also possibly eliminates the network as > single point of failure in your setup. Choose your bonding mode > carefully. > > Greetings, Casper > > Op ma 7 mei 2018 om 16:37 schreef Wilson, Steven M <st...@pu... > <mailto:st...@pu...>>: > > Hi, > > > I'm considering implementing a dedicated network for our chunk > servers to use soley for replication among themselves. By doing > this, I hope to separate the chunk traffic from the clients from > the replication traffic that takes place among the chunk servers. > If my understanding is correct, this is not achieved by using the > REMAP_* options in mfsmaster.cfg which only separates out the > traffic to/from the master. > > > If anyone else has done this, I'd be grateful to hear about your > experience, especially in these two areas: > > 1) what level of performance improvement was seen > > 2) what needed to be done in the MooseFS configuration and OS > networking to implement it > > > Thanks! > > Steve > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! > http://sdm.link/slashdot_________________________________________ > moosefs-users mailing list > moo...@li... > <mailto:moo...@li...> > https://lists.sourceforge.net/lists/listinfo/moosefs-users > > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > _________________________________________ > moosefs-users mailing list > moo...@li... > https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Wilson, S. M <st...@pu...> - 2018-05-14 17:19:15
|
Thanks, Casper and Raf, for your suggestions! I am using MooseFS in a workgroup setting in an office environment and not in a server room so my options are somewhat limited. Central IT services provide several 1Gbps network ports per office. These are then connected to directly switches found in equpment rooms on each floor of the building. Most of my chunk servers have dual NICs so I thought that I could add a "replication network" between chunk servers (at least those that are located in the same room) using a switch that I install in the room. One NIC in each server would be connected to the network provided by central IT and the other NIC would be connected to my local network. I had given some thought to using bonding/teaming but I don't see a good way to take advantage of this concept in our environment. Thanks again! Steve ________________________________ From: R.C. <mil...@gm...> Sent: Saturday, May 12, 2018 12:43 PM To: moo...@li... Subject: Re: [MooseFS-Users] Separate network for chunk servers Hi Steve, if your concern is about reducing network traffic on your switches and you plan to install MooseFS (master and chuncks) on dedicated HW in a dedicated rack or so, just place a switch between MooseFS units and the rest of your network. The switch will keep the MooseFS internal traffic right behind the switch. You can then connect this switch to the rest of your network with SFP port or a dedicated uplink connection (if your current switches have one of these) instead of a standard cat6 cable, which would of course become in this case a bottleneck. Once MooseFS is "behind" the switch, you can furtherly improve its bandwidth by implementing Bonding (as suggested by Casper) or Teaming. See here for a complete comparison: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-comparison_of_network_teaming_to_bonding Hope it helps Bye Raf Il 12/05/2018 17:51, Casper Langemeijer ha scritto: Hi Steve, As an alternative I'd like to suggest Linux Network Bonding to you. You should at least read up on it if you don't know it already. https://wiki.linuxfoundation.org/networking/bonding Bonding allows you to use the extra network bandwidth effectively, for *both* replication and client traffic. This also possibly eliminates the network as single point of failure in your setup. Choose your bonding mode carefully. Greetings, Casper Op ma 7 mei 2018 om 16:37 schreef Wilson, Steven M <st...@pu...<mailto:st...@pu...>>: Hi, I'm considering implementing a dedicated network for our chunk servers to use soley for replication among themselves. By doing this, I hope to separate the chunk traffic from the clients from the replication traffic that takes place among the chunk servers. If my understanding is correct, this is not achieved by using the REMAP_* options in mfsmaster.cfg which only separates out the traffic to/from the master. If anyone else has done this, I'd be grateful to hear about your experience, especially in these two areas: 1) what level of performance improvement was seen 2) what needed to be done in the MooseFS configuration and OS networking to implement it Thanks! Steve ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot_________________________________________ moosefs-users mailing list moo...@li...<mailto:moo...@li...> https://lists.sourceforge.net/lists/listinfo/moosefs-users ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _________________________________________ moosefs-users mailing list moo...@li...<mailto:moo...@li...> https://lists.sourceforge.net/lists/listinfo/moosefs-users |
|
From: Wilson, S. M <st...@pu...> - 2018-06-06 18:26:48
|
________________________________
From: Wilson, Steven M <st...@pu...>
Sent: Monday, May 7, 2018 10:36 AM
To: MooseFS-Users
Subject: [MooseFS-Users] Separate network for chunk servers
Hi,
I'm considering implementing a dedicated network for our chunk servers to use soley for replication among themselves. By doing this, I hope to separate the chunk traffic from the clients from the replication traffic that takes place among the chunk servers. If my understanding is correct, this is not achieved by using the REMAP_* options in mfsmaster.cfg which only separates out the traffic to/from the master.
If anyone else has done this, I'd be grateful to hear about your experience, especially in these two areas:
1) what level of performance improvement was seen
2) what needed to be done in the MooseFS configuration and OS networking to implement it
?Thanks!
Steve
=====================================================================================
In case it's helpful to anyone else, I did find an easy way to shift traffic between chunk servers onto a separate network. On each chunk server I created a few iptables rules using NAT like the following (these would need to be repeated for each additional chunk server):
?iptables -t nat -A OUTPUT -s $MY_IP/32 -d $OTHER_CS_IP/32 -j DNAT --to-destination $OTHER_CS_PRIV_IP -p tcp -m tcp --dport $CS_PORT
iptables -t nat -A POSTROUTING -s $MY_IP/32 -d $OTHER_CS_PRIV_IP/32 -j SNAT --to-source $MY_PRIV_IP
I did some performance testing and found that, for fairly obvious reasons, this private chunk server network only helps when the network is near saturation and most of the I/O is writing to the file system (which generates inter-chunkserver traffic for producing redundant copies of the data). When this type of environment is simulated, I saw between 10 to 20% improvement in speed for writing files to the file system.
Steve
|
|
From: Piotr R. K. <pio...@mo...> - 2018-06-06 21:21:28
|
Thanks for sharing! Peter -- Piotr Robert Konopelko | mobile: +48 601 476 440 MooseFS Client Support Team | moosefs.com <http://moosefs.com/> GitHub <https://github.com/moosefs/moosefs> | Twitter <https://twitter.com/moosefs> | Facebook <https://www.facebook.com/moosefs> | LinkedIn <https://www.linkedin.com/company/moosefs> > On 6 Jun 2018, at 8:26 PM, Wilson, Steven M <st...@pu... <mailto:st...@pu...>> wrote: > > > From: Wilson, Steven M <st...@pu... <mailto:st...@pu...>> > Sent: Monday, May 7, 2018 10:36 AM > To: MooseFS-Users > Subject: [MooseFS-Users] Separate network for chunk servers > > Hi, > > I'm considering implementing a dedicated network for our chunk servers to use soley for replication among themselves. By doing this, I hope to separate the chunk traffic from the clients from the replication traffic that takes place among the chunk servers. If my understanding is correct, this is not achieved by using the REMAP_* options in mfsmaster.cfg which only separates out the traffic to/from the master. > > If anyone else has done this, I'd be grateful to hear about your experience, especially in these two areas: > 1) what level of performance improvement was seen > 2) what needed to be done in the MooseFS configuration and OS networking to implement it > > Thanks! > Steve > > ===================================================================================== > > In case it's helpful to anyone else, I did find an easy way to shift traffic between chunk servers onto a separate network. On each chunk server I created a few iptables rules using NAT like the following (these would need to be repeated for each additional chunk server): > iptables -t nat -A OUTPUT -s $MY_IP/32 -d $OTHER_CS_IP/32 -j DNAT --to-destination $OTHER_CS_PRIV_IP -p tcp -m tcp --dport $CS_PORT > iptables -t nat -A POSTROUTING -s $MY_IP/32 -d $OTHER_CS_PRIV_IP/32 -j SNAT --to-source $MY_PRIV_IP > > I did some performance testing and found that, for fairly obvious reasons, this private chunk server network only helps when the network is near saturation and most of the I/O is writing to the file system (which generates inter-chunkserver traffic for producing redundant copies of the data). When this type of environment is simulated, I saw between 10 to 20% improvement in speed for writing files to the file system. > > Steve > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org <http://slashdot.org/>! http://sdm.link/slashdot_________________________________________ <http://sdm.link/slashdot_________________________________________> > moosefs-users mailing list > moo...@li... <mailto:moo...@li...> > https://lists.sourceforge.net/lists/listinfo/moosefs-users <https://lists.sourceforge.net/lists/listinfo/moosefs-users> Best regards, Peter -- Piotr Robert Konopelko | mobile: +48 601 476 440 MooseFS Client Support Team | moosefs.com <http://moosefs.com/> GitHub <https://github.com/moosefs/moosefs> | Twitter <https://twitter.com/moosefs> | Facebook <https://www.facebook.com/moosefs> | LinkedIn <https://www.linkedin.com/company/moosefs> |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X