Hi guys, first thanks a bunch for the great work. I just started testing this WAN Emulator and I can tell you that so far I like it better then most open source efforts (dummynet or nistnet based).
There are two requirements needed for accurate WAN simulation as it pertains to link bandwidth limitation.
Packet loss due to congestion (obsolete packets or packets that wait in the queue for too long)
Packet serialization (how long it takes to place a X size packet on the wire)
The first I believe is supported but I haven't been able to see it work.
I have a VM running WANem with 2 NICs that are routing between the server and test network.
I set the following parameters on both NICs:
Packet Limit = 32 (I tried different values from 100 down to 1)
Bandwidth 128Kbps
Delay 10ms (just so the packets counters would show)
I then generate a large amount of data between a server and client (proximately 1+Mbps worth of traffic). Server is Win2K3SP2 and client is WinXPSP3.
Now I believe I should be seeing dropped packets on those counters since the queues should only be holding a max of 32 packets and anything over that should be dropped as per your documentation - "The limit is specified in number of packets. If the output forwarding queue for the selected interface exceeds this limit then WANem drops packets."
3rd party link monitoring software indicates that the bandwidth IS being limited to ~128Kbps as expected but it does NOT detect any packet loss!
Is this a bug or am I doing something wrong?
As for the packet serialization, this is not as important but I simply simulated a 128Kbps link and then sent a large ping across (1400 bytes), the TTL remained at ~1 when it should have increased to about 90ms. Is serialization not simulated?
Your answers regarding the first point are highly appreciated and again thanks for the great tool!
Sincerely,
Nuno
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
To keep it short - I agree this is partly a bug. Needs to be fixed.
--------------------------------------------------------------------------------------------
Why this happens?
WANem in implemented as 2 custom queues in series on a network interface in the egress direction.
1st queue is netem - which is responsible for emulating delay, loss all other characteristics except bandwidth. It also has a limit parameter which is the one exposed in the Web GUI.
2nd queue is htb - which emulates bandwidth. It does not have a limit parameter. [This queue does not apply if there is no bandwidth setting]
In other words - an outgoing packet will pass through netem queue first and then hits htb queue before being transmitted on the external interface.
You are setting a netem delay of 10 ms. You are injecting a traffic of 1 Mbps which is approx 83 packets per second (assuming 1500 MTU). So the spacing between packets is 12 ms approx. So there is not much of a queuing in the netem queue, and packets pass on to the htb queue.
The htb queue seems to buffer packets without an official limit.
So that is why I beleive we dont see a packet drop.
In fact if your 1 Mbps + traffic injector is TCP based then you would not see any drops as tcp adjusts its sending rate as per the network conditions
--------------------------------------------------------------------------------------------
Of course the netem queue "limit" does pose an issue for emulating large bandwidth. I have seen people reporting that they had to increase this parameter to emulate larger bandwidths - right now I dont know the exact bandwidth they tried to emulate
---------------------------------------------------------------------------------------------
I also did a small experiment in the Web GUI which I think supports my analysis.
Set the Packet Limit to 1.
Set the delay to 100 ms.
Click "Apply settings" (GUI responds slow)
Click "Refresh Settings" (GUI responds slow)
Click "Check current status"
This is what I got ...
Interface: eth0
Delay: 10ms
WANem packet limit: 1 packets
Bytes sent: 23641 Packets sent: 18 Packets dropped: 5,
When I changed just the packet limit back to thousand - no packets dropped.
---------------------------------------------------------------------------------------------
So coming back - yes, Packet Limit currently does not apply to the Banddwidth Limitation option. Ideally it should. I am investigating the use of tbf queue instead of htb to see if the packet limit can be applied to bandwidth limitation also.
---------------------------------------------------------------------------------------------
For your second question - packet serialization will not happen the way you expect.
The htb queue just decides how many packets to release in a given time interval based on the Bandwidth setting. I does not seem to give much regard to individual packet size. However the packet is still transmitted at the physical interface rate.
Try a even larger ping spanning multiple packets - you should get a more satisfactory result.
---------------------------------------------------------------------------------------------
For more info on netem and htb check out their web sites. I think the WANem document has a reference to the URLs.
Thanks a lot for your observations Nuno - it was an eye opener. While fixing this may take some time - I will update the documentation for the next release.
Regards,
M.K.Nambiar
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks for the reply and for looking into this.
I work for a WAN Optimization company, so I do these types of tests more often then most. We live off of WAN inefficiencies! :-)
As I told you, I've worked with other WAN Emulators, and Shunra offerings are the only ones that do proper packet loss when you configure bucket size limitation.
All other Linux/BSD based offerings don't do this, I guess because they all use htb queuing.
If you guys could overcome this, it would be very cool!
If you want me to test new versions please let me know, I would be happy to put it through it's passes.
Once again thanks.
Nuno
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Just curious about your comments concerning BSD WAN emulation. PF/ALTQ can use HFSC which is sort of like a HTB scheduler, but PF also supports CBQ and a few others). Dummynet uses a WFQ style scheduler for it's queues. Can you be more specific as to what you tested and the problems you saw?
thx
jim
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've played around with DummyNet (both with a custom interface and via M0n0wall) and also NistNet, and I haven't been able to detect packet loss due to congestion on an external network monitoring device with either. I also have a Network Nightmare device (also BSD based) but I can't remember off the top of my head if it dropped packets the way I need it to. (have to re-test).
The only emulator that I've tested that has been able to do this is Shunra Cloud/Storm.
With the Cloud you have to configure bucket limitation and drop method (Drop Tail or RED), as it is not enabled by default. You can also of course configure the bucked size in packets or bytes.
I don't know why this is, as I'm not that familiar with the queuing algorithms each use and their specific properties as it pertains to bucket size and how drops occur.
While reading the Netem page, I noticed a caveat in the packet loss section:
"When loss is used locally (not on a bridge or router), the loss is reported to the upper level protocols. This may cause TCP to resend and behave as if there was no loss. When testing protocol reponse to loss it is best to use a netem on a bridge or router"
I did configure WANem as a virtual device with two interfaces and routing between the two, so this should not apply to my test I would assume!
The tests that I have performed do show the correct bandwidth limitation, so if I configure a 128Kbps link, and start a FTP transfer, the external network monitor will show that I'm only pushing that much traffic, as the TCP session can't scale above that limit, however, it does not detect any packet loss on the network!
So it looks like the TCP stack is being notified of packet loss and adjusting the throughput, but it's doing it internally to the WAN emulation device somehow without putting anything on the wire?!?!?
If you have a good understanding of how the queues behave and have any suggestions, I would appreciate it.
Nuno
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
To be clear, I don't use WANem. The "appliance" model didn't work for me, and I needed to impair both sent and received packets at different rates based on various source and destination routed networks on the same interfaces. I watch the SF list because I still think it's a very cool project, and they've said they're working on an install rpm for a future release.
My current WAN emulator is FreeBSD/IPfirewall/dummynet. I don't use a web gui to configure.
How/where are you measuring dropped packets?
You can tell dummynet to drop randomly n% of packets in a pipe. Did you try that and what did your measurement device report?
In ipfirewall, "queue" keyword is used in two ways (unfortunatly). At the IPfirewall rule, a queue is used to create asymetric bandwidth allocation based on the WFQ+. In the case of dummynet pipes, the queue keyword is used to define the pipe "buffer" in either slots of MTU or bytes. The default is 50 slots, but if you wanted to force more drops, you could trim that down. The ipfw command can give you stats on the pipes. If you defined dynamic pipes using a mask, remember that you get another set of slots for each new pipe created.
The one thing I wish dummynet had was a way to express "jitter" as a random delay range.
jim
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
To be clear, I don't use WANem. The "appliance" model didn't work for me, and I needed to impair both sent and received packets at different rates based on various source and destination routed networks on the same interfaces. I watch the SF list because I still think it's a very cool project, and they've said they're working on an install rpm for a future release.
My current WAN emulator is FreeBSD/IPfirewall/dummynet. I don't use a web gui to configure.
How/where are you measuring dropped packets?
You can tell dummynet to drop randomly n% of packets in a pipe. Did you try that and what did your measurement device report?
In ipfirewall, "queue" keyword is used in two ways (unfortunatly). At the IPfirewall rule, a queue is used to create asymetric bandwidth allocation based on the WFQ+. In the case of dummynet pipes, the queue keyword is used to define the pipe "buffer" in either slots of MTU or bytes. The default is 50 slots, but if you wanted to force more drops, you could trim that down. The ipfw command can give you stats on the pipes. If you defined dynamic pipes using a mask, remember that you get another set of slots for each new pipe created.
The one thing I wish dummynet had was a way to express "jitter" as a random delay range.
jim
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I was also finding that WANem was doing a lot of buffering no matter what I selected for the Packet Limit value. I was testing WANem using a SIP voice/video client to see the affects of a congested/throttled network on our SIP endpoints. What I found was that when I enabled a bandwidth limit, for a period of time, packet loss would remain at 0% until the packets overflowed some unknown queue. At the point where the queues overflow, packet loss jumps but the problem is now my voice/video was delayed in time by the length of the queue.
I think I managed to make WANem behave more like I expected by adding the following tc command:
# tc qdisc add dev eth0 parent 10:1 handle 20: pfifo limit 50
See if that works for you.
mknambiar can you verify if what I did above is correct? I am in no means a an expert in Linux traffic control commands.
Christian
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi guys, first thanks a bunch for the great work. I just started testing this WAN Emulator and I can tell you that so far I like it better then most open source efforts (dummynet or nistnet based).
There are two requirements needed for accurate WAN simulation as it pertains to link bandwidth limitation.
Packet loss due to congestion (obsolete packets or packets that wait in the queue for too long)
Packet serialization (how long it takes to place a X size packet on the wire)
The first I believe is supported but I haven't been able to see it work.
I have a VM running WANem with 2 NICs that are routing between the server and test network.
I set the following parameters on both NICs:
Packet Limit = 32 (I tried different values from 100 down to 1)
Bandwidth 128Kbps
Delay 10ms (just so the packets counters would show)
I then generate a large amount of data between a server and client (proximately 1+Mbps worth of traffic). Server is Win2K3SP2 and client is WinXPSP3.
The WANem 'Check Current Status" button shows:
Interface: eth1
Delay: 10ms
Bandwidth: 128Kbits/s
WANem packet limit: 32 packets
Bytes sent: 16244769 Packets sent: 22648 Packets dropped: 0,
Interface: eth0
Delay: 10ms
Bandwidth: 128Kbits/s
WANem packet limit: 32 packets
Bytes sent: 749271 Packets sent: 10218 Packets dropped: 0,
Now I believe I should be seeing dropped packets on those counters since the queues should only be holding a max of 32 packets and anything over that should be dropped as per your documentation - "The limit is specified in number of packets. If the output forwarding queue for the selected interface exceeds this limit then WANem drops packets."
3rd party link monitoring software indicates that the bandwidth IS being limited to ~128Kbps as expected but it does NOT detect any packet loss!
Is this a bug or am I doing something wrong?
As for the packet serialization, this is not as important but I simply simulated a 128Kbps link and then sent a large ping across (1400 bytes), the TTL remained at ~1 when it should have increased to about 90ms. Is serialization not simulated?
Your answers regarding the first point are highly appreciated and again thanks for the great tool!
Sincerely,
Nuno
Hi Nuno,
I looked into this issue.
To keep it short - I agree this is partly a bug. Needs to be fixed.
--------------------------------------------------------------------------------------------
Why this happens?
WANem in implemented as 2 custom queues in series on a network interface in the egress direction.
1st queue is netem - which is responsible for emulating delay, loss all other characteristics except bandwidth. It also has a limit parameter which is the one exposed in the Web GUI.
2nd queue is htb - which emulates bandwidth. It does not have a limit parameter. [This queue does not apply if there is no bandwidth setting]
In other words - an outgoing packet will pass through netem queue first and then hits htb queue before being transmitted on the external interface.
You are setting a netem delay of 10 ms. You are injecting a traffic of 1 Mbps which is approx 83 packets per second (assuming 1500 MTU). So the spacing between packets is 12 ms approx. So there is not much of a queuing in the netem queue, and packets pass on to the htb queue.
The htb queue seems to buffer packets without an official limit.
So that is why I beleive we dont see a packet drop.
In fact if your 1 Mbps + traffic injector is TCP based then you would not see any drops as tcp adjusts its sending rate as per the network conditions
--------------------------------------------------------------------------------------------
Of course the netem queue "limit" does pose an issue for emulating large bandwidth. I have seen people reporting that they had to increase this parameter to emulate larger bandwidths - right now I dont know the exact bandwidth they tried to emulate
---------------------------------------------------------------------------------------------
I also did a small experiment in the Web GUI which I think supports my analysis.
Set the Packet Limit to 1.
Set the delay to 100 ms.
Click "Apply settings" (GUI responds slow)
Click "Refresh Settings" (GUI responds slow)
Click "Check current status"
This is what I got ...
Interface: eth0
Delay: 10ms
WANem packet limit: 1 packets
Bytes sent: 23641 Packets sent: 18 Packets dropped: 5,
When I changed just the packet limit back to thousand - no packets dropped.
---------------------------------------------------------------------------------------------
So coming back - yes, Packet Limit currently does not apply to the Banddwidth Limitation option. Ideally it should. I am investigating the use of tbf queue instead of htb to see if the packet limit can be applied to bandwidth limitation also.
---------------------------------------------------------------------------------------------
For your second question - packet serialization will not happen the way you expect.
The htb queue just decides how many packets to release in a given time interval based on the Bandwidth setting. I does not seem to give much regard to individual packet size. However the packet is still transmitted at the physical interface rate.
Try a even larger ping spanning multiple packets - you should get a more satisfactory result.
---------------------------------------------------------------------------------------------
For more info on netem and htb check out their web sites. I think the WANem document has a reference to the URLs.
Thanks a lot for your observations Nuno - it was an eye opener. While fixing this may take some time - I will update the documentation for the next release.
Regards,
M.K.Nambiar
Thanks for the reply and for looking into this.
I work for a WAN Optimization company, so I do these types of tests more often then most. We live off of WAN inefficiencies! :-)
As I told you, I've worked with other WAN Emulators, and Shunra offerings are the only ones that do proper packet loss when you configure bucket size limitation.
All other Linux/BSD based offerings don't do this, I guess because they all use htb queuing.
If you guys could overcome this, it would be very cool!
If you want me to test new versions please let me know, I would be happy to put it through it's passes.
Once again thanks.
Nuno
Just curious about your comments concerning BSD WAN emulation. PF/ALTQ can use HFSC which is sort of like a HTB scheduler, but PF also supports CBQ and a few others). Dummynet uses a WFQ style scheduler for it's queues. Can you be more specific as to what you tested and the problems you saw?
thx
jim
I've played around with DummyNet (both with a custom interface and via M0n0wall) and also NistNet, and I haven't been able to detect packet loss due to congestion on an external network monitoring device with either. I also have a Network Nightmare device (also BSD based) but I can't remember off the top of my head if it dropped packets the way I need it to. (have to re-test).
The only emulator that I've tested that has been able to do this is Shunra Cloud/Storm.
With the Cloud you have to configure bucket limitation and drop method (Drop Tail or RED), as it is not enabled by default. You can also of course configure the bucked size in packets or bytes.
I don't know why this is, as I'm not that familiar with the queuing algorithms each use and their specific properties as it pertains to bucket size and how drops occur.
While reading the Netem page, I noticed a caveat in the packet loss section:
"When loss is used locally (not on a bridge or router), the loss is reported to the upper level protocols. This may cause TCP to resend and behave as if there was no loss. When testing protocol reponse to loss it is best to use a netem on a bridge or router"
I did configure WANem as a virtual device with two interfaces and routing between the two, so this should not apply to my test I would assume!
The tests that I have performed do show the correct bandwidth limitation, so if I configure a 128Kbps link, and start a FTP transfer, the external network monitor will show that I'm only pushing that much traffic, as the TCP session can't scale above that limit, however, it does not detect any packet loss on the network!
So it looks like the TCP stack is being notified of packet loss and adjusting the throughput, but it's doing it internally to the WAN emulation device somehow without putting anything on the wire?!?!?
If you have a good understanding of how the queues behave and have any suggestions, I would appreciate it.
Nuno
To be clear, I don't use WANem. The "appliance" model didn't work for me, and I needed to impair both sent and received packets at different rates based on various source and destination routed networks on the same interfaces. I watch the SF list because I still think it's a very cool project, and they've said they're working on an install rpm for a future release.
My current WAN emulator is FreeBSD/IPfirewall/dummynet. I don't use a web gui to configure.
How/where are you measuring dropped packets?
You can tell dummynet to drop randomly n% of packets in a pipe. Did you try that and what did your measurement device report?
In ipfirewall, "queue" keyword is used in two ways (unfortunatly). At the IPfirewall rule, a queue is used to create asymetric bandwidth allocation based on the WFQ+. In the case of dummynet pipes, the queue keyword is used to define the pipe "buffer" in either slots of MTU or bytes. The default is 50 slots, but if you wanted to force more drops, you could trim that down. The ipfw command can give you stats on the pipes. If you defined dynamic pipes using a mask, remember that you get another set of slots for each new pipe created.
The one thing I wish dummynet had was a way to express "jitter" as a random delay range.
jim
To be clear, I don't use WANem. The "appliance" model didn't work for me, and I needed to impair both sent and received packets at different rates based on various source and destination routed networks on the same interfaces. I watch the SF list because I still think it's a very cool project, and they've said they're working on an install rpm for a future release.
My current WAN emulator is FreeBSD/IPfirewall/dummynet. I don't use a web gui to configure.
How/where are you measuring dropped packets?
You can tell dummynet to drop randomly n% of packets in a pipe. Did you try that and what did your measurement device report?
In ipfirewall, "queue" keyword is used in two ways (unfortunatly). At the IPfirewall rule, a queue is used to create asymetric bandwidth allocation based on the WFQ+. In the case of dummynet pipes, the queue keyword is used to define the pipe "buffer" in either slots of MTU or bytes. The default is 50 slots, but if you wanted to force more drops, you could trim that down. The ipfw command can give you stats on the pipes. If you defined dynamic pipes using a mask, remember that you get another set of slots for each new pipe created.
The one thing I wish dummynet had was a way to express "jitter" as a random delay range.
jim
Hey Nuno,
I was also finding that WANem was doing a lot of buffering no matter what I selected for the Packet Limit value. I was testing WANem using a SIP voice/video client to see the affects of a congested/throttled network on our SIP endpoints. What I found was that when I enabled a bandwidth limit, for a period of time, packet loss would remain at 0% until the packets overflowed some unknown queue. At the point where the queues overflow, packet loss jumps but the problem is now my voice/video was delayed in time by the length of the queue.
I think I managed to make WANem behave more like I expected by adding the following tc command:
# tc qdisc add dev eth0 parent 10:1 handle 20: pfifo limit 50
See if that works for you.
mknambiar can you verify if what I did above is correct? I am in no means a an expert in Linux traffic control commands.
Christian
Sorry Christian-
for some reason I missed seeing your question. Can you tell me what WANem settings you had used prior to running the tc command.
Regards,
M.K.Nambiar