After examining the OPLK stack, the PCAP driver in particular, I noticed in edrv-pcap_linux.c that sending of frames is implemented using pcap_sendpacket(). Given that OPLK implements time-triggered sending of the PReq frames, I am wondering why more efficient frame sending (i.e. pcap_sendqueue_transmit()) is not used?
Also, we captured some the traffic for the network with the MN and two FPGA-based CNs (Altera DE2-115) connected in line topology. After several experiments we configured PResTimeout for both CNs to be 25us (this value preserves correct network operation without collisions) and WaitSoC parameter to 0 (sending the first Preq immediately after SoC). As expected, configured Timeout values deviates significantly, which is not surprise as we use PCAP driver in the MN. However, interesting effect is that measured times SoC-PReq1, PReq1-PReq2 and PReq2-SoA are about 30us, 49us, and 32us, respectively, which is odd, because I would expect Preq1-PReq2 and PReq2-SoA to have comparable values (as both CN1 and CN2 are configured with the same PResTimeout of 25us). Can someone explain such discrepancy? I assume this is an effect of the PCAP driver implementation.
Cheers,
Mladen
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
sounds promising! It would be interesting what SoC jitter could be achieved by using the sendqueue interface - do you already have results?
Which OPLK version did you use observing such deviations? in V2.5.0 the immediate mode support was added which reduces latency in Rx direction; maybe it affects Tx as well (positively or negatively...)!?
Best regards,
Joerg
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We have not integrated sendqueue pcap interface yet. Certainly, I think it would be worth trying. I just wanted to check if somebody already thought about it, because I would not like to dedicate too much effort to something that (possibly) would not provide any benefit. :) However, given the way openPOWERLINK is designed (time-triggered frame sending by the MN), I think that some performance gain could be achieved.
Regarding the second question, we have used V2.2.1 (i.e. the one that does not use immediate mode). I suspect that this kind of behavior is related to the frame buffering in the NIC driver, however, I have not find a way to confirm it yet. Additionally, we experimented with different settings of the PResTimeout values for the same network setup. Below you can find the results. I note that we used special equipment with highly accurate timestamps for collecting the Wireshark pcap files.
Configuration with PResTimeout values set to 25us
SoC-PReq1: 29.5us
PReq1-PReq2: 48.5us
PReq2-SoA: 31.8us
Configuration with PResTimeout values set to 50us
SoC-PReq1: 29.6us
PReq1-PReq2: 73.4us
PReq2-SoA: 66.5us
Configuration with PResTimeout values set to 100us
SoC-PReq1: 55.4us
PReq1-PReq2: 107.9us
PReq2-SoA: 117.4us
I note here that above values are average values. Of course, there is some jitter (in the order of us) and some outliers (in the order of tens of us). For all cases WaitSoC2PReq was set to 0. Could this cause some load to the CPU or the driver and affect the behavior in some way. Also, it is interesting to see that SoC-PReq is increased when the PResTimeout is increased, which is a bit odd to me.
Once again, I note that the network operated without errors (no collisions) and without dropping packets.
Regards,
Mladen
Last edit: Mladen Knezic 2016-11-27
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
One additional remark. We compiled the OPLK for Linux patched with RT_PREEMPT using the "Link to Application" configuration option for CFG_BUILD_KERNEL_STACK (as we modified demo_mn_qt example app). Could "Linux Userspace Daemon" configuration option offer better performance?
Regards,
Mladen
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I assume that with "better performance" you mean less packet jitter, right?
Unfortunately we don't really have extensive test series comparing default pcap mode to the pcap deamon, but in general I wouldn't expect a lot of network performance improvements if any at all.
Instead, the pcap deamon option should decrease the cpu load a bit.
Regards,
Norman
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I also think it is worth trying. The needed modifications for a trial might be limited to edrvcyclic, since this module receives a Tx buffer list from dllk, which could be forwarded to the PCAP's sendqueue interface. You might look into the edrvcyclic.c focusing on EDRV_USE_TTTX set to TRUE (used with i210) or edrvcyclic-openmac.c which employ the time triggered Tx hardware support.
Please keep the community posted about your progress! As soon as you have results we could discuss the next steps...
Best regards,
Joerg
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi all,
After examining the OPLK stack, the PCAP driver in particular, I noticed in edrv-pcap_linux.c that sending of frames is implemented using pcap_sendpacket(). Given that OPLK implements time-triggered sending of the PReq frames, I am wondering why more efficient frame sending (i.e. pcap_sendqueue_transmit()) is not used?
Also, we captured some the traffic for the network with the MN and two FPGA-based CNs (Altera DE2-115) connected in line topology. After several experiments we configured PResTimeout for both CNs to be 25us (this value preserves correct network operation without collisions) and WaitSoC parameter to 0 (sending the first Preq immediately after SoC). As expected, configured Timeout values deviates significantly, which is not surprise as we use PCAP driver in the MN. However, interesting effect is that measured times SoC-PReq1, PReq1-PReq2 and PReq2-SoA are about 30us, 49us, and 32us, respectively, which is odd, because I would expect Preq1-PReq2 and PReq2-SoA to have comparable values (as both CN1 and CN2 are configured with the same PResTimeout of 25us). Can someone explain such discrepancy? I assume this is an effect of the PCAP driver implementation.
Cheers,
Mladen
Hello Mladen,
sounds promising! It would be interesting what SoC jitter could be achieved by using the sendqueue interface - do you already have results?
Which OPLK version did you use observing such deviations? in V2.5.0 the immediate mode support was added which reduces latency in Rx direction; maybe it affects Tx as well (positively or negatively...)!?
Best regards,
Joerg
Hi Joerg,
Thank you for your answer.
We have not integrated sendqueue pcap interface yet. Certainly, I think it would be worth trying. I just wanted to check if somebody already thought about it, because I would not like to dedicate too much effort to something that (possibly) would not provide any benefit. :) However, given the way openPOWERLINK is designed (time-triggered frame sending by the MN), I think that some performance gain could be achieved.
Regarding the second question, we have used V2.2.1 (i.e. the one that does not use immediate mode). I suspect that this kind of behavior is related to the frame buffering in the NIC driver, however, I have not find a way to confirm it yet. Additionally, we experimented with different settings of the PResTimeout values for the same network setup. Below you can find the results. I note that we used special equipment with highly accurate timestamps for collecting the Wireshark pcap files.
Configuration with PResTimeout values set to 25us
SoC-PReq1: 29.5us
PReq1-PReq2: 48.5us
PReq2-SoA: 31.8us
Configuration with PResTimeout values set to 50us
SoC-PReq1: 29.6us
PReq1-PReq2: 73.4us
PReq2-SoA: 66.5us
Configuration with PResTimeout values set to 100us
SoC-PReq1: 55.4us
PReq1-PReq2: 107.9us
PReq2-SoA: 117.4us
I note here that above values are average values. Of course, there is some jitter (in the order of us) and some outliers (in the order of tens of us). For all cases WaitSoC2PReq was set to 0. Could this cause some load to the CPU or the driver and affect the behavior in some way. Also, it is interesting to see that SoC-PReq is increased when the PResTimeout is increased, which is a bit odd to me.
Once again, I note that the network operated without errors (no collisions) and without dropping packets.
Regards,
Mladen
Last edit: Mladen Knezic 2016-11-27
One additional remark. We compiled the OPLK for Linux patched with RT_PREEMPT using the "Link to Application" configuration option for CFG_BUILD_KERNEL_STACK (as we modified demo_mn_qt example app). Could "Linux Userspace Daemon" configuration option offer better performance?
Regards,
Mladen
Hi Mladen,
I assume that with "better performance" you mean less packet jitter, right?
Unfortunately we don't really have extensive test series comparing default pcap mode to the pcap deamon, but in general I wouldn't expect a lot of network performance improvements if any at all.
Instead, the pcap deamon option should decrease the cpu load a bit.
Regards,
Norman
Hi Mladen,
I also think it is worth trying. The needed modifications for a trial might be limited to edrvcyclic, since this module receives a Tx buffer list from dllk, which could be forwarded to the PCAP's sendqueue interface. You might look into the edrvcyclic.c focusing on EDRV_USE_TTTX set to TRUE (used with i210) or edrvcyclic-openmac.c which employ the time triggered Tx hardware support.
Please keep the community posted about your progress! As soon as you have results we could discuss the next steps...
Best regards,
Joerg