Re: [Linuxptp-users] offset explosion and irregular phc offsets

[Keller, J. E <jac...@in...>]

Hi Julien,

It appears you are using the e1000e driver. Could you get me the device
IDs for the parts that you are doing 1588/PTP on? I believe that what
you are experiencing is actually a driver bug, related to an overflow on
one of the timestamp registers. I am not certain if it has been fixed in
the version of the kernel you are using... Or did you download the
release driver from sourceforge instead of using the upstream?

There is a newer version of the e1000e driver on sourceforge,
http://sourceforge.net/projects/e1000/files/e1000e%20stable/

Also, if you are using the upstream (in-kernel) driver, I believe there
have been some crucial fixes to resolve an issue like this.

I will try to get back to you on this quickly.

Regards,
Jake

On Sun, 2013-08-11 at 22:54 +0100, Julien Houles wrote:
> 
> Configuration :
> Linux 3.5.3
> Intel 82574l with e1000e-2.3.2 drivers
> xenomai-2.6.2.1
> 
>  Because of this message :
> « failed to read clock: Invalid argument »
> I had to modify phc2sys.c :
> The clock_gettime function return is no longer checked in read_phc()
> and update_sync_offset()
> 
> Dear linuxptp users and developers,
> 
> For the introduction (quite long) look at the end of the message.
> We have 50 computers (single board computers with 5 ethernet ports)
>  which we would like to synchronize with a precision of less than one
> microsecond.
> The first solution we imagine is making a chain. We bought 3 boards
> and built the following setup :
> - port 5 (p37p1, ptp5) board 1 linked to port 5 (p37p1, ptp5) board 2
> port 5 board 1 is master and port 5 board 2 is slave
> - port 4 (p36p1, ptp4) board 2 linked to port 4 (p36p1, ptp4)  board 3
> port 4 board 2 is master and port 4 board 3 is slave
> 
> We launch :
> On board 1 :
> ptp4l -i p37p1 -m -P
> phc2sys -s p37p1 -O 0 -m
> 
>  On board 2 :
> ptp4l -i p37p1 -m -P -s
> phc2sys -s /dev/ptp5 -c /dev/ptp4 -m -O 0
> ptp4l -i p36p1 -m -P
> phc2sys -s p37p1 -O 0 -m
> 
> On board 3:
> ptp4l -i p36p1 -m -P -s
> phc2sys -s p36p1 -O 0 -m
> 
>  The ptp4l seems to work well on the 3 boards, the master offset
> converges to a few nanoseconds in a short time. 
> 
>  First problem :
> But after a few minutes, the master offset prints very high
> values (you can have the details at the end of message):
> 
> ptp4l[175293.088]: master offset 40 s2 freq +11975 path delay 751 
> ptp4l[175294.088]: master offset 31 s2 freq +11978 path delay 751 
> ptp4l[175295.089]: master offset -70368744177643 s2 freq -599999999
> path delay 753 
> ptp4l[175296.088]: master offset -70368145816436 s2 freq -599999999
> path delay 4701 
> ptp4l[175297.088]: master offset -70367546002731 s2 freq -599999999
> path delay 8628 
> ptp4l[175298.088]: master offset -70366946185845 s2 freq -599999999
> path delay 12574 
> 
> I’ve got the same behaviour with phc2sys. Have you ever met this
> problem ?
> 
>  Second Problem :
> 
> The rtc synchronization didn’t work well at the beginning on all the
> boards. I had irregular and quite high phc offsets between a ptp clock
> (master) and the CLOCK_REALTIME (slave):
> 
> phc2sys[23132.781]: phc offset 1234 s2 freq -35737 delay 5619 
> phc2sys[23133.781]: phc offset 2024 s2 freq -34577 delay 5609 
> phc2sys[23134.781]: phc offset -1298 s2 freq -37291 delay 5598 
> phc2sys[23135.781]: phc offset -5331 s2 freq -41714 delay 5634 
> phc2sys[23136.782]: phc offset 3452 s2 freq -34530 delay 5594 
> phc2sys[23137.782]: phc offset -4248 s2 freq -41194 delay 5619 
> phc2sys[23138.782]: phc offset -54 s2 freq -38275 delay 5644 
> phc2sys[23139.782]: phc offset 8843 s2 freq -29394 delay 5639 
> phc2sys[23140.783]: phc offset -7006 s2 freq -42590 delay 5630 
>  
> But, one day, I didn’t know how it happened, the results were much
> better (0 to few tens of ns) on one board. A few days after, the
> results became better on an other one. And they are still good now.
> Only the last one still has bad results (see at the end of message)
> (It’s not a problem coming from the boards because I tried on other
> hardwares and I had the same behaviour). Have you met this kind of
> issue ?
> 
>  
> General questions :
> 
> What do you think of such a chain ? Do you think the offset between
> board 1 and board 50  will be acceptable (~ 1µs) ? 
> Could we use a mechanism built on a ethernet hub or switch (regular
> switch, not a IEEE1588 one because too expensive) ? We’ll only use
> these links for a synchronization purpose. Can we expect a constant
> delay with a store and forward switch ? Or should we use a hub ?
> 
> 
> Thank you very much,
> 
>  
> 
> Julien.
>  
> 
> Introduction :
> 
> I'm in charge of the data acquisition for a gamma-ray telescope
> camera. The camera is made of 300 electronic boards, each one sends
> data through a 1 Gb ethernet link. These links are gathered in a stack
> of switches linked to a camera server via two 10 Gb ethernet links. 
> When a board detects and interesting event, a trigger is sent to all
> the other boards. When a board receives this trigger, it generates a
> packet containing the data acquired and sends them on its gigabit
> link. Therefore, we have 300 packets sent exactly (almost) at the same
> time by the boards for each event. The average trigger rate is 10 kHz
> but could be much higher locally because of the poissonian
> distribution of the triggers. The two 10 Gb links are big enough to
> evacuate the average flow but we need to bufferize in the switches for
> higher instantaneous rates.
> We would like to validate the switch stack and the whole acquisition
> chain (the boards won't be ready in a short term). That's why we would
> like to build a simulator of a half camera, at first, providing 150
> gigabit ports. The simulator must be as cheap as possible. We imagined
> a cluster of 50 small motherboads with 5 gigabit ports each
> (http://www.minipc.de/catalog/il/1637 + daughter board with 4 ports),
> using 3 ports for data transfer. We managed to reduce the deltaT
> between the sending of packets on each port to less than 1µs (deltaT
> between the first packet sent, on port 1 and the last packet sent on
> port 3) by using Xenomai and a modified Rtnet library and drivers. We
> modified the drivers in order to reduce the serialization to the
> descriptors adresses passing to the etnernet controllers.
> Now, we have to optimize the inter motherboards synchronization. We
> won't use any interrupt based synch because of a too large jitter on
> interrupt latency. We would like to synchronize the boards rtc with
> ptp and use a preset timing scenario common to all the motherboards.
> Each board would know when to send the next packet. The core dedicated
> to data sending would poll on the rtc (at least when approaching the
> next trigger, use of timers) and send the packet when the right date
> is reached. The other core could be dedicated to ptp synchronization. 
> 
>  
>  PTP outputs :
> 
> BOARD 1 :
> [root@marctaSBC ~]# ptp4l -i p37p1 -m -P 
> ptp4l[22915.242]: selected /dev/ptp5 as PTP clock 
> ptp4l[22915.244]: failed to read out the clock frequency adjustment:
> Operation not supported 
> ptp4l[22915.249]: port 1: INITIALIZING to LISTENING on INITIALIZE 
> ptp4l[22915.249]: port 0: INITIALIZING to LISTENING on INITIALIZE 
> ptp4l[22921.249]: port 1: LISTENING to MASTER on
> ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES 
> [root@marctaSBC linuxptp]# ./phc2sys -s p37p1 -O 0 -m 
> phc2sys[23122.777]: phc offset -24005 s0 freq +0 delay 5644 
> phc2sys[23123.778]: phc offset -21394 s1 freq -34426 delay 5624 
> phc2sys[23124.778]: phc offset 98 s2 freq -34328 delay 5639 
> phc2sys[23125.779]: phc offset -8958 s2 freq -43354 delay 5635 
> phc2sys[23126.779]: phc offset 75 s2 freq -37009 delay 5625 
> phc2sys[23127.779]: phc offset 1537 s2 freq -35524 delay 5629 
> phc2sys[23128.780]: phc offset 54 s2 freq -36546 delay 5609 
> phc2sys[23129.780]: phc offset -4552 s2 freq -41136 delay 5645 
> phc2sys[23130.780]: phc offset -2171 s2 freq -40121 delay 5660 
> phc2sys[23131.781]: phc offset 5434 s2 freq -33167 delay 5619 
> phc2sys[23132.781]: phc offset 1234 s2 freq -35737 delay 5619 
> phc2sys[23133.781]: phc offset 2024 s2 freq -34577 delay 5609 
> phc2sys[23134.781]: phc offset -1298 s2 freq -37291 delay 5598 
> phc2sys[23135.781]: phc offset -5331 s2 freq -41714 delay 5634 
> phc2sys[23136.782]: phc offset 3452 s2 freq -34530 delay 5594 
> phc2sys[23137.782]: phc offset -4248 s2 freq -41194 delay 5619 
> phc2sys[23138.782]: phc offset -54 s2 freq -38275 delay 5644 
> phc2sys[23139.782]: phc offset 8843 s2 freq -29394 delay 5639 
> phc2sys[23140.783]: phc offset -7006 s2 freq -42590 delay 5630 
> phc2sys[23141.783]: phc offset 4072 s2 freq -33614 delay 5855 
> phc2sys[23142.783]: phc offset -1236 s2 freq -37700 delay 5649 
> phc2sys[23143.783]: phc offset 2287 s2 freq -34548 delay 5644 
> phc2sys[23144.784]: phc offset -1243 s2 freq -37392 delay 5629 
> ......
> phc2sys[23739.928]: phc offset 6668 s2 freq -29655 delay 5603 
> phc2sys[23740.928]: phc offset -3730 s2 freq -38052 delay 5819 
> phc2sys[23741.928]: phc offset -640 s2 freq -36081 delay 5774 
> phc2sys[23742.928]: phc offset -9226 s2 freq -44859 delay 5615 
> phc2sys[23743.929]: phc offset -23 s2 freq -38424 delay 5639 
> phc2sys[23744.929]: phc offset 3002 s2 freq -35406 delay 5759 
> phc2sys[23745.929]: phc offset -70368744179916 s2 freq -500000 delay
> 5815 
> phc2sys[23746.929]: phc offset -70368743713845 s2 freq -500000 delay
> 5611 
> phc2sys[23747.930]: phc offset -70368743252576 s2 freq -500000 delay
> 5677 
> phc2sys[23748.930]: phc offset -70368742789004 s2 freq -500000 delay
> 5602 
> phc2sys[23749.930]: phc offset -70368742327066 s2 freq -500000 delay
> 5636 
> phc2sys[23750.931]: phc offset -70368741860621 s2 freq -500000 delay
> 5892 
>  
> BOARD 2 :
> [root@marctaSBC linuxptp]# ptp4l -i p37p1 -m -P -s 
> ptp4l[174464.513]: selected /dev/ptp5 as PTP clock 
> ptp4l[174464.515]: failed to read out the clock frequency adjustment:
> Operation not supported 
> ptp4l[174464.521]: port 1: INITIALIZING to LISTENING on INITIALIZE 
> ptp4l[174464.522]: port 0: INITIALIZING to LISTENING on INITIALIZE 
> ptp4l[174472.039]: port 1: new foreign master 002246.fffe.12838c-1 
> ptp4l[174476.039]: selected best master clock 002246.fffe.12838c 
> ptp4l[174476.039]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE 
> ptp4l[174477.041]: master offset 190119 s0 freq +0 path delay 770 
> ptp4l[174478.040]: master offset 202039 s1 freq +11920 path delay 770 
> ptp4l[174479.040]: master offset -4131 s2 freq +7789 path delay 769 
> ptp4l[174479.040]: port 1: UNCALIBRATED to SLAVE on
> MASTER_CLOCK_SELECTED 
> ptp4l[174480.040]: master offset 59 s2 freq +10740 path delay 769 
> ptp4l[174481.040]: master offset 1342 s2 freq +12040 path delay 767 
> ptp4l[174482.040]: master offset 1206 s2 freq +12307 path delay 769 
> ptp4l[174483.040]: master offset 823 s2 freq +12286 path delay 769 
> ptp4l[174484.040]: master offset 535 s2 freq +12245 path delay 769 
> ptp4l[174485.040]: master offset 205 s2 freq +12075 path delay 771 
> ptp4l[174486.040]: master offset 148 s2 freq +12080 path delay 771 
> ptp4l[174487.040]: master offset 55 s2 freq +12031 path delay 767 
> ptp4l[174488.040]: master offset -1 s2 freq +11991 path delay 767 
> ptp4l[174489.042]: master offset -4 s2 freq +11988 path delay 767 
> ptp4l[174490.040]: master offset -3 s2 freq +11988 path delay 765 
> ptp4l[174491.041]: master offset -2 s2 freq +11988 path delay 762 
> ptp4l[174492.041]: master offset 1 s2 freq +11990 path delay 758 
> ptp4l[174493.041]: master offset 1 s2 freq +11991 path delay 756 
> ptp4l[174494.041]: master offset 0 s2 freq +11990 path delay 756 
> ptp4l[174495.041]: master offset -2 s2 freq +11988 path delay 756 
> ptp4l[174496.041]: master offset -5 s2 freq +11984 path delay 758 
> .........................
> ptp4l[175290.088]: master offset -29 s2 freq +11935 path delay 753 
> ptp4l[175291.088]: master offset -77 s2 freq +11879 path delay 751 
> ptp4l[175292.089]: master offset 9 s2 freq +11942 path delay 751 
> ptp4l[175293.088]: master offset 40 s2 freq +11975 path delay 751 
> ptp4l[175294.088]: master offset 31 s2 freq +11978 path delay 751 
> ptp4l[175295.089]: master offset -70368744177643 s2 freq -599999999
> path delay 753 
> ptp4l[175296.088]: master offset -70368145816436 s2 freq -599999999
> path delay 4701 
> ptp4l[175297.088]: master offset -70367546002731 s2 freq -599999999
> path delay 8628 
> ptp4l[175298.088]: master offset -70366946185845 s2 freq -599999999
> path delay 12574 
> ptp4l[175299.088]: master offset -70366346373840 s2 freq -599999999
> path delay 16521 
> ptp4l[175300.090]: master offset -70365746557429 s2 freq -599999999
> path delay 20782 
> ptp4l[175301.089]: master offset -70365146747351 s2 freq -599999999
> path delay 24696 
> ptp4l[175302.089]: master offset -70364546918013 s2 freq -599999999
> path delay 28662
>  
> 
> [root@marctaSBC linuxptp]# ./phc2sys -s /dev/ptp5 -c /dev/ptp4 -m -O
> 0 
> phc2sys[174550.979]: failed to read out the clock frequency
> adjustment: Operation not supported 
> phc2sys[174551.981]: phc offset 141058 s0 freq +0 delay 10880 
> phc2sys[174552.981]: phc offset 143298 s1 freq +2238 delay 10808 
> phc2sys[174553.982]: phc offset -4053 s2 freq -1815 delay 10799 
> phc2sys[174554.983]: phc offset 49 s2 freq +1072 delay 10800 
> phc2sys[174555.983]: phc offset 1344 s2 freq +2381 delay 10919 
> phc2sys[174556.984]: phc offset 1057 s2 freq +2497 delay 10535 
> phc2sys[174557.985]: phc offset 1176 s2 freq +2934 delay 10839 
> phc2sys[174558.985]: phc offset 560 s2 freq +2670 delay 10919 
> phc2sys[174559.986]: phc offset 82 s2 freq +2360 delay 10919 
> phc2sys[174560.987]: phc offset 109 s2 freq +2412 delay 10919 
> phc2sys[174561.987]: phc offset -14 s2 freq +2322 delay 10959 
> phc2sys[174562.988]: phc offset 70 s2 freq +2401 delay 10879 
> phc2sys[174563.989]: phc offset -29 s2 freq +2323 delay 10799 
> phc2sys[174564.989]: phc offset -110 s2 freq +2234 delay 10919 
> phc2sys[174565.989]: phc offset 31 s2 freq +2342 delay 10879 
> phc2sys[174566.989]: phc offset -171 s2 freq +2149 delay 10415 
> phc2sys[174567.990]: phc offset 199 s2 freq +2468 delay 10879 
> phc2sys[174568.990]: phc offset 114 s2 freq +2442 delay 10919 
> phc2sys[174569.990]: phc offset -13 s2 freq +2350 delay 10879 
> phc2sys[174570.991]: phc offset -5 s2 freq +2354 delay 10839 
> phc2sys[174571.991]: phc offset 64 s2 freq +2421 delay 10839 
> .........................
> phc2sys[175291.197]: phc offset 35 s2 freq +2327 delay 10839 
> phc2sys[175292.197]: phc offset 5 s2 freq +2308 delay 10879 
> phc2sys[175293.197]: phc offset -2 s2 freq +2302 delay 10959 
> phc2sys[175294.197]: phc offset -10 s2 freq +2294 delay 10879 
> phc2sys[175295.198]: phc offset -65055841 s2 freq -65053540 delay
> 10879 
> phc2sys[175296.198]: phc offset -599947449 s2 freq -599999999 delay
> 11630 
> phc2sys[175297.198]: phc offset -599989526 s2 freq -599999999 delay
> 17471 
> phc2sys[175298.199]: phc offset -600004949 s2 freq -599999999 delay
> 17343 
> phc2sys[175299.199]: phc offset -600020181 s2 freq -599999999 delay
> 17471 
> phc2sys[175300.199]: phc offset -600035608 s2 freq -599999999 delay
> 17471 
> phc2sys[175301.199]: phc offset -600051031 s2 freq -599999999 delay
> 17727 
>  
>  
> [root@marctaSBC linuxptp]# ptp4l -i p36p1 -m -P 
> ptp4l[174595.636]: selected /dev/ptp4 as PTP clock 
> ptp4l[174595.637]: failed to read out the clock frequency adjustment:
> Operation not supported 
> ptp4l[174595.644]: port 1: INITIALIZING to LISTENING on INITIALIZE 
> ptp4l[174595.646]: port 0: INITIALIZING to LISTENING on INITIALIZE 
> ptp4l[174601.644]: port 1: LISTENING to MASTER on
> ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES 
>  
> [root@marctaSBC linuxptp]# ./phc2sys -s p37p1 -O 0 -m 
> phc2sys[174620.093]: phc offset 113271 s0 freq +0 delay 5474 
> phc2sys[174621.094]: phc offset 113264 s1 freq -38072 delay 5484 
> phc2sys[174622.094]: phc offset -28 s2 freq -38100 delay 5494 
> phc2sys[174623.095]: phc offset 40 s2 freq -38040 delay 5484 
> phc2sys[174624.095]: phc offset -75 s2 freq -38143 delay 5494 
> phc2sys[174625.095]: phc offset 35 s2 freq -38056 delay 5554 
> phc2sys[174626.095]: phc offset -20 s2 freq -38100 delay 5464 
> phc2sys[174627.095]: phc offset 134 s2 freq -37952 delay 5494 
> phc2sys[174628.096]: phc offset 49 s2 freq -37997 delay 5504 
> phc2sys[174629.096]: phc offset -113 s2 freq -38144 delay 5504 
> phc2sys[174630.096]: phc offset -13 s2 freq -38078 delay 5504 
> phc2sys[174631.096]: phc offset 8 s2 freq -38061 delay 5494 
> phc2sys[174632.097]: phc offset 0 s2 freq -38067 delay 5545 
> phc2sys[174633.097]: phc offset 9 s2 freq -38058 delay 5504 
> phc2sys[174634.097]: phc offset -42 s2 freq -38106 delay 5514 
> phc2sys[174635.097]: phc offset -89 s2 freq -38166 delay 5474 
> phc2sys[174636.098]: phc offset 24 s2 freq -38079 delay 5529 
> phc2sys[174637.098]: phc offset 19 s2 freq -38077 delay 5464 
> phc2sys[174638.098]: phc offset 47 s2 freq -38044 delay 5494 
> phc2sys[174639.098]: phc offset 16 s2 freq -38060 delay 5484 
> phc2sys[174640.098]: phc offset -25 s2 freq -38097 delay 5484 
> phc2sys[174641.099]: phc offset 80 s2 freq -37999 delay 5564 
> phc2sys[174642.099]: phc offset -4 s2 freq -38059 delay 5559 
> phc2sys[174643.099]: phc offset -47 s2 freq -38103 delay 5484 
> ..........................
> phc2sys[175291.248]: phc offset 14 s2 freq -38014 delay 5494 
> phc2sys[175292.248]: phc offset -138 s2 freq -38162 delay 5479 
> phc2sys[175293.248]: phc offset -14 s2 freq -38079 delay 5534 
> phc2sys[175294.249]: phc offset 45 s2 freq -38025 delay 5484 
> phc2sys[175295.249]: phc offset -95729399 s2 freq -500000 delay 5484 
> phc2sys[175296.249]: phc offset -695151138 s2 freq -500000 delay 5496 
> phc2sys[175297.249]: phc offset -1294563525 s2 freq -500000 delay
> 5497 
> phc2sys[175298.249]: phc offset -1893973598 s2 freq -500000 delay
> 5466 
> phc2sys[175299.250]: phc offset -2493408464 s2 freq -500000 delay
> 5466 
> phc2sys[175300.250]: phc offset -3092833604 s2 freq -500000 delay
> 5507 
> phc2sys[175301.250]: phc offset -3692242108 s2 freq -500000 delay
> 5511 
> 
> BOARD 3 :
> [root@marctaSBC ~]# ptp4l -i p36p1 -m -P -s 
> ptp4l[23024.269]: selected /dev/ptp3 as PTP clock 
> ptp4l[23024.271]: failed to read out the clock frequency adjustment:
> Operation not supported 
> ptp4l[23024.277]: port 1: INITIALIZING to LISTENING on INITIALIZE 
> ptp4l[23024.277]: port 0: INITIALIZING to LISTENING on INITIALIZE 
> ptp4l[23031.766]: port 1: new foreign master 002246.fffe.1283c7-1 
> ptp4l[23035.766]: selected best master clock 002246.fffe.1283c7 
> ptp4l[23035.767]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE 
> ptp4l[23036.768]: master offset 93070 s0 freq +0 path delay 751 
> ptp4l[23037.769]: master offset 62384 s1 freq -30686 path delay 751 
> ptp4l[23038.768]: master offset -5272 s2 freq -35958 path delay 754 
> ptp4l[23038.768]: port 1: UNCALIBRATED to SLAVE on
> MASTER_CLOCK_SELECTED 
> ptp4l[23039.768]: master offset -35 s2 freq -32303 path delay 754 
> ptp4l[23040.769]: master offset 1422 s2 freq -30856 path delay 756 
> ptp4l[23041.768]: master offset 1486 s2 freq -30366 path delay 757 
> ptp4l[23042.768]: master offset 1110 s2 freq -30296 path delay 759 
> ptp4l[23043.768]: master offset 600 s2 freq -30473 path delay 759 
> ptp4l[23044.768]: master offset 340 s2 freq -30553 path delay 761 
> ptp4l[23045.769]: master offset 85 s2 freq -30706 path delay 764 
> ptp4l[23046.768]: master offset -159 s2 freq -30925 path delay 764 
> ptp4l[23047.768]: master offset 61 s2 freq -30752 path delay 762 
> ptp4l[23048.768]: master offset 62 s2 freq -30733 path delay 762 
> ptp4l[23049.768]: master offset 28 s2 freq -30748 path delay 762 
> ptp4l[23050.768]: master offset 40 s2 freq -30728 path delay 758 
> ptp4l[23051.768]: master offset 5 s2 freq -30751 path delay 758 
> ptp4l[23052.768]: master offset -87 s2 freq -30841 path delay 758 
> ptp4l[23053.769]: master offset -21 s2 freq -30802 path delay 758 
> ptp4l[23054.768]: master offset -53 s2 freq -30840 path delay 758 
> ptp4l[23055.768]: master offset 67 s2 freq -30736 path delay 760 
> ptp4l[23056.769]: master offset 131 s2 freq -30652 path delay 758 
> ptp4l[23057.768]: master offset -66 s2 freq -30809 path delay 760 
> ptp4l[23058.768]: master offset -37 s2 freq -30800 path delay 758 
> ptp4l[23059.768]: master offset -110 s2 freq -30884 path delay 758 
> ptp4l[23060.768]: master offset -50 s2 freq -30857 path delay 760 
> .................................
> ptp4l[23719.791]: master offset -93 s2 freq -30892 path delay 768 
> ptp4l[23720.793]: master offset -30 s2 freq -30856 path delay 766 
> ptp4l[23721.792]: master offset -9 s2 freq -30844 path delay 766 
> ptp4l[23722.792]: master offset 13 s2 freq -30825 path delay 766 
> ptp4l[23723.791]: master offset -30626099 s2 freq -30656933 path delay
> 766 
> ptp4l[23724.791]: master offset -316853487 s2 freq -326072151 path
> delay 479 
> ptp4l[23725.792]: master offset -591085085 s2 freq -599999999 path
> delay -1810 
> ptp4l[23726.791]: master offset -591811507 s2 freq -599999999 path
> delay -1768 
> ptp4l[23727.792]: master offset -591864410 s2 freq -599999999 path
> delay -1724 
> ptp4l[23728.793]: master offset -591917378 s2 freq -599999999 path
> delay -1679 
> ptp4l[23729.792]: master offset -591970343 s2 freq -599999999 path
> delay -1637 
> ptp4l[23730.792]: master offset -592023184 s2 freq -599999999 path
> delay -1591 
> ptp4l[23731.792]: master offset -592076091 s2 freq -599999999 path
> delay -1543 
>  
> 
> [root@marctaSBC linuxptp]# ./phc2sys -s p36p1 -O 0 -m 
> phc2sys[23107.224]: phc offset 305036 s0 freq +0 delay 6383 
> phc2sys[23108.225]: phc offset 305003 s1 freq -48438 delay 6293 
> phc2sys[23109.225]: phc offset 79 s2 freq -48359 delay 6365 
> phc2sys[23110.226]: phc offset 216 s2 freq -48198 delay 6323 
> phc2sys[23111.226]: phc offset 146 s2 freq -48203 delay 6357 
> phc2sys[23112.226]: phc offset 93 s2 freq -48213 delay 6353 
> phc2sys[23113.226]: phc offset -26 s2 freq -48304 delay 6293 
> phc2sys[23114.227]: phc offset -62 s2 freq -48348 delay 6405 
> phc2sys[23115.227]: phc offset -148 s2 freq -48452 delay 6357 
> phc2sys[23116.227]: phc offset -114 s2 freq -48463 delay 6293 
> phc2sys[23117.227]: phc offset -29 s2 freq -48412 delay 6293 
> phc2sys[23118.227]: phc offset 82 s2 freq -48309 delay 6353 
> phc2sys[23119.228]: phc offset 167 s2 freq -48200 delay 6356 
> phc2sys[23120.228]: phc offset 85 s2 freq -48232 delay 6353 
> phc2sys[23121.228]: phc offset 82 s2 freq -48209 delay 6293 
> phc2sys[23122.228]: phc offset 25 s2 freq -48242 delay 6353 
> phc2sys[23123.228]: phc offset -35 s2 freq -48294 delay 6353 
> phc2sys[23124.229]: phc offset 33 s2 freq -48237 delay 6372 
> phc2sys[23125.229]: phc offset -88 s2 freq -48348 delay 6357 
> phc2sys[23126.229]: phc offset -98 s2 freq -48384 delay 6406 
> phc2sys[23127.229]: phc offset -33 s2 freq -48349 delay 6353 
> phc2sys[23128.230]: phc offset -18 s2 freq -48343 delay 6292 
> phc2sys[23129.230]: phc offset 50 s2 freq -48281 delay 6353 
> phc2sys[23130.230]: phc offset -8 s2 freq -48324 delay 6417 
> phc2sys[23131.230]: phc offset -26 s2 freq -48344 delay 6537 
> phc2sys[23132.230]: phc offset 22 s2 freq -48304 delay 6356 
> phc2sys[23133.231]: phc offset -5 s2 freq -48324 delay 6447 
> phc2sys[23134.231]: phc offset 21 s2 freq -48300 delay 6413 
> phc2sys[23135.231]: phc offset 188 s2 freq -48127 delay 6356 
> phc2sys[23136.231]: phc offset 82 s2 freq -48176 delay 6353 
> phc2sys[23137.232]: phc offset -88 s2 freq -48322 delay 6293 
> phc2sys[23138.232]: phc offset -259 s2 freq -48519 delay 6383 
> phc2sys[23139.232]: phc offset -51 s2 freq -48389 delay 6361 
> phc2sys[23140.232]: phc offset 169 s2 freq -48184 delay 6353 
> ..............................
> phc2sys[23719.363]: phc offset -14 s2 freq -48237 delay 6293 
> phc2sys[23720.363]: phc offset -95 s2 freq -48322 delay 6342 
> phc2sys[23721.363]: phc offset 14 s2 freq -48242 delay 6357 
> phc2sys[23722.363]: phc offset -2 s2 freq -48253 delay 6357 
> phc2sys[23723.364]: phc offset -11 s2 freq -48263 delay 6293 
> phc2sys[23724.364]: phc offset -17523658 s2 freq -500000 delay 6293 
> phc2sys[23725.364]: phc offset -216745503 s2 freq -500000 delay 6296 
> phc2sys[23726.364]: phc offset -698745647 s2 freq -500000 delay 6360 
> phc2sys[23727.364]: phc offset -1298102527 s2 freq -500000 delay 6359 
> phc2sys[23728.365]: phc offset -1897535882 s2 freq -500000 delay 6344 
> phc2sys[23729.365]: phc offset -2496933568 s2 freq -500000 delay 6420 
> phc2sys[23730.365]: phc offset -3096290190 s2 freq -500000 delay 6296 
> phc2sys[23731.365]: phc offset -3695643149 s2 freq -500000 delay 6356 
>