[Linuxptp-users] offset explosion and irregular phc offsets
PTP IEEE 1588 stack for Linux
Brought to you by:
rcochran
Menu
▾
▴
[Linuxptp-users] offset explosion and irregular phc offsets
|
From: Julien H. <ju...@ya...> - 2013-08-11 21:55:08
|
Configuration : Linux 3.5.3 Intel 82574l with e1000e-2.3.2 drivers xenomai-2.6.2.1 Because of this message : « failed to read clock: Invalid argument » I had to modify phc2sys.c : The clock_gettime function return is no longer checked in read_phc() and update_sync_offset() Dear linuxptp users and developers, For the introduction (quite long) look at the end of the message. We have 50 computers (single board computers with 5 ethernet ports) which we would like to synchronize with a precision of less than one microsecond. The first solution we imagine is making a chain. We bought 3 boards and built the following setup : - port 5 (p37p1, ptp5) board 1 linked to port 5 (p37p1, ptp5) board 2 port 5 board 1 is master and port 5 board 2 is slave - port 4 (p36p1, ptp4) board 2 linked to port 4 (p36p1, ptp4) board 3 port 4 board 2 is master and port 4 board 3 is slave We launch : On board 1 : ptp4l -i p37p1 -m -P phc2sys -s p37p1 -O 0 -m On board 2 : ptp4l -i p37p1 -m -P -s phc2sys -s /dev/ptp5 -c /dev/ptp4 -m -O 0 ptp4l -i p36p1 -m -P phc2sys -s p37p1 -O 0 -m On board 3: ptp4l -i p36p1 -m -P -s phc2sys -s p36p1 -O 0 -m The ptp4l seems to work well on the 3 boards, the master offset converges to a few nanoseconds in a short time. First problem : But after a few minutes, the master offset prints very high values (you can have the details at the end of message): ptp4l[175293.088]: master offset 40 s2 freq +11975 path delay 751 ptp4l[175294.088]: master offset 31 s2 freq +11978 path delay 751 ptp4l[175295.089]: master offset -70368744177643 s2 freq -599999999 path delay 753 ptp4l[175296.088]: master offset -70368145816436 s2 freq -599999999 path delay 4701 ptp4l[175297.088]: master offset -70367546002731 s2 freq -599999999 path delay 8628 ptp4l[175298.088]: master offset -70366946185845 s2 freq -599999999 path delay 12574 I’ve got the same behaviour with phc2sys. Have you ever met this problem ? Second Problem : The rtc synchronization didn’t work well at the beginning on all the boards. I had irregular and quite high phc offsets between a ptp clock (master) and the CLOCK_REALTIME (slave): phc2sys[23132.781]: phc offset 1234 s2 freq -35737 delay 5619 phc2sys[23133.781]: phc offset 2024 s2 freq -34577 delay 5609 phc2sys[23134.781]: phc offset -1298 s2 freq -37291 delay 5598 phc2sys[23135.781]: phc offset -5331 s2 freq -41714 delay 5634 phc2sys[23136.782]: phc offset 3452 s2 freq -34530 delay 5594 phc2sys[23137.782]: phc offset -4248 s2 freq -41194 delay 5619 phc2sys[23138.782]: phc offset -54 s2 freq -38275 delay 5644 phc2sys[23139.782]: phc offset 8843 s2 freq -29394 delay 5639 phc2sys[23140.783]: phc offset -7006 s2 freq -42590 delay 5630 But, one day, I didn’t know how it happened, the results were much better (0 to few tens of ns) on one board. A few days after, the results became better on an other one. And they are still good now. Only the last one still has bad results (see at the end of message) (It’s not a problem coming from the boards because I tried on other hardwares and I had the same behaviour). Have you met this kind of issue ? General questions : What do you think of such a chain ? Do you think the offset between board 1 and board 50 will be acceptable (~ 1µs) ? Could we use a mechanism built on a ethernet hub or switch (regular switch, not a IEEE1588 one because too expensive) ? We’ll only use these links for a synchronization purpose. Can we expect a constant delay with a store and forward switch ? Or should we use a hub ? Thank you very much, Julien. Introduction : I'm in charge of the data acquisition for a gamma-ray telescope camera. The camera is made of 300 electronic boards, each one sends data through a 1 Gb ethernet link. These links are gathered in a stack of switches linked to a camera server via two 10 Gb ethernet links. When a board detects and interesting event, a trigger is sent to all the other boards. When a board receives this trigger, it generates a packet containing the data acquired and sends them on its gigabit link. Therefore, we have 300 packets sent exactly (almost) at the same time by the boards for each event. The average trigger rate is 10 kHz but could be much higher locally because of the poissonian distribution of the triggers. The two 10 Gb links are big enoughto evacuate the average flow but we need to bufferize in the switches for higher instantaneous rates. We would like to validate the switch stack and the whole acquisition chain (the boards won't be ready in a short term). That's why we wouldlike to build a simulator of a half camera, at first, providing 150 gigabit ports. The simulator must be as cheap as possible. We imagined a cluster of 50 small motherboads with 5 gigabit ports each (http://www.minipc.de/catalog/il/1637 + daughter board with 4 ports), using 3 ports for data transfer. We managed to reduce the deltaT between the sending of packets on each port to less than 1µs (deltaT between the first packet sent, on port 1 and the last packet sent on port 3) by using Xenomai and a modified Rtnet library and drivers. We modified the drivers in order to reduce the serialization to the descriptors adresses passing to the etnernet controllers. Now, we have to optimize the inter motherboards synchronization. We won't use any interrupt based synch because of a too large jitter on interrupt latency. We would like to synchronize the boards rtc with ptp and use a preset timing scenario common to all the motherboards. Each board would know when to send the next packet. The core dedicated to data sending would poll on the rtc (at least when approaching the next trigger, use of timers) and send the packet when the right date is reached. The other core could be dedicated to ptp synchronization. PTP outputs : BOARD 1 : [root@marctaSBC ~]# ptp4l -i p37p1 -m -P ptp4l[22915.242]: selected /dev/ptp5 as PTP clock ptp4l[22915.244]: failed to read out the clock frequency adjustment: Operation not supported ptp4l[22915.249]: port 1: INITIALIZING to LISTENING on INITIALIZE ptp4l[22915.249]: port 0: INITIALIZING to LISTENING on INITIALIZE ptp4l[22921.249]: port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES [root@marctaSBC linuxptp]# ./phc2sys -s p37p1 -O 0 -m phc2sys[23122.777]: phc offset -24005 s0 freq +0 delay 5644 phc2sys[23123.778]: phc offset -21394 s1 freq -34426 delay 5624 phc2sys[23124.778]: phc offset 98 s2 freq -34328 delay 5639 phc2sys[23125.779]: phc offset -8958 s2 freq -43354 delay 5635 phc2sys[23126.779]: phc offset 75 s2 freq -37009 delay 5625 phc2sys[23127.779]: phc offset 1537 s2 freq -35524 delay 5629 phc2sys[23128.780]: phc offset 54 s2 freq -36546 delay 5609 phc2sys[23129.780]: phc offset -4552 s2 freq -41136 delay 5645 phc2sys[23130.780]: phc offset -2171 s2 freq -40121 delay 5660 phc2sys[23131.781]: phc offset 5434 s2 freq -33167 delay 5619 phc2sys[23132.781]: phc offset 1234 s2 freq -35737 delay 5619 phc2sys[23133.781]: phc offset 2024 s2 freq -34577 delay 5609 phc2sys[23134.781]: phc offset -1298 s2 freq -37291 delay 5598 phc2sys[23135.781]: phc offset -5331 s2 freq -41714 delay 5634 phc2sys[23136.782]: phc offset 3452 s2 freq -34530 delay 5594 phc2sys[23137.782]: phc offset -4248 s2 freq -41194 delay 5619 phc2sys[23138.782]: phc offset -54 s2 freq -38275 delay 5644 phc2sys[23139.782]: phc offset 8843 s2 freq -29394 delay 5639 phc2sys[23140.783]: phc offset -7006 s2 freq -42590 delay 5630 phc2sys[23141.783]: phc offset 4072 s2 freq -33614 delay 5855 phc2sys[23142.783]: phc offset -1236 s2 freq -37700 delay 5649 phc2sys[23143.783]: phc offset 2287 s2 freq -34548 delay 5644 phc2sys[23144.784]: phc offset -1243 s2 freq -37392 delay 5629 ...... phc2sys[23739.928]: phc offset 6668 s2 freq -29655 delay 5603 phc2sys[23740.928]: phc offset -3730 s2 freq -38052 delay 5819 phc2sys[23741.928]: phc offset -640 s2 freq -36081 delay 5774 phc2sys[23742.928]: phc offset -9226 s2 freq -44859 delay 5615 phc2sys[23743.929]: phc offset -23 s2 freq -38424 delay 5639 phc2sys[23744.929]: phc offset 3002 s2 freq -35406 delay 5759 phc2sys[23745.929]: phc offset -70368744179916 s2 freq -500000 delay 5815 phc2sys[23746.929]: phc offset -70368743713845 s2 freq -500000 delay 5611 phc2sys[23747.930]: phc offset -70368743252576 s2 freq -500000 delay 5677 phc2sys[23748.930]: phc offset -70368742789004 s2 freq -500000 delay 5602 phc2sys[23749.930]: phc offset -70368742327066 s2 freq -500000 delay 5636 phc2sys[23750.931]: phc offset -70368741860621 s2 freq -500000 delay 5892 BOARD 2 : [root@marctaSBC linuxptp]# ptp4l -i p37p1 -m -P -s ptp4l[174464.513]: selected /dev/ptp5 as PTP clock ptp4l[174464.515]: failed to read out the clock frequency adjustment: Operation not supported ptp4l[174464.521]: port 1: INITIALIZING to LISTENING on INITIALIZE ptp4l[174464.522]: port 0: INITIALIZING to LISTENING on INITIALIZE ptp4l[174472.039]: port 1: new foreign master 002246.fffe.12838c-1 ptp4l[174476.039]: selected best master clock 002246.fffe.12838c ptp4l[174476.039]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE ptp4l[174477.041]: master offset 190119 s0 freq +0 path delay 770 ptp4l[174478.040]: master offset 202039 s1 freq +11920 path delay 770 ptp4l[174479.040]: master offset -4131 s2 freq +7789 path delay 769 ptp4l[174479.040]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED ptp4l[174480.040]: master offset 59 s2 freq +10740 path delay 769 ptp4l[174481.040]: master offset 1342 s2 freq +12040 path delay 767 ptp4l[174482.040]: master offset 1206 s2 freq +12307 path delay 769 ptp4l[174483.040]: master offset 823 s2 freq +12286 path delay 769 ptp4l[174484.040]: master offset 535 s2 freq +12245 path delay 769 ptp4l[174485.040]: master offset 205 s2 freq +12075 path delay 771 ptp4l[174486.040]: master offset 148 s2 freq +12080 path delay 771 ptp4l[174487.040]: master offset 55 s2 freq +12031 path delay 767 ptp4l[174488.040]: master offset -1 s2 freq +11991 path delay 767 ptp4l[174489.042]: master offset -4 s2 freq +11988 path delay 767 ptp4l[174490.040]: master offset -3 s2 freq +11988 path delay 765 ptp4l[174491.041]: master offset -2 s2 freq +11988 path delay 762 ptp4l[174492.041]: master offset 1 s2 freq +11990 path delay 758 ptp4l[174493.041]: master offset 1 s2 freq +11991 path delay 756 ptp4l[174494.041]: master offset 0 s2 freq +11990 path delay 756 ptp4l[174495.041]: master offset -2 s2 freq +11988 path delay 756 ptp4l[174496.041]: master offset -5 s2 freq +11984 path delay 758 ......................... ptp4l[175290.088]: master offset -29 s2 freq +11935 path delay 753 ptp4l[175291.088]: master offset -77 s2 freq +11879 path delay 751 ptp4l[175292.089]: master offset 9 s2 freq +11942 path delay 751 ptp4l[175293.088]: master offset 40 s2 freq +11975 path delay 751 ptp4l[175294.088]: master offset 31 s2 freq +11978 path delay 751 ptp4l[175295.089]: master offset -70368744177643 s2 freq -599999999 path delay 753 ptp4l[175296.088]: master offset -70368145816436 s2 freq -599999999 path delay 4701 ptp4l[175297.088]: master offset -70367546002731 s2 freq -599999999 path delay 8628 ptp4l[175298.088]: master offset -70366946185845 s2 freq -599999999 path delay 12574 ptp4l[175299.088]: master offset -70366346373840 s2 freq -599999999 path delay 16521 ptp4l[175300.090]: master offset -70365746557429 s2 freq -599999999 path delay 20782 ptp4l[175301.089]: master offset -70365146747351 s2 freq -599999999 path delay 24696 ptp4l[175302.089]: master offset -70364546918013 s2 freq -599999999 path delay 28662 [root@marctaSBC linuxptp]# ./phc2sys -s /dev/ptp5 -c /dev/ptp4 -m -O 0 phc2sys[174550.979]: failed to read out the clock frequency adjustment: Operation not supported phc2sys[174551.981]: phc offset 141058 s0 freq +0 delay 10880 phc2sys[174552.981]: phc offset 143298 s1 freq +2238 delay 10808 phc2sys[174553.982]: phc offset -4053 s2 freq -1815 delay 10799 phc2sys[174554.983]: phc offset 49 s2 freq +1072 delay 10800 phc2sys[174555.983]: phc offset 1344 s2 freq +2381 delay 10919 phc2sys[174556.984]: phc offset 1057 s2 freq +2497 delay 10535 phc2sys[174557.985]: phc offset 1176 s2 freq +2934 delay 10839 phc2sys[174558.985]: phc offset 560 s2 freq +2670 delay 10919 phc2sys[174559.986]: phc offset 82 s2 freq +2360 delay 10919 phc2sys[174560.987]: phc offset 109 s2 freq +2412 delay 10919 phc2sys[174561.987]: phc offset -14 s2 freq +2322 delay 10959 phc2sys[174562.988]: phc offset 70 s2 freq +2401 delay 10879 phc2sys[174563.989]: phc offset -29 s2 freq +2323 delay 10799 phc2sys[174564.989]: phc offset -110 s2 freq +2234 delay 10919 phc2sys[174565.989]: phc offset 31 s2 freq +2342 delay 10879 phc2sys[174566.989]: phc offset -171 s2 freq +2149 delay 10415 phc2sys[174567.990]: phc offset 199 s2 freq +2468 delay 10879 phc2sys[174568.990]: phc offset 114 s2 freq +2442 delay 10919 phc2sys[174569.990]: phc offset -13 s2 freq +2350 delay 10879 phc2sys[174570.991]: phc offset -5 s2 freq +2354 delay 10839 phc2sys[174571.991]: phc offset 64 s2 freq +2421 delay 10839 ......................... phc2sys[175291.197]: phc offset 35 s2 freq +2327 delay 10839 phc2sys[175292.197]: phc offset 5 s2 freq +2308 delay 10879 phc2sys[175293.197]: phc offset -2 s2 freq +2302 delay 10959 phc2sys[175294.197]: phc offset -10 s2 freq +2294 delay 10879 phc2sys[175295.198]: phc offset -65055841 s2 freq -65053540 delay 10879 phc2sys[175296.198]: phc offset -599947449 s2 freq -599999999 delay 11630 phc2sys[175297.198]: phc offset -599989526 s2 freq -599999999 delay 17471 phc2sys[175298.199]: phc offset -600004949 s2 freq -599999999 delay 17343 phc2sys[175299.199]: phc offset -600020181 s2 freq -599999999 delay 17471 phc2sys[175300.199]: phc offset -600035608 s2 freq -599999999 delay 17471 phc2sys[175301.199]: phc offset -600051031 s2 freq -599999999 delay 17727 [root@marctaSBC linuxptp]# ptp4l -i p36p1 -m -P ptp4l[174595.636]: selected /dev/ptp4 as PTP clock ptp4l[174595.637]: failed to read out the clock frequency adjustment: Operation not supported ptp4l[174595.644]: port 1: INITIALIZING to LISTENING on INITIALIZE ptp4l[174595.646]: port 0: INITIALIZING to LISTENING on INITIALIZE ptp4l[174601.644]: port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES [root@marctaSBC linuxptp]# ./phc2sys -s p37p1 -O 0 -m phc2sys[174620.093]: phc offset 113271 s0 freq +0 delay 5474 phc2sys[174621.094]: phc offset 113264 s1 freq -38072 delay 5484 phc2sys[174622.094]: phc offset -28 s2 freq -38100 delay 5494 phc2sys[174623.095]: phc offset 40 s2 freq -38040 delay 5484 phc2sys[174624.095]: phc offset -75 s2 freq -38143 delay 5494 phc2sys[174625.095]: phc offset 35 s2 freq -38056 delay 5554 phc2sys[174626.095]: phc offset -20 s2 freq -38100 delay 5464 phc2sys[174627.095]: phc offset 134 s2 freq -37952 delay 5494 phc2sys[174628.096]: phc offset 49 s2 freq -37997 delay 5504 phc2sys[174629.096]: phc offset -113 s2 freq -38144 delay 5504 phc2sys[174630.096]: phc offset -13 s2 freq -38078 delay 5504 phc2sys[174631.096]: phc offset 8 s2 freq -38061 delay 5494 phc2sys[174632.097]: phc offset 0 s2 freq -38067 delay 5545 phc2sys[174633.097]: phc offset 9 s2 freq -38058 delay 5504 phc2sys[174634.097]: phc offset -42 s2 freq -38106 delay 5514 phc2sys[174635.097]: phc offset -89 s2 freq -38166 delay 5474 phc2sys[174636.098]: phc offset 24 s2 freq -38079 delay 5529 phc2sys[174637.098]: phc offset 19 s2 freq -38077 delay 5464 phc2sys[174638.098]: phc offset 47 s2 freq -38044 delay 5494 phc2sys[174639.098]: phc offset 16 s2 freq -38060 delay 5484 phc2sys[174640.098]: phc offset -25 s2 freq -38097 delay 5484 phc2sys[174641.099]: phc offset 80 s2 freq -37999 delay 5564 phc2sys[174642.099]: phc offset -4 s2 freq -38059 delay 5559 phc2sys[174643.099]: phc offset -47 s2 freq -38103 delay 5484 .......................... phc2sys[175291.248]: phc offset 14 s2 freq -38014 delay 5494 phc2sys[175292.248]: phc offset -138 s2 freq -38162 delay 5479 phc2sys[175293.248]: phc offset -14 s2 freq -38079 delay 5534 phc2sys[175294.249]: phc offset 45 s2 freq -38025 delay 5484 phc2sys[175295.249]: phc offset -95729399 s2 freq -500000 delay 5484 phc2sys[175296.249]: phc offset -695151138 s2 freq -500000 delay 5496 phc2sys[175297.249]: phc offset -1294563525 s2 freq -500000 delay 5497 phc2sys[175298.249]: phc offset -1893973598 s2 freq -500000 delay 5466 phc2sys[175299.250]: phc offset -2493408464 s2 freq -500000 delay 5466 phc2sys[175300.250]: phc offset -3092833604 s2 freq -500000 delay 5507 phc2sys[175301.250]: phc offset -3692242108 s2 freq -500000 delay 5511 BOARD 3 : [root@marctaSBC ~]# ptp4l -i p36p1 -m -P -s ptp4l[23024.269]: selected /dev/ptp3 as PTP clock ptp4l[23024.271]: failed to read out the clock frequency adjustment: Operation not supported ptp4l[23024.277]: port 1: INITIALIZING to LISTENING on INITIALIZE ptp4l[23024.277]: port 0: INITIALIZING to LISTENING on INITIALIZE ptp4l[23031.766]: port 1: new foreign master 002246.fffe.1283c7-1 ptp4l[23035.766]: selected best master clock 002246.fffe.1283c7 ptp4l[23035.767]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE ptp4l[23036.768]: master offset 93070 s0 freq +0 path delay 751 ptp4l[23037.769]: master offset 62384 s1 freq -30686 path delay 751 ptp4l[23038.768]: master offset -5272 s2 freq -35958 path delay 754 ptp4l[23038.768]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED ptp4l[23039.768]: master offset -35 s2 freq -32303 path delay 754 ptp4l[23040.769]: master offset 1422 s2 freq -30856 path delay 756 ptp4l[23041.768]: master offset 1486 s2 freq -30366 path delay 757 ptp4l[23042.768]: master offset 1110 s2 freq -30296 path delay 759 ptp4l[23043.768]: master offset 600 s2 freq -30473 path delay 759 ptp4l[23044.768]: master offset 340 s2 freq -30553 path delay 761 ptp4l[23045.769]: master offset 85 s2 freq -30706 path delay 764 ptp4l[23046.768]: master offset -159 s2 freq -30925 path delay 764 ptp4l[23047.768]: master offset 61 s2 freq -30752 path delay 762 ptp4l[23048.768]: master offset 62 s2 freq -30733 path delay 762 ptp4l[23049.768]: master offset 28 s2 freq -30748 path delay 762 ptp4l[23050.768]: master offset 40 s2 freq -30728 path delay 758 ptp4l[23051.768]: master offset 5 s2 freq -30751 path delay 758 ptp4l[23052.768]: master offset -87 s2 freq -30841 path delay 758 ptp4l[23053.769]: master offset -21 s2 freq -30802 path delay 758 ptp4l[23054.768]: master offset -53 s2 freq -30840 path delay 758 ptp4l[23055.768]: master offset 67 s2 freq -30736 path delay 760 ptp4l[23056.769]: master offset 131 s2 freq -30652 path delay 758 ptp4l[23057.768]: master offset -66 s2 freq -30809 path delay 760 ptp4l[23058.768]: master offset -37 s2 freq -30800 path delay 758 ptp4l[23059.768]: master offset -110 s2 freq -30884 path delay 758 ptp4l[23060.768]: master offset -50 s2 freq -30857 path delay 760 ................................. ptp4l[23719.791]: master offset -93 s2 freq -30892 path delay 768 ptp4l[23720.793]: master offset -30 s2 freq -30856 path delay 766 ptp4l[23721.792]: master offset -9 s2 freq -30844 path delay 766 ptp4l[23722.792]: master offset 13 s2 freq -30825 path delay 766 ptp4l[23723.791]: master offset -30626099 s2 freq -30656933 path delay 766 ptp4l[23724.791]: master offset -316853487 s2 freq -326072151 path delay 479 ptp4l[23725.792]: master offset -591085085 s2 freq -599999999 path delay -1810 ptp4l[23726.791]: master offset -591811507 s2 freq -599999999 path delay -1768 ptp4l[23727.792]: master offset -591864410 s2 freq -599999999 path delay -1724 ptp4l[23728.793]: master offset -591917378 s2 freq -599999999 path delay -1679 ptp4l[23729.792]: master offset -591970343 s2 freq -599999999 path delay -1637 ptp4l[23730.792]: master offset -592023184 s2 freq -599999999 path delay -1591 ptp4l[23731.792]: master offset -592076091 s2 freq -599999999 path delay -1543 [root@marctaSBC linuxptp]# ./phc2sys -s p36p1 -O 0 -m phc2sys[23107.224]: phc offset 305036 s0 freq +0 delay 6383 phc2sys[23108.225]: phc offset 305003 s1 freq -48438 delay 6293 phc2sys[23109.225]: phc offset 79 s2 freq -48359 delay 6365 phc2sys[23110.226]: phc offset 216 s2 freq -48198 delay 6323 phc2sys[23111.226]: phc offset 146 s2 freq -48203 delay 6357 phc2sys[23112.226]: phc offset 93 s2 freq -48213 delay 6353 phc2sys[23113.226]: phc offset -26 s2 freq -48304 delay 6293 phc2sys[23114.227]: phc offset -62 s2 freq -48348 delay 6405 phc2sys[23115.227]: phc offset -148 s2 freq -48452 delay 6357 phc2sys[23116.227]: phc offset -114 s2 freq -48463 delay 6293 phc2sys[23117.227]: phc offset -29 s2 freq -48412 delay 6293 phc2sys[23118.227]: phc offset 82 s2 freq -48309 delay 6353 phc2sys[23119.228]: phc offset 167 s2 freq -48200 delay 6356 phc2sys[23120.228]: phc offset 85 s2 freq -48232 delay 6353 phc2sys[23121.228]: phc offset 82 s2 freq -48209 delay 6293 phc2sys[23122.228]: phc offset 25 s2 freq -48242 delay 6353 phc2sys[23123.228]: phc offset -35 s2 freq -48294 delay 6353 phc2sys[23124.229]: phc offset 33 s2 freq -48237 delay 6372 phc2sys[23125.229]: phc offset -88 s2 freq -48348 delay 6357 phc2sys[23126.229]: phc offset -98 s2 freq -48384 delay 6406 phc2sys[23127.229]: phc offset -33 s2 freq -48349 delay 6353 phc2sys[23128.230]: phc offset -18 s2 freq -48343 delay 6292 phc2sys[23129.230]: phc offset 50 s2 freq -48281 delay 6353 phc2sys[23130.230]: phc offset -8 s2 freq -48324 delay 6417 phc2sys[23131.230]: phc offset -26 s2 freq -48344 delay 6537 phc2sys[23132.230]: phc offset 22 s2 freq -48304 delay 6356 phc2sys[23133.231]: phc offset -5 s2 freq -48324 delay 6447 phc2sys[23134.231]: phc offset 21 s2 freq -48300 delay 6413 phc2sys[23135.231]: phc offset 188 s2 freq -48127 delay 6356 phc2sys[23136.231]: phc offset 82 s2 freq -48176 delay 6353 phc2sys[23137.232]: phc offset -88 s2 freq -48322 delay 6293 phc2sys[23138.232]: phc offset -259 s2 freq -48519 delay 6383 phc2sys[23139.232]: phc offset -51 s2 freq -48389 delay 6361 phc2sys[23140.232]: phc offset 169 s2 freq -48184 delay 6353 .............................. phc2sys[23719.363]: phc offset -14 s2 freq -48237 delay 6293 phc2sys[23720.363]: phc offset -95 s2 freq -48322 delay 6342 phc2sys[23721.363]: phc offset 14 s2 freq -48242 delay 6357 phc2sys[23722.363]: phc offset -2 s2 freq -48253 delay 6357 phc2sys[23723.364]: phc offset -11 s2 freq -48263 delay 6293 phc2sys[23724.364]: phc offset -17523658 s2 freq -500000 delay 6293 phc2sys[23725.364]: phc offset -216745503 s2 freq -500000 delay 6296 phc2sys[23726.364]: phc offset -698745647 s2 freq -500000 delay 6360 phc2sys[23727.364]: phc offset -1298102527 s2 freq -500000 delay 6359 phc2sys[23728.365]: phc offset -1897535882 s2 freq -500000 delay 6344 phc2sys[23729.365]: phc offset -2496933568 s2 freq -500000 delay 6420 phc2sys[23730.365]: phc offset -3096290190 s2 freq -500000 delay 6296 phc2sys[23731.365]: phc offset -3695643149 s2 freq -500000 delay 6356 |
