Thread: [Linuxptp-users] [issue] phc2sys results large jitter with multiple '-c'
PTP IEEE 1588 stack for Linux
Brought to you by:
rcochran
Menu
▾
▴
linuxptp-users
|
From: egg c. <egg...@gm...> - 2023-06-12 04:24:38
|
Hi, Just upgraded to v4.0, and trying the new feature of 'phc2sys' with multiple '-c'. It seems not working very well on my test machine, with large offset jitters. Here's my configuration: 1) ptp0~ptp3 are 4 ports on a 4 port i350, ptp4~ptp9 are separated i210 ports. 2) run ts2phc on ptp3(enp1s0f3), to sync ptp3's phc with external 1pps 3) run phc2sys to sync other phcs with ptp3: phc2sys -s /dev/ptp3 -c /dev/ptp2 -c /dev/ptp1 -c /dev/ptp0 -O 0 -f /etc/linuxptp/phc2sys.conf -l 7 -N 10 phc2sys[496846.875]: /dev/ptp0 phc offset -5 s2 freq +117755 delay 10782 phc2sys[496846.875]: /dev/ptp1 phc offset -950 s2 freq +117226 delay 12527 phc2sys[496846.876]: /dev/ptp2 phc offset -139 s2 freq +118014 delay 13198 phc2sys[496847.876]: /dev/ptp0 phc offset -6 s2 freq +117753 delay 10751 phc2sys[496847.877]: /dev/ptp1 phc offset -244 s2 freq +117647 delay 11983 phc2sys[496847.877]: /dev/ptp2 phc offset -630 s2 freq +117481 delay 12094 phc2sys[496848.877]: /dev/ptp0 phc offset -20 s2 freq +117737 delay 10079 phc2sys[496848.878]: /dev/ptp1 phc offset 135 s2 freq +117953 delay 12015 phc2sys[496848.878]: /dev/ptp2 phc offset 220 s2 freq +118142 delay 12671 phc2sys[496849.878]: /dev/ptp0 phc offset 22 s2 freq +117773 delay 10767 phc2sys[496849.879]: /dev/ptp1 phc offset -607 s2 freq +117251 delay 12815 phc2sys[496849.879]: /dev/ptp2 phc offset -520 s2 freq +117468 delay 13406 phc2sys[496850.880]: /dev/ptp0 phc offset -13 s2 freq +117745 delay 10735 phc2sys[496850.880]: /dev/ptp1 phc offset -95 s2 freq +117581 delay 12382 phc2sys[496850.880]: /dev/ptp2 phc offset -337 s2 freq +117495 delay 12623 phc2sys[496851.881]: /dev/ptp0 phc offset -11 s2 freq +117743 delay 10782 phc2sys[496851.881]: /dev/ptp1 phc offset -178 s2 freq +117470 delay 12927 phc2sys[496851.881]: /dev/ptp2 phc offset -13 s2 freq +117718 delay 12399 phc2sys[496852.882]: /dev/ptp0 phc offset 9 s2 freq +117760 delay 10094 phc2sys[496852.882]: /dev/ptp1 phc offset 618 s2 freq +118212 delay 12255 phc2sys[496852.882]: /dev/ptp2 phc offset -280 s2 freq +117447 delay 12142 phc2sys[496853.883]: /dev/ptp0 phc offset 11 s2 freq +117764 delay 10751 phc2sys[496853.883]: /dev/ptp1 phc offset 216 s2 freq +117996 delay 11758 phc2sys[496853.884]: /dev/ptp2 phc offset 147 s2 freq +117790 delay 13919 phc2sys[496854.884]: /dev/ptp0 phc offset -2 s2 freq +117755 delay 10751 phc2sys[496854.884]: /dev/ptp1 phc offset -437 s2 freq +117408 delay 12350 phc2sys[496854.885]: /dev/ptp2 phc offset 749 s2 freq +118436 delay 13054 phc2sys[496855.885]: /dev/ptp0 phc offset -25 s2 freq +117731 delay 10783 phc2sys[496855.885]: /dev/ptp1 phc offset 711 s2 freq +118425 delay 12622 phc2sys[496855.886]: /dev/ptp2 phc offset -76 s2 freq +117836 delay 12526 phc2sys[496856.886]: /dev/ptp0 phc offset 21 s2 freq +117769 delay 10767 phc2sys[496856.887]: /dev/ptp1 phc offset -641 s2 freq +117286 delay 12542 phc2sys[496856.887]: /dev/ptp2 phc offset -400 s2 freq +117489 delay 12878 phc2sys[496857.887]: /dev/ptp0 phc offset -21 s2 freq +117734 delay 10942 phc2sys[496857.888]: /dev/ptp1 phc offset 81 s2 freq +117816 delay 11678 phc2sys[496857.888]: /dev/ptp2 phc offset -16 s2 freq +117753 delay 12446 phc2sys[496858.889]: /dev/ptp0 phc offset 10 s2 freq +117758 delay 10783 phc2sys[496858.889]: /dev/ptp1 phc offset -106 s2 freq +117653 delay 11951 phc2sys[496858.889]: /dev/ptp2 phc offset -269 s2 freq +117495 delay 13231 phc2sys[496859.890]: /dev/ptp0 phc offset -3 s2 freq +117748 delay 10767 phc2sys[496859.890]: /dev/ptp1 phc offset 75 s2 freq +117802 delay 12750 phc2sys[496859.890]: /dev/ptp2 phc offset 446 s2 freq +118129 delay 12798 phc2sys[496860.891]: /dev/ptp0 phc offset 3 s2 freq +117754 delay 10063 phc2sys[496860.891]: /dev/ptp1 phc offset 235 s2 freq +117985 delay 11407 phc2sys[496860.891]: /dev/ptp2 phc offset -362 s2 freq +117455 delay 12159 phc2sys[496861.892]: /dev/ptp0 phc offset 4 s2 freq +117755 delay 10767 phc2sys[496861.892]: /dev/ptp1 phc offset -371 s2 freq +117449 delay 12799 phc2sys[496861.893]: /dev/ptp2 phc offset -85 s2 freq +117624 delay 12527 phc2sys[496862.893]: /dev/ptp0 phc offset -4 s2 freq +117749 delay 10783 phc2sys[496862.893]: /dev/ptp1 phc offset -352 s2 freq +117357 delay 12798 phc2sys[496862.894]: /dev/ptp2 phc offset 135 s2 freq +117818 delay 12958 phc2sys[496863.894]: /dev/ptp0 phc offset 26 s2 freq +117777 delay 10767 phc2sys[496863.895]: /dev/ptp1 phc offset 447 s2 freq +118050 delay 11966 phc2sys[496863.895]: /dev/ptp2 phc offset -345 s2 freq +117379 delay 13854 phc2sys[496864.895]: /dev/ptp0 phc offset -5 s2 freq +117754 delay 10782 phc2sys[496864.896]: /dev/ptp1 phc offset 207 s2 freq +117944 delay 12494 phc2sys[496864.896]: /dev/ptp2 phc offset 480 s2 freq +118100 delay 13694 phc2sys[496865.896]: /dev/ptp0 phc offset -21 s2 freq +117737 delay 10751 phc2sys[496865.897]: /dev/ptp1 phc offset -447 s2 freq +117352 delay 13102 phc2sys[496865.897]: /dev/ptp2 phc offset 413 s2 freq +118177 delay 13039 phc2sys[496866.898]: /dev/ptp0 phc offset 15 s2 freq +117766 delay 10782 phc2sys[496866.898]: /dev/ptp1 phc offset 254 s2 freq +117919 delay 12414 phc2sys[496866.898]: /dev/ptp2 phc offset -152 s2 freq +117736 delay 12591 phc2sys[496867.899]: /dev/ptp0 phc offset 16 s2 freq +117772 delay 10767 phc2sys[496867.899]: /dev/ptp1 phc offset 93 s2 freq +117834 delay 11775 phc2sys[496867.899]: /dev/ptp2 phc offset -360 s2 freq +117482 delay 12862 phc2sys[496868.900]: /dev/ptp0 phc offset -48 s2 freq +117713 delay 10415 phc2sys[496868.900]: /dev/ptp1 phc offset 71 s2 freq +117840 delay 11247 phc2sys[496868.900]: /dev/ptp2 phc offset -102 s2 freq +117632 delay 11070 phc2sys[496869.901]: /dev/ptp0 phc offset 28 s2 freq +117774 delay 10799 phc2sys[496869.901]: /dev/ptp1 phc offset 123 s2 freq +117914 delay 12639 phc2sys[496869.901]: /dev/ptp2 phc offset -1 s2 freq +117703 delay 12782 phc2sys[496870.902]: /dev/ptp0 phc offset -2 s2 freq +117753 delay 10095 phc2sys[496870.902]: /dev/ptp1 phc offset 161 s2 freq +117989 delay 11775 phc2sys[496870.902]: /dev/ptp2 phc offset 96 s2 freq +117800 delay 12286 phc2sys[496871.903]: /dev/ptp0 phc offset 16 s2 freq +117770 delay 10767 phc2sys[496871.903]: /dev/ptp1 phc offset -305 s2 freq +117571 delay 12303 phc2sys[496871.903]: /dev/ptp2 phc offset -189 s2 freq +117543 delay 13023 phc2sys[496872.904]: /dev/ptp0 phc offset -5 s2 freq +117754 delay 10782 phc2sys[496872.904]: /dev/ptp1 phc offset -191 s2 freq +117593 delay 11807 phc2sys[496872.904]: /dev/ptp2 phc offset 352 s2 freq +118028 delay 11950 phc2sys[496873.905]: /dev/ptp0 phc offset 31 s2 freq +117788 delay 10751 phc2sys[496873.905]: /dev/ptp1 phc offset -40 s2 freq +117687 delay 12574 phc2sys[496873.906]: /dev/ptp2 phc offset 99 s2 freq +117880 delay 13359 phc2sys[496874.906]: /dev/ptp0 phc offset -8 s2 freq +117759 delay 10095 phc2sys[496874.906]: /dev/ptp1 phc offset -154 s2 freq +117561 delay 11726 phc2sys[496874.907]: /dev/ptp2 phc offset 81 s2 freq +117892 delay 12158 phc2sys[496875.907]: /dev/ptp0 phc offset -23 s2 freq +117741 delay 10766 phc2sys[496875.907]: /dev/ptp1 phc offset 300 s2 freq +117969 delay 12111 phc2sys[496875.908]: /dev/ptp2 phc offset 147 s2 freq +117982 delay 12527 phc2sys[496876.908]: /dev/ptp0 phc offset -18 s2 freq +117739 delay 10782 phc2sys[496876.908]: /dev/ptp1 phc offset 134 s2 freq +117893 delay 11951 phc2sys[496876.909]: /dev/ptp2 phc offset -568 s2 freq +117311 delay 11807 phc2sys[496877.909]: /dev/ptp0 phc offset 3 s2 freq +117755 delay 10767 phc2sys[496877.909]: /dev/ptp1 phc offset -270 s2 freq +117529 delay 12222 phc2sys[496877.910]: /dev/ptp2 phc offset -206 s2 freq +117503 delay 12639 Notice that the offset of ptp1 and ptp2 are not stable, with jitter of hundreds of nanoseconds, the last one in the command line which is polled first every time, eg ptp0 here, is stable. I have done a lot of tests, it's only related to the position the dest phc appended to the dest list. 4) running multiple phc2sys instance, each instance with only one sink, it works well as expected. phc2sys -s /dev/ptp3 -c /dev/ptp2 -O 0 -f /etc/linuxptp/phc2sys.conf -l 7 -N 10 phc2sys[497690.108]: /dev/ptp2 phc offset 2 s2 freq +117754 delay 10750 phc2sys[497691.108]: /dev/ptp2 phc offset 1 s2 freq +117754 delay 10750 phc2sys[497692.109]: /dev/ptp2 phc offset -7 s2 freq +117746 delay 10927 phc2sys[497693.109]: /dev/ptp2 phc offset 26 s2 freq +117777 delay 10751 phc2sys[497694.110]: /dev/ptp2 phc offset -12 s2 freq +117747 delay 10750 phc2sys[497695.110]: /dev/ptp2 phc offset -9 s2 freq +117746 delay 10750 phc2sys[497696.111]: /dev/ptp2 phc offset 2 s2 freq +117755 delay 10735 phc2sys[497697.111]: /dev/ptp2 phc offset 1 s2 freq +117754 delay 10063 phc2sys[497698.112]: /dev/ptp2 phc offset -3 s2 freq +117750 delay 10766 phc2sys[497699.112]: /dev/ptp2 phc offset -20 s2 freq +117733 delay 10767 phc2sys[497700.113]: /dev/ptp2 phc offset 38 s2 freq +117785 delay 10415 phc2sys[497701.113]: /dev/ptp2 phc offset -11 s2 freq +117747 delay 10751 phc2sys[497702.114]: /dev/ptp2 phc offset 26 s2 freq +117781 delay 10751 phc2sys[497703.114]: /dev/ptp2 phc offset -53 s2 freq +117709 delay 10079 phc2sys[497704.114]: /dev/ptp2 phc offset 5 s2 freq +117752 delay 10751 phc2sys[497705.115]: /dev/ptp2 phc offset 4 s2 freq +117752 delay 10751 phc2sys[497706.115]: /dev/ptp2 phc offset 2 s2 freq +117751 delay 10750 phc2sys[497707.116]: /dev/ptp2 phc offset 0 s2 freq +117750 delay 10750 phc2sys[497708.116]: /dev/ptp2 phc offset -8 s2 freq +117742 delay 10063 phc2sys[497709.117]: /dev/ptp2 phc offset 24 s2 freq +117771 delay 10750 phc2sys[497710.117]: /dev/ptp2 phc offset -21 s2 freq +117734 delay 10767 phc2sys[497711.118]: /dev/ptp2 phc offset 15 s2 freq +117763 delay 10095 phc2sys[497712.118]: /dev/ptp2 phc offset -1 s2 freq +117752 delay 10751 phc2sys[497713.119]: /dev/ptp2 phc offset 3 s2 freq +117756 delay 10750 phc2sys[497714.119]: /dev/ptp2 phc offset 18 s2 freq +117771 delay 10751 phc2sys[497715.120]: /dev/ptp2 phc offset 1 s2 freq +117760 delay 10719 phc2sys[497716.120]: /dev/ptp2 phc offset -8 s2 freq +117751 delay 10094 phc2sys[497717.121]: /dev/ptp2 phc offset -12 s2 freq +117745 delay 10750 phc2sys[497718.121]: /dev/ptp2 phc offset -7 s2 freq +117746 delay 10750 phc2sys[497719.122]: /dev/ptp2 phc offset -17 s2 freq +117734 delay 10079 phc2sys[497720.122]: /dev/ptp2 phc offset 11 s2 freq +117757 delay 10767 phc2sys[497721.123]: /dev/ptp2 phc offset 17 s2 freq +117766 delay 10751 phc2sys[497722.123]: /dev/ptp2 phc offset -11 s2 freq +117743 delay 10271 phc2sys[497723.124]: /dev/ptp2 phc offset 44 s2 freq +117795 delay 10750 phc2sys[497724.124]: /dev/ptp2 phc offset -10 s2 freq +117754 delay 10735 phc2sys[497725.125]: /dev/ptp2 phc offset -40 s2 freq +117721 delay 10750 phc2sys[497726.125]: /dev/ptp2 phc offset -7 s2 freq +117742 delay 10751 phc2sys[497727.126]: /dev/ptp2 phc offset 1 s2 freq +117748 delay 10095 phc2sys[497728.126]: /dev/ptp2 phc offset 29 s2 freq +117776 delay 10750 phc2sys -s /dev/ptp3 -c /dev/ptp1 -O 0 -f /etc/linuxptp/phc2sys.conf -l 7 -N 10 phc2sys[497690.656]: /dev/ptp1 phc offset -11 s2 freq +117745 delay 10767 phc2sys[497691.657]: /dev/ptp1 phc offset 7 s2 freq +117760 delay 10782 phc2sys[497692.657]: /dev/ptp1 phc offset 1 s2 freq +117756 delay 10783 phc2sys[497693.658]: /dev/ptp1 phc offset -16 s2 freq +117739 delay 10783 phc2sys[497694.658]: /dev/ptp1 phc offset 23 s2 freq +117774 delay 10798 phc2sys[497695.659]: /dev/ptp1 phc offset -28 s2 freq +117729 delay 10111 phc2sys[497696.659]: /dev/ptp1 phc offset 20 s2 freq +117769 delay 10798 phc2sys[497697.660]: /dev/ptp1 phc offset -12 s2 freq +117743 delay 10798 phc2sys[497698.660]: /dev/ptp1 phc offset 7 s2 freq +117758 delay 10287 phc2sys[497699.661]: /dev/ptp1 phc offset -5 s2 freq +117749 delay 10782 phc2sys[497700.661]: /dev/ptp1 phc offset 0 s2 freq +117752 delay 10799 phc2sys[497701.661]: /dev/ptp1 phc offset 4 s2 freq +117756 delay 10223 phc2sys[497702.662]: /dev/ptp1 phc offset -4 s2 freq +117749 delay 10767 phc2sys[497703.662]: /dev/ptp1 phc offset 7 s2 freq +117759 delay 10783 phc2sys[497704.663]: /dev/ptp1 phc offset 9 s2 freq +117763 delay 10126 phc2sys[497705.663]: /dev/ptp1 phc offset -31 s2 freq +117726 delay 10783 phc2sys[497706.664]: /dev/ptp1 phc offset 2 s2 freq +117750 delay 10799 phc2sys[497707.664]: /dev/ptp1 phc offset 0 s2 freq +117748 delay 10767 phc2sys[497708.665]: /dev/ptp1 phc offset 4 s2 freq +117752 delay 10799 phc2sys[497709.665]: /dev/ptp1 phc offset 8 s2 freq +117757 delay 10783 phc2sys[497710.666]: /dev/ptp1 phc offset -4 s2 freq +117748 delay 10767 phc2sys[497711.666]: /dev/ptp1 phc offset -4 s2 freq +117747 delay 10782 phc2sys[497712.667]: /dev/ptp1 phc offset 29 s2 freq +117778 delay 10127 phc2sys[497713.667]: /dev/ptp1 phc offset -10 s2 freq +117748 delay 10798 phc2sys[497714.668]: /dev/ptp1 phc offset -20 s2 freq +117735 delay 10095 phc2sys[497715.668]: /dev/ptp1 phc offset 8 s2 freq +117757 delay 10783 phc2sys[497716.669]: /dev/ptp1 phc offset 27 s2 freq +117778 delay 10783 phc2sys[497717.669]: /dev/ptp1 phc offset -9 s2 freq +117751 delay 10127 phc2sys[497718.670]: /dev/ptp1 phc offset 0 s2 freq +117757 delay 10783 phc2sys[497719.670]: /dev/ptp1 phc offset -12 s2 freq +117745 delay 10767 phc2sys[497720.671]: /dev/ptp1 phc offset -2 s2 freq +117751 delay 10095 phc2sys[497721.671]: /dev/ptp1 phc offset 9 s2 freq +117762 delay 10783 phc2sys[497722.671]: /dev/ptp1 phc offset 1 s2 freq +117756 delay 10782 phc2sys[497723.672]: /dev/ptp1 phc offset -16 s2 freq +117740 delay 10798 phc2sys[497724.672]: /dev/ptp1 phc offset -13 s2 freq +117738 delay 10815 phc2sys[497725.673]: /dev/ptp1 phc offset 13 s2 freq +117760 delay 10111 phc2sys[497726.673]: /dev/ptp1 phc offset 5 s2 freq +117756 delay 10766 phc2sys[497727.674]: /dev/ptp1 phc offset 13 s2 freq +117765 delay 10783 phc2sys[497728.674]: /dev/ptp1 phc offset 4 s2 freq +117760 delay 10127 5) not only the 4 ports on the same nic, any combination of ptp0~ptp9 results the same. Have gone through the phc2sys codes, find nothing reasonable that explains this issue so far. I can see when the large jitter happens, the measured delay also varies a lot, perhapse it's related to the pcie tranfication process? |
|
From: Miroslav L. <mli...@re...> - 2023-06-12 06:53:59
|
On Mon, Jun 12, 2023 at 12:24:19PM +0800, egg car wrote: > 5) not only the 4 ports on the same nic, any combination of ptp0~ptp9 results > the same. > > > Have gone through the phc2sys codes, find nothing reasonable that explains > this issue so far. > > I can see when the large jitter happens, the measured delay also varies a lot, > perhapse it's related to the pcie tranfication process? Yes, it's related to PCIe delays. The problem is that the kernel doesn't provide an ioctl to measure offset between two PHCs, so phc2sys has to do it in user space using clock_gettime(), which doesn't work very well (large delay, jitter and asymmetry). For PHC vs system clock measurements there is an optimized path in the kernel which for most drivers limits the delay to a single PCIe read. If you need to synchronize multiple PHCs to each other, it's better to use one phc2sys instance to synchronize the system clock to the source PHC and then another phc2sys instance to synchronize the rest of PHCs to the system clock. -- Miroslav Lichvar |
|
From: egg c. <egg...@gm...> - 2023-06-12 08:06:37
|
Dear Miro, > Yes, it's related to PCIe delays. The problem is that the kernel > doesn't provide an ioctl to measure offset between two PHCs, so > phc2sys has to do it in user space using clock_gettime(), which > doesn't work very well (large delay, jitter and asymmetry). Yes it seems like the clock_gettime() api is not very accurate, the delay introduced by the switchover between userspace and kernel space may vary a lot. But it's confusing me that when I run several independent phc2sys instances at the same time, every one just looks fine, the offset value varies in the range of two-digit nanoseconds. Let me do more tests to locate the problem. Regards, Yuezhen Luan |
|
From: Nemo C. <nem...@gm...> - 2023-06-16 00:07:09
|
Hi LinuxPTPUsers, I have this weird problem sometimes with phc2syc. Please look at the log below, ptp4l service is perfectly okay. But the phc2sys is stuck (although it measures the offset correctly) and not adjusting the system time for some reason. Why would this happen? Waiting for more than 30 minutes didn't fix this problem. Several reboots fixes the issue. 1970-01-01T00:05:09.600718+00:00 phc2sys: [304.725] CLOCK_REALTIME phc offset -1686774497008453238 s2 freq -100000000 delay 13721970-01-01T00:05:09.628181+00:00 ptp4l: [304.752] master offset 198 s3 freq -35590 path delay 3231970-01-01T00:05:10.611304+00:00 phc2sys: [305.735] CLOCK_REALTIME phc offset -1686774496916583694 s2 freq -100000000 delay 13721970-01-01T00:05:10.728644+00:00 ptp4l: [305.853] master offset -63 s3 freq -35792 path delay 3231970-01-01T00:05:11.618971+00:00 phc2sys: [306.743] CLOCK_REALTIME phc offset -1686774496824974153 s2 freq -100000000 delay 13731970-01-01T00:05:11.829106+00:00 ptp4l: [306.953] master offset -121 s3 freq -35869 path delay 3231970-01-01T00:05:12.717043+00:00 phc2sys: [307.841] CLOCK_REALTIME phc offset -1686774496725151408 s2 freq -100000000 delay 14081970-01-01T00:05:12.929540+00:00 ptp4l: [308.054] master offset 111 s3 freq -35673 path delay 3231970-01-01T00:05:13.721458+00:00 phc2sys: [308.846] CLOCK_REALTIME phc offset -1686774496633835940 s2 freq -100000000 delay 13731970-01-01T00:05:14.030032+00:00 ptp4l: [309.154] master offset -112 s3 freq -35863 path delay 323 Thanks, Nemo |
|
From: Ed B. <br...@ar...> - 2023-06-16 13:29:40
Attachments:
smime.p7s
|
You need either of: 1) a battery backup for the real-time clock so that it starts at a reasonable time. 2) Use -w option to phc2sys so that it waits until ptp4l is synchronized. 3) Use the -S option to phc2sys so that it steps the clock if the offset is above some threshold. By default, phc2sys will step the clock only once at startup, but here the clock has jumped from the epoch after phc2sys started. It would take a very long time (530 years in this case) to adjust for that without stepping the clock. - Ed On 6/15/23 19:06, Nemo Crypto wrote: > Hi LinuxPTPUsers, > > I have this weird problem sometimes with phc2syc. Please look at the log > below, ptp4l service is perfectly okay. But the phc2sys is stuck > (although it measures the offset correctly) and not adjusting the system > time for some reason. Why would this happen? Waiting for more than 30 > minutes didn't fix this problem. > > Several reboots fixes the issue. > > 1970-01-01T00:05:09.600718+00:00 phc2sys: [304.725] CLOCK_REALTIME phc offset -1686774497008453238 s2 freq -100000000 delay 1372 > > 1970-01-01T00:05:09.628181+00:00 ptp4l: [304.752] master offset 198 s3 freq -35590 path delay 323 > > 1970-01-01T00:05:10.611304+00:00 phc2sys: [305.735] CLOCK_REALTIME phc offset -1686774496916583694 s2 freq -100000000 delay 1372 > > 1970-01-01T00:05:10.728644+00:00 ptp4l: [305.853] master offset -63 s3 freq -35792 path delay 323 > > 1970-01-01T00:05:11.618971+00:00 phc2sys: [306.743] CLOCK_REALTIME phc offset -1686774496824974153 s2 freq -100000000 delay 1373 > > 1970-01-01T00:05:11.829106+00:00 ptp4l: [306.953] master offset -121 s3 freq -35869 path delay 323 > > 1970-01-01T00:05:12.717043+00:00 phc2sys: [307.841] CLOCK_REALTIME phc offset -1686774496725151408 s2 freq -100000000 delay 1408 > > 1970-01-01T00:05:12.929540+00:00 ptp4l: [308.054] master offset 111 s3 freq -35673 path delay 323 > > 1970-01-01T00:05:13.721458+00:00 phc2sys: [308.846] CLOCK_REALTIME phc offset -1686774496633835940 s2 freq -100000000 delay 1373 > > 1970-01-01T00:05:14.030032+00:00 ptp4l: [309.154] master offset -112 s3 freq -35863 path delay 323 > > > > Thanks, > Nemo > > > > > > > _______________________________________________ > Linuxptp-users mailing list > Lin...@li... > https://lists.sourceforge.net/lists/listinfo/linuxptp-users |
|
From: Nemo C. <nem...@gm...> - 2023-06-16 15:19:32
|
Thank you Ed!
I am using -w option but not -S. I now understand why the clock step is not happening. But with -w phc2sys should start only after ptp4l, isn't it?
If yes, then why would this happen - "the clock has jumped from the epoch after phc2sys started".
The only way is probably that "-w" option is not effective?
Thanks,
Nemo
On Friday, 16 June, 2023 at 09:33:23 am GMT-4, Ed Branch <br...@ar...> wrote:
You need either of:
1) a battery backup for the real-time clock so that it starts at a
reasonable time.
2) Use -w option to phc2sys so that it waits until ptp4l is synchronized.
3) Use the -S option to phc2sys so that it steps the clock if the offset
is above some threshold.
By default, phc2sys will step the clock only once at startup, but here
the clock has jumped from the epoch after phc2sys started. It would take
a very long time (530 years in this case) to adjust for that without
stepping the clock.
- Ed
On 6/15/23 19:06, Nemo Crypto wrote:
> Hi LinuxPTPUsers,
>
> I have this weird problem sometimes with phc2syc. Please look at the log
> below, ptp4l service is perfectly okay. But the phc2sys is stuck
> (although it measures the offset correctly) and not adjusting the system
> time for some reason. Why would this happen? Waiting for more than 30
> minutes didn't fix this problem.
>
> Several reboots fixes the issue.
>
> 1970-01-01T00:05:09.600718+00:00 phc2sys: [304.725] CLOCK_REALTIME phc offset -1686774497008453238 s2 freq -100000000 delay 1372
>
> 1970-01-01T00:05:09.628181+00:00 ptp4l: [304.752] master offset 198 s3 freq -35590 path delay 323
>
> 1970-01-01T00:05:10.611304+00:00 phc2sys: [305.735] CLOCK_REALTIME phc offset -1686774496916583694 s2 freq -100000000 delay 1372
>
> 1970-01-01T00:05:10.728644+00:00 ptp4l: [305.853] master offset -63 s3 freq -35792 path delay 323
>
> 1970-01-01T00:05:11.618971+00:00 phc2sys: [306.743] CLOCK_REALTIME phc offset -1686774496824974153 s2 freq -100000000 delay 1373
>
> 1970-01-01T00:05:11.829106+00:00 ptp4l: [306.953] master offset -121 s3 freq -35869 path delay 323
>
> 1970-01-01T00:05:12.717043+00:00 phc2sys: [307.841] CLOCK_REALTIME phc offset -1686774496725151408 s2 freq -100000000 delay 1408
>
> 1970-01-01T00:05:12.929540+00:00 ptp4l: [308.054] master offset 111 s3 freq -35673 path delay 323
>
> 1970-01-01T00:05:13.721458+00:00 phc2sys: [308.846] CLOCK_REALTIME phc offset -1686774496633835940 s2 freq -100000000 delay 1373
>
> 1970-01-01T00:05:14.030032+00:00 ptp4l: [309.154] master offset -112 s3 freq -35863 path delay 323
>
>
>
> Thanks,
> Nemo
>
>
>
>
>
>
> _______________________________________________
> Linuxptp-users mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linuxptp-users
_______________________________________________
Linuxptp-users mailing list
Lin...@li...
https://lists.sourceforge.net/lists/listinfo/linuxptp-users
|
|
From: Ed B. <br...@ar...> - 2023-06-16 19:05:11
Attachments:
smime.p7s
|
TLDR: It could also be the GM. I see this on embedded networks, ie. the whole network including the GM is on the same power switch, it all comes up together, and none of the RTC's have battery backup. Maybe the "GM" is just another embedded computer that gets time from GPS or something like that. So the "GM" starts serving time over PTP starting at the epoch, everything sync's to it, then the GM jumps when it gets GPS lock. Alternately, the GM just takes longer to boot for whatever reason. So some random host is selected as master, serving the epoch over PTP, and all the other hosts sync to that. Then, when the GM starts serving the real time, the other hosts switch to it. But again, phc2sys has already sync'd to the wrong time. - Ed On 6/16/23 10:19, Nemo Crypto wrote: > Thank you Ed! > I am using -w option but not -S. I now understand why the clock step is > not happening. But with -w phc2sys should start only after ptp4l, isn't it? > If yes, then why would this happen - "the clock has jumped from the > epoch after phc2sys started". > > The only way is probably that "-w" option is not effective? > > > Thanks, > Nemo > > > On Friday, 16 June, 2023 at 09:33:23 am GMT-4, Ed Branch > <br...@ar...> wrote: > > > You need either of: > > 1) a battery backup for the real-time clock so that it starts at a > reasonable time. > 2) Use -w option to phc2sys so that it waits until ptp4l is synchronized. > 3) Use the -S option to phc2sys so that it steps the clock if the offset > is above some threshold. > > By default, phc2sys will step the clock only once at startup, but here > the clock has jumped from the epoch after phc2sys started. It would take > a very long time (530 years in this case) to adjust for that without > stepping the clock. > > - Ed > > On 6/15/23 19:06, Nemo Crypto wrote: > > Hi LinuxPTPUsers, > > > > I have this weird problem sometimes with phc2syc. Please look at the log > > below, ptp4l service is perfectly okay. But the phc2sys is stuck > > (although it measures the offset correctly) and not adjusting the system > > time for some reason. Why would this happen? Waiting for more than 30 > > minutes didn't fix this problem. > > > > Several reboots fixes the issue. > > > > 1970-01-01T00:05:09.600718+00:00 phc2sys: [304.725] CLOCK_REALTIME > phc offset -1686774497008453238 s2 freq -100000000 delay 1372 > > > > 1970-01-01T00:05:09.628181+00:00 ptp4l: [304.752] master offset > 198 s3 freq -35590 path delay 323 > > > > 1970-01-01T00:05:10.611304+00:00 phc2sys: [305.735] CLOCK_REALTIME > phc offset -1686774496916583694 s2 freq -100000000 delay 1372 > > > > 1970-01-01T00:05:10.728644+00:00 ptp4l: [305.853] master offset > -63 s3 freq -35792 path delay 323 > > > > 1970-01-01T00:05:11.618971+00:00 phc2sys: [306.743] CLOCK_REALTIME > phc offset -1686774496824974153 s2 freq -100000000 delay 1373 > > > > 1970-01-01T00:05:11.829106+00:00 ptp4l: [306.953] master offset > -121 s3 freq -35869 path delay 323 > > > > 1970-01-01T00:05:12.717043+00:00 phc2sys: [307.841] CLOCK_REALTIME > phc offset -1686774496725151408 s2 freq -100000000 delay 1408 > > > > 1970-01-01T00:05:12.929540+00:00 ptp4l: [308.054] master offset > 111 s3 freq -35673 path delay 323 > > > > 1970-01-01T00:05:13.721458+00:00 phc2sys: [308.846] CLOCK_REALTIME > phc offset -1686774496633835940 s2 freq -100000000 delay 1373 > > > > 1970-01-01T00:05:14.030032+00:00 ptp4l: [309.154] master offset > -112 s3 freq -35863 path delay 323 > > > > > > > > Thanks, > > Nemo > > > > > > > > > > > > > > > _______________________________________________ > > Linuxptp-users mailing list > > Lin...@li... > <mailto:Lin...@li...> > > https://lists.sourceforge.net/lists/listinfo/linuxptp-users > <https://lists.sourceforge.net/lists/listinfo/linuxptp-users> > > _______________________________________________ > Linuxptp-users mailing list > Lin...@li... > <mailto:Lin...@li...> > https://lists.sourceforge.net/lists/listinfo/linuxptp-users > <https://lists.sourceforge.net/lists/listinfo/linuxptp-users> > |
|
From: Nemo C. <nem...@gm...> - 2023-06-17 00:24:43
|
This is helpful, thank you Ed.
Thanks,
Nemo
On Friday, 16 June, 2023 at 03:08:38 pm GMT-4, Ed Branch <br...@ar...> wrote:
TLDR: It could also be the GM.
I see this on embedded networks, ie. the whole network including the GM
is on the same power switch, it all comes up together, and none of the
RTC's have battery backup.
Maybe the "GM" is just another embedded computer that gets time from GPS
or something like that. So the "GM" starts serving time over PTP
starting at the epoch, everything sync's to it, then the GM jumps when
it gets GPS lock.
Alternately, the GM just takes longer to boot for whatever reason. So
some random host is selected as master, serving the epoch over PTP, and
all the other hosts sync to that. Then, when the GM starts serving the
real time, the other hosts switch to it. But again, phc2sys has already
sync'd to the wrong time.
- Ed
On 6/16/23 10:19, Nemo Crypto wrote:
> Thank you Ed!
> I am using -w option but not -S. I now understand why the clock step is
> not happening. But with -w phc2sys should start only after ptp4l, isn't it?
> If yes, then why would this happen - "the clock has jumped from the
> epoch after phc2sys started".
>
> The only way is probably that "-w" option is not effective?
>
>
> Thanks,
> Nemo
>
>
> On Friday, 16 June, 2023 at 09:33:23 am GMT-4, Ed Branch
> <br...@ar...> wrote:
>
>
> You need either of:
>
> 1) a battery backup for the real-time clock so that it starts at a
> reasonable time.
> 2) Use -w option to phc2sys so that it waits until ptp4l is synchronized.
> 3) Use the -S option to phc2sys so that it steps the clock if the offset
> is above some threshold.
>
> By default, phc2sys will step the clock only once at startup, but here
> the clock has jumped from the epoch after phc2sys started. It would take
> a very long time (530 years in this case) to adjust for that without
> stepping the clock.
>
> - Ed
>
> On 6/15/23 19:06, Nemo Crypto wrote:
> > Hi LinuxPTPUsers,
> >
> > I have this weird problem sometimes with phc2syc. Please look at the log
> > below, ptp4l service is perfectly okay. But the phc2sys is stuck
> > (although it measures the offset correctly) and not adjusting the system
> > time for some reason. Why would this happen? Waiting for more than 30
> > minutes didn't fix this problem.
> >
> > Several reboots fixes the issue.
> >
> > 1970-01-01T00:05:09.600718+00:00 phc2sys: [304.725] CLOCK_REALTIME
> phc offset -1686774497008453238 s2 freq -100000000 delay 1372
> >
> > 1970-01-01T00:05:09.628181+00:00 ptp4l: [304.752] master offset
> 198 s3 freq -35590 path delay 323
> >
> > 1970-01-01T00:05:10.611304+00:00 phc2sys: [305.735] CLOCK_REALTIME
> phc offset -1686774496916583694 s2 freq -100000000 delay 1372
> >
> > 1970-01-01T00:05:10.728644+00:00 ptp4l: [305.853] master offset
> -63 s3 freq -35792 path delay 323
> >
> > 1970-01-01T00:05:11.618971+00:00 phc2sys: [306.743] CLOCK_REALTIME
> phc offset -1686774496824974153 s2 freq -100000000 delay 1373
> >
> > 1970-01-01T00:05:11.829106+00:00 ptp4l: [306.953] master offset
> -121 s3 freq -35869 path delay 323
> >
> > 1970-01-01T00:05:12.717043+00:00 phc2sys: [307.841] CLOCK_REALTIME
> phc offset -1686774496725151408 s2 freq -100000000 delay 1408
> >
> > 1970-01-01T00:05:12.929540+00:00 ptp4l: [308.054] master offset
> 111 s3 freq -35673 path delay 323
> >
> > 1970-01-01T00:05:13.721458+00:00 phc2sys: [308.846] CLOCK_REALTIME
> phc offset -1686774496633835940 s2 freq -100000000 delay 1373
> >
> > 1970-01-01T00:05:14.030032+00:00 ptp4l: [309.154] master offset
> -112 s3 freq -35863 path delay 323
> >
> >
> >
> > Thanks,
> > Nemo
>
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > Linuxptp-users mailing list
> > Lin...@li...
> <mailto:Lin...@li...>
> > https://lists.sourceforge.net/lists/listinfo/linuxptp-users
> <https://lists.sourceforge.net/lists/listinfo/linuxptp-users>
>
> _______________________________________________
> Linuxptp-users mailing list
> Lin...@li...
> <mailto:Lin...@li...>
> https://lists.sourceforge.net/lists/listinfo/linuxptp-users
> <https://lists.sourceforge.net/lists/listinfo/linuxptp-users>
>
_______________________________________________
Linuxptp-users mailing list
Lin...@li...
https://lists.sourceforge.net/lists/listinfo/linuxptp-users
|
|
From: Jacob K. <jac...@in...> - 2023-06-21 05:07:25
|
On 6/11/2023 11:53 PM, Miroslav Lichvar wrote: > On Mon, Jun 12, 2023 at 12:24:19PM +0800, egg car wrote: >> 5) not only the 4 ports on the same nic, any combination of ptp0~ptp9 results >> the same. >> >> >> Have gone through the phc2sys codes, find nothing reasonable that explains >> this issue so far. >> >> I can see when the large jitter happens, the measured delay also varies a lot, >> perhapse it's related to the pcie tranfication process? > > Yes, it's related to PCIe delays. The problem is that the kernel > doesn't provide an ioctl to measure offset between two PHCs, so > phc2sys has to do it in user space using clock_gettime(), which > doesn't work very well (large delay, jitter and asymmetry). > > For PHC vs system clock measurements there is an optimized path in the > kernel which for most drivers limits the delay to a single PCIe read. > > If you need to synchronize multiple PHCs to each other, it's better to > use one phc2sys instance to synchronize the system clock to the source > PHC and then another phc2sys instance to synchronize the rest of PHCs > to the system clock. > In theory the kernel could be extended with an interface to perform the comparison between two clocks in-kernel without a switch. That would be a bit better than doing it in user space. However, on newer platforms its likely going to be better to still use the PHC + system clock at least on systems which support PCIe PTM, as this allows capturing the PTP hardware clock time and system clock time in hardware. I do not think it is possible to have arbitrary clocks go through PTM.. I suppose you could do something like the following tho: 1) use PTM / getcrosstimestamp to get a hardware cross timestamp of Clock A + the system time 2) use PTM / getcrosstimestamp to get a hardware cross timestamp of Clock B + the system time 3) calculate difference between system time A and system time B measurements 4) use that to get a pretty good difference between Clock A and Clock B time by using this relation: (offset of A to B) = (offset of A to system) + (offset of B to system) - ((system time from first PTM) - (system time from second PTM)) |
|
From: Miroslav L. <mli...@re...> - 2023-06-21 07:52:28
|
On Tue, Jun 20, 2023 at 10:07:14PM -0700, Jacob Keller wrote: > On 6/11/2023 11:53 PM, Miroslav Lichvar wrote: > > If you need to synchronize multiple PHCs to each other, it's better to > > use one phc2sys instance to synchronize the system clock to the source > > PHC and then another phc2sys instance to synchronize the rest of PHCs > > to the system clock. > > > > In theory the kernel could be extended with an interface to perform the > comparison between two clocks in-kernel without a switch. That would be > a bit better than doing it in user space. The drivers could provide a callback for other drivers to read the lowest part of the timestamp and another callback to complete the timestamp. One issue with this approach would be increased delay due to the measurement doing 3 PCIe reads instead of 1 PCIe read and 2 system clock reads. Another issue might with hardware which cannot read the lowest part of the timestamp (latching) twice without reading the rest, if such a thing exists. A better approach would be to use the system clock as a reference, similarly to what you described for PTM, except doing it multiple times in the style of PTP_SYS_OFFSET_EXTENDED with interleaved readings SYS, PHC1, SYS, PHC2, SYS, PHC1, SYS ... and combining the measurements in the kernel or user space. -- Miroslav Lichvar |
|
From: Keller, J. E <jac...@in...> - 2023-06-21 20:57:14
|
> -----Original Message----- > From: Miroslav Lichvar <mli...@re...> > Sent: Wednesday, June 21, 2023 12:52 AM > To: Keller, Jacob E <jac...@in...> > Cc: lin...@li... > Subject: Re: [Linuxptp-users] [issue] phc2sys results large jitter with multiple '-c' > > On Tue, Jun 20, 2023 at 10:07:14PM -0700, Jacob Keller wrote: > > On 6/11/2023 11:53 PM, Miroslav Lichvar wrote: > > > If you need to synchronize multiple PHCs to each other, it's better to > > > use one phc2sys instance to synchronize the system clock to the source > > > PHC and then another phc2sys instance to synchronize the rest of PHCs > > > to the system clock. > > > > > > > In theory the kernel could be extended with an interface to perform the > > comparison between two clocks in-kernel without a switch. That would be > > a bit better than doing it in user space. > > The drivers could provide a callback for other drivers to read the > lowest part of the timestamp and another callback to complete the > timestamp. One issue with this approach would be increased delay due > to the measurement doing 3 PCIe reads instead of 1 PCIe read and > 2 system clock reads. Another issue might with hardware which cannot > read the lowest part of the timestamp (latching) twice without reading > the rest, if such a thing exists. > > A better approach would be to use the system clock as a reference, > similarly to what you described for PTM, except doing it multiple > times in the style of PTP_SYS_OFFSET_EXTENDED with interleaved > readings SYS, PHC1, SYS, PHC2, SYS, PHC1, SYS ... and combining the > measurements in the kernel or user space. > > -- > Miroslav Lichvar Ya that would work. I don't know if there is sufficient demand to add such an interface though. Thanks, Jake |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X