Thread: [Linuxptp-users] Sudden phc offset jump
PTP IEEE 1588 stack for Linux
Brought to you by:
rcochran
Menu
▾
▴
linuxptp-users
|
From: ramesh t <ram...@ya...> - 2021-11-02 09:02:44
|
hi, We are using ptp3.0 version in C3 mode. Both ptp4l and phc2sys (to sync system clock) are running fine. Ptp4l rms value is in single digits and we see a sudden jump in phc offset as captured below resulting in date change. Oct 26 19:15:48 phc2sys_slave: [1177711.923] CLOCK_REALTIME phc offset -97 s2 freq +3454 delay 2479 Oct 26 19:15:49 ptp4l_slave: [1177712.827] rms 3 max 7 freq -2363 +/- 5 delay 86 +/- 1 Oct 26 19:15:49 phc2sys_slave: [1177712.924] CLOCK_REALTIME phc offset -61 s2 freq +3461 delay 2433 Oct 12 05:18:20 ptp4l_slave: [1177713.819] rms 3 max 8 freq -2364 +/- 5 delay 86 +/- 1 Oct 19 08:03:31 phc2sys_slave: [1177713.924] clockcheck: clock jumped backward or running slower than expected! Oct 19 08:03:31 phc2sys_slave: [1177713.924] CLOCK_REALTIME phc offset -645138994849912 s0 freq +3461 delay 2478 Oct 25 18:44:33 ptp4l_slave: [1177714.811] rms 3 max 6 freq -2361 +/- 4 delay 86 +/- 1 Sep 30 01:35:47 phc2sys_slave: [1177714.924] clockcheck: clock jumped backward or running slower than expected! Sep 30 01:35:47 phc2sys_slave: [1177714.924] CLOCK_REALTIME phc offset -2310003858831797 s0 freq +3461 delay 2472 Oct 12 07:16:54 ptp4l_slave: [1177715.803] rms 3 max 5 freq -2362 +/- 5 delay 87 +/- 1 Is there any known issues with ptp3.0 version? Please suggest. regards, Ramesh |
|
From: Miroslav L. <mli...@re...> - 2021-11-02 09:07:42
|
On Tue, Nov 02, 2021 at 09:02:19AM +0000, ramesh t via Linuxptp-devel wrote: > hi, > > We are using ptp3.0 version in C3 mode. Both ptp4l and phc2sys (to sync system clock) are running fine. > > Ptp4l rms value is in single digits and we see a sudden jump in phc offset as captured below resulting in date change. > > Oct 26 19:15:48 phc2sys_slave: [1177711.923] CLOCK_REALTIME phc offset -97 s2 freq +3454 delay 2479 > Oct 26 19:15:49 ptp4l_slave: [1177712.827] rms 3 max 7 freq -2363 +/- 5 delay 86 +/- 1 > Oct 26 19:15:49 phc2sys_slave: [1177712.924] CLOCK_REALTIME phc offset -61 s2 freq +3461 delay 2433 > Oct 12 05:18:20 ptp4l_slave: [1177713.819] rms 3 max 8 freq -2364 +/- 5 delay 86 +/- 1 > Oct 19 08:03:31 phc2sys_slave: [1177713.924] clockcheck: clock jumped backward or running slower than expected! > Oct 19 08:03:31 phc2sys_slave: [1177713.924] CLOCK_REALTIME phc offset -645138994849912 s0 freq +3461 delay 2478 > Oct 25 18:44:33 ptp4l_slave: [1177714.811] rms 3 max 6 freq -2361 +/- 4 delay 86 +/- 1 > Sep 30 01:35:47 phc2sys_slave: [1177714.924] clockcheck: clock jumped backward or running slower than expected! > Sep 30 01:35:47 phc2sys_slave: [1177714.924] CLOCK_REALTIME phc offset -2310003858831797 s0 freq +3461 delay 2472 > Oct 12 07:16:54 ptp4l_slave: [1177715.803] rms 3 max 5 freq -2362 +/- 5 delay 87 +/- 1 > > Is there any known issues with ptp3.0 version? Please suggest. What hardware do you use? It's most likely a driver bug or something else (e.g. an NTP client) is trying to control the clock. -- Miroslav Lichvar |
|
From: ramesh t <ram...@ya...> - 2021-11-02 09:56:01
|
hi, We are using Dell Supermicro 6212 with Intel E810 NIC card. And system was stable with ptp2.0 load. We have disable ntpd, chronyd, systemd-timesyncd and systemd-timedated services. regards, Ramesh On Tuesday, November 2, 2021, 02:37:27 PM GMT+5:30, Miroslav Lichvar <mli...@re...> wrote: On Tue, Nov 02, 2021 at 09:02:19AM +0000, ramesh t via Linuxptp-devel wrote: > hi, > > We are using ptp3.0 version in C3 mode. Both ptp4l and phc2sys (to sync system clock) are running fine. > > Ptp4l rms value is in single digits and we see a sudden jump in phc offset as captured below resulting in date change. > > Oct 26 19:15:48 phc2sys_slave: [1177711.923] CLOCK_REALTIME phc offset -97 s2 freq +3454 delay 2479 > Oct 26 19:15:49 ptp4l_slave: [1177712.827] rms 3 max 7 freq -2363 +/- 5 delay 86 +/- 1 > Oct 26 19:15:49 phc2sys_slave: [1177712.924] CLOCK_REALTIME phc offset -61 s2 freq +3461 delay 2433 > Oct 12 05:18:20 ptp4l_slave: [1177713.819] rms 3 max 8 freq -2364 +/- 5 delay 86 +/- 1 > Oct 19 08:03:31 phc2sys_slave: [1177713.924] clockcheck: clock jumped backward or running slower than expected! > Oct 19 08:03:31 phc2sys_slave: [1177713.924] CLOCK_REALTIME phc offset -645138994849912 s0 freq +3461 delay 2478 > Oct 25 18:44:33 ptp4l_slave: [1177714.811] rms 3 max 6 freq -2361 +/- 4 delay 86 +/- 1 > Sep 30 01:35:47 phc2sys_slave: [1177714.924] clockcheck: clock jumped backward or running slower than expected! > Sep 30 01:35:47 phc2sys_slave: [1177714.924] CLOCK_REALTIME phc offset -2310003858831797 s0 freq +3461 delay 2472 > Oct 12 07:16:54 ptp4l_slave: [1177715.803] rms 3 max 5 freq -2362 +/- 5 delay 87 +/- 1 > > Is there any known issues with ptp3.0 version? Please suggest. What hardware do you use? It's most likely a driver bug or something else (e.g. an NTP client) is trying to control the clock. -- Miroslav Lichvar |
|
From: ramesh t <ram...@ya...> - 2021-11-02 15:32:14
|
hi, Any suggestion for the below mentioned issue? Please let me know. regards, Ramesh On Tuesday, November 2, 2021, 03:27:54 PM GMT+5:30, ramesh t via Linuxptp-users <lin...@li...> wrote: hi, We are using Dell Supermicro 6212 with Intel E810 NIC card. And system was stable with ptp2.0 load. We have disable ntpd, chronyd, systemd-timesyncd and systemd-timedated services. regards, Ramesh On Tuesday, November 2, 2021, 02:37:27 PM GMT+5:30, Miroslav Lichvar <mli...@re...> wrote: On Tue, Nov 02, 2021 at 09:02:19AM +0000, ramesh t via Linuxptp-devel wrote: > hi, > > We are using ptp3.0 version in C3 mode. Both ptp4l and phc2sys (to sync system clock) are running fine. > > Ptp4l rms value is in single digits and we see a sudden jump in phc offset as captured below resulting in date change. > > Oct 26 19:15:48 phc2sys_slave: [1177711.923] CLOCK_REALTIME phc offset -97 s2 freq +3454 delay 2479 > Oct 26 19:15:49 ptp4l_slave: [1177712.827] rms 3 max 7 freq -2363 +/- 5 delay 86 +/- 1 > Oct 26 19:15:49 phc2sys_slave: [1177712.924] CLOCK_REALTIME phc offset -61 s2 freq +3461 delay 2433 > Oct 12 05:18:20 ptp4l_slave: [1177713.819] rms 3 max 8 freq -2364 +/- 5 delay 86 +/- 1 > Oct 19 08:03:31 phc2sys_slave: [1177713.924] clockcheck: clock jumped backward or running slower than expected! > Oct 19 08:03:31 phc2sys_slave: [1177713.924] CLOCK_REALTIME phc offset -645138994849912 s0 freq +3461 delay 2478 > Oct 25 18:44:33 ptp4l_slave: [1177714.811] rms 3 max 6 freq -2361 +/- 4 delay 86 +/- 1 > Sep 30 01:35:47 phc2sys_slave: [1177714.924] clockcheck: clock jumped backward or running slower than expected! > Sep 30 01:35:47 phc2sys_slave: [1177714.924] CLOCK_REALTIME phc offset -2310003858831797 s0 freq +3461 delay 2472 > Oct 12 07:16:54 ptp4l_slave: [1177715.803] rms 3 max 5 freq -2362 +/- 5 delay 87 +/- 1 > > Is there any known issues with ptp3.0 version? Please suggest. What hardware do you use? It's most likely a driver bug or something else (e.g. an NTP client) is trying to control the clock. -- Miroslav Lichvar _______________________________________________ Linuxptp-users mailing list Lin...@li... https://lists.sourceforge.net/lists/listinfo/linuxptp-users |
|
From: Richard C. <ric...@gm...> - 2021-11-02 16:40:14
|
On Tue, Nov 02, 2021 at 03:32:04PM +0000, ramesh t wrote: > hi, > > Any suggestion for the below mentioned issue? Please let me know. Looks like a HW and/or driver issue. I would start by validating the HW/driver. For example, a long term test of phc_ctl eth0 get HTH, Richard |
|
From: ramesh t <ram...@ya...> - 2021-11-03 03:21:03
|
Thanks Richard. Will trigger a script to capture the output. regards, Ramesh On Tuesday, November 2, 2021, 10:02:01 PM GMT+5:30, Richard Cochran <ric...@gm...> wrote: On Tue, Nov 02, 2021 at 03:32:04PM +0000, ramesh t wrote: > hi, > > Any suggestion for the below mentioned issue? Please let me know. Looks like a HW and/or driver issue. I would start by validating the HW/driver. For example, a long term test of phc_ctl eth0 get HTH, Richard |
|
From: Keller, J. E <jac...@in...> - 2021-11-02 19:58:40
|
On 11/2/2021 2:55 AM, ramesh t via Linuxptp-users wrote: > hi, > > We are using Dell Supermicro 6212 with Intel E810 NIC card. And system was stable with ptp2.0 load. > We have disable ntpd, chronyd, systemd-timesyncd and systemd-timedated services. > Hi! Thanks for the report! The E810 code was recently upstreamed. Any chance you can tell which kernel version you are using? I believe this is a manifestation of a known issue that we're currently trying to track down. I am currently forwarding this to the engineer who is working on what I believe is the same issue. I will do my best to get back to you here if I hear anything else. Any further information you could provide may help us root cause this! Thanks, Jake |
|
From: ramesh t <ram...@ya...> - 2021-11-03 14:44:29
|
Hello Jake, Please find the requested info below. NIC driver: driver: ice version: 1.3.2 firmware-version: 2.30 0x80006c8d 1.2877.0 Kernel version: uname -a Linux cyswy002r-mvnr-p004-sdlas00222a-mv-du001-55d64d7457-dmp8s 4.19.177-rt72-2.ph3-rt #1-photon SMP PREEMPT RT Fri Mar 5 02:22:24 UTC 2021 x86_64 GNU/Linux Please suggest. regards, Ramesh On Wednesday, November 3, 2021, 01:38:01 AM GMT+5:30, Keller, Jacob E <jac...@in...> wrote: On 11/2/2021 2:55 AM, ramesh t via Linuxptp-users wrote: > hi, > > We are using Dell Supermicro 6212 with Intel E810 NIC card. And system was stable with ptp2.0 load. > We have disable ntpd, chronyd, systemd-timesyncd and systemd-timedated services. > Hi! Thanks for the report! The E810 code was recently upstreamed. Any chance you can tell which kernel version you are using? I believe this is a manifestation of a known issue that we're currently trying to track down. I am currently forwarding this to the engineer who is working on what I believe is the same issue. I will do my best to get back to you here if I hear anything else. Any further information you could provide may help us root cause this! Thanks, Jake _______________________________________________ Linuxptp-users mailing list Lin...@li... https://lists.sourceforge.net/lists/listinfo/linuxptp-users |
|
From: Keller, J. E <jac...@in...> - 2021-11-03 19:03:54
|
On 11/3/2021 7:44 AM, ramesh t wrote: > Hello Jake, > > Please find the requested info below. > NIC driver: > driver: ice > version: 1.3.2 > firmware-version: 2.30 0x80006c8d 1.2877.0 > > Kernel version: > uname -a > Linux cyswy002r-mvnr-p004-sdlas00222a-mv-du001-55d64d7457-dmp8s 4.19.177-rt72-2.ph3-rt #1-photon SMP PREEMPT RT Fri Mar 5 02:22:24 UTC 2021 x86_64 GNU/Linux > > Please suggest. > > regards, > Ramesh This looks like you're using the out-of-tree driver we publish on source forge. You could try using a later version (I think we recently published the 1.6.7 driver which was published in September. There has been some work on PTP since the 1.3.x drivers and its quite possible things have improved. Alternatively, you could try a newer kernel which has the PTP support upstream now, since v5.14 Hope this helps, Jake |
|
From: ramesh t <ram...@ya...> - 2021-11-11 16:54:06
|
hi Richard, Verified on the nodes where the problem re-occurred after almost a week using phc_ctl. NIC phc time was fine. But only system time had changed. Also as suggested was running a script to capture phc_ctl periodically, but the issue didn't happen on those node. Even nodes running ptp2.0 didn't report the problem. Nov 10 23:19:13 phc2sys: [43031.919] CLOCK_REALTIME phc offset -62 s2 freq +4363 delay 2331 Nov 10 23:19:14 ptp4l: [43032.308] rms 3 max 6 freq -834 +/- 4 delay 83 +/- 1 Nov 10 02:38:29 phc2sys: [43032.919] clockcheck: clock jumped backward or running slower than expected! Nov 10 02:38:29 phc2sys: [43032.919] CLOCK_REALTIME phc offset -74444707194042 s0 freq +4363 delay 2459 Nov 10 17:47:19 ptp4l: [43033.302] rms 3 max 7 freq -837 +/- 5 delay 83 +/- 1 Nov 10 22:36:11 phc2sys: [93661.492] CLOCK_REALTIME phc offset 34 s2 freq +9568 delay 2352 Nov 10 22:36:12 ptp4l: [93662.280] rms 3 max 5 freq +3086 +/- 4 delay 85 +/- 0 Nov 4 12:50:40 phc2sys: [93662.493] clockcheck: clock jumped backward or running slower than expected! Nov 4 12:50:40 phc2sys: [93662.493] CLOCK_REALTIME phc offset -553532738053986 s0 freq +9568 delay 2455 Nov 10 15:54:49 ptp4l: [93663.272] rms 2 max 5 freq +3084 +/- 3 delay 86 +/- 1 I guess there are two possibilities. 1) Some process is changing the time? Is there a way to monitor which all processes modify the system time? 2) phc2sys code has issues in 3.0 version? Please suggest. regards, Ramesh On Tuesday, November 2, 2021, 10:02:01 PM GMT+5:30, Richard Cochran <ric...@gm...> wrote: On Tue, Nov 02, 2021 at 03:32:04PM +0000, ramesh t wrote: > hi, > > Any suggestion for the below mentioned issue? Please let me know. Looks like a HW and/or driver issue. I would start by validating the HW/driver. For example, a long term test of phc_ctl eth0 get HTH, Richard |
|
From: ramesh t <ram...@ya...> - 2021-11-25 05:56:49
|
hi Jake, Richard, Observed issue with ptp 2.0 also. But based on below logs. Nov 10 03:49:04 ptp4l: [21956.414] rms 3 max 6 freq +2783 +/- 5 delay 85 +/- 1 Nov 10 03:49:05 phc2sys: [21956.824] CLOCK_REALTIME phc offset 8 s2 freq +9775 delay 2252 Nov 4 12:50:40 ptp4l: [21957.406] rms 3 max 5 freq +2780 +/- 5 delay 86 +/- 1 Nov 6 05:55:10 phc2sys: [21957.824] clockcheck: clock jumped backward or running slower than expected! Nov 6 05:55:10 phc2sys: [21957.824] CLOCK_REALTIME phc offset -338035766568736 s0 freq +9775 delay 2461 Nov 6 07:39:56 ptp4l: [21958.399] rms 2 max 3 freq +2783 +/- 3 delay 85 +/- 1 Nov 9 21:52:19 phc2sys: [21958.824] Nov 9 21:52:19 phc2sys: [21958.824] CLOCK_REALTIME phc offset -21408060435248 sclockcheck: clock jumped forward or running faster than expected!0 freq +9775 delay 2468 Nov 9 21:52:19 ptp4l: [21959.393] rms 3 max 6 freq +2784 +/- 4 delay 84 +/- 1 Nov 9 21:52:20 phc2sys: [21959.825] CLOCK_REALTIME phc offset -21408060435245 s2 freq +9772 delay 2472 Observing time change 3 to 4 times? ptp4l seems to have proper value ie phc offset -21408060435248 (Nov 9 21.52.19 + 21408 secs (5 hour + 56 minutes) around Nov 10 3:49. Please suggest. regards, Ramesh On Thursday, November 11, 2021, 10:26:28 PM GMT+5:30, ramesh t via Linuxptp-users <lin...@li...> wrote: hi Richard, Verified on the nodes where the problem re-occurred after almost a week using phc_ctl. NIC phc time was fine. But only system time had changed. Also as suggested was running a script to capture phc_ctl periodically, but the issue didn't happen on those node. Even nodes running ptp2.0 didn't report the problem. Nov 10 23:19:13 phc2sys: [43031.919] CLOCK_REALTIME phc offset -62 s2 freq +4363 delay 2331 Nov 10 23:19:14 ptp4l: [43032.308] rms 3 max 6 freq -834 +/- 4 delay 83 +/- 1 Nov 10 02:38:29 phc2sys: [43032.919] clockcheck: clock jumped backward or running slower than expected! Nov 10 02:38:29 phc2sys: [43032.919] CLOCK_REALTIME phc offset -74444707194042 s0 freq +4363 delay 2459 Nov 10 17:47:19 ptp4l: [43033.302] rms 3 max 7 freq -837 +/- 5 delay 83 +/- 1 Nov 10 22:36:11 phc2sys: [93661.492] CLOCK_REALTIME phc offset 34 s2 freq +9568 delay 2352 Nov 10 22:36:12 ptp4l: [93662.280] rms 3 max 5 freq +3086 +/- 4 delay 85 +/- 0 Nov 4 12:50:40 phc2sys: [93662.493] clockcheck: clock jumped backward or running slower than expected! Nov 4 12:50:40 phc2sys: [93662.493] CLOCK_REALTIME phc offset -553532738053986 s0 freq +9568 delay 2455 Nov 10 15:54:49 ptp4l: [93663.272] rms 2 max 5 freq +3084 +/- 3 delay 86 +/- 1 I guess there are two possibilities. 1) Some process is changing the time? Is there a way to monitor which all processes modify the system time? 2) phc2sys code has issues in 3.0 version? Please suggest. regards, Ramesh On Tuesday, November 2, 2021, 10:02:01 PM GMT+5:30, Richard Cochran <ric...@gm...> wrote: On Tue, Nov 02, 2021 at 03:32:04PM +0000, ramesh t wrote: > hi, > > Any suggestion for the below mentioned issue? Please let me know. Looks like a HW and/or driver issue. I would start by validating the HW/driver. For example, a long term test of phc_ctl eth0 get HTH, Richard _______________________________________________ Linuxptp-users mailing list Lin...@li... https://lists.sourceforge.net/lists/listinfo/linuxptp-users |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X