linuxptp-users

[Linuxptp-users] ptp4l and SyncE

[Cliff S. <th...@go...>]

I'm looking for tips for optimally configuring ptp4l in a SyncE-enabled
network.

Consider the following architecture - the grandmaster's clock and the PTP
clock in my slave's NIC share the same frequency source via SyncE. The end
goal is to synchronize my slave's Linux system time to the grandmaster.

Configuration thoughts:
- It seems to me that ptp4l should only examine the time offset between the
master and slave, and not try to estimate or correct for differences in
frequency. I couldn't find a setting for this, and it seems like frequency
estimation is baked pretty deeply into the existing control algorithms.

- It seems reasonable to always step the NIC clock rather than slew its
frequency. I guess I'd configure that by setting an extremely small step
threshold. phc2sys would still slew the system clock to preserve system
clock monotonicity (as an aside, I really wish that Linux split the
concepts of 'how long a second is' from 'what time is it now' so that the
latter could be slewed independently)

- Since this is a SyncE network, maybe the error introduced by having ptp4l
try to estimate frequency will be low enough to not matter, and so maybe I
don't need to do any special configuration.

I'd appreciate any thoughts. Does anyone have any pointers to documentation
or any analysis on how SyncE and 1588 should work together in general?

Thanks!

Re: [Linuxptp-users] ptp4l and SyncE

[Miroslav L. <mli...@re...>]

On Mon, Apr 23, 2018 at 07:20:25PM +0000, Cliff Spradlin via Linuxptp-users wrote:
> Configuration thoughts:
> - It seems to me that ptp4l should only examine the time offset between the
> master and slave, and not try to estimate or correct for differences in
> frequency. I couldn't find a setting for this, and it seems like frequency
> estimation is baked pretty deeply into the existing control algorithms.

ptp4l can be configured to use the nullf servo. It doesn't change the
frequency and can correct the clock only by stepping (if allowed by
the *step_threshold options). This assumes the measured offset is
consistenly below one nanosecond.

I have a pair of directly connected X550 cards, which behave a bit
like SyncE with the frequency. Their clocks seem to be locked to each
other or some other common source. I'm not sure how this works and I'd
love to see some explanation.

However, in this case the measured offset is larger than 1ns and the
nullf servo is constantly stepping or trying to step the clock. The PI
servo configured with a very small I constant is a better choice here.
It just takes longer to start. I guess a new option could be added to
turn it into a pure "P" servo.

-- 
Miroslav Lichvar

Re: [Linuxptp-users] ptp4l and SyncE

[Keller, J. E <jac...@in...>]

> -----Original Message-----
> From: Miroslav Lichvar [mailto:mli...@re...]
> Sent: Tuesday, April 24, 2018 12:16 AM
> To: Cliff Spradlin <th...@go...>
> Cc: lin...@li...
> Subject: Re: [Linuxptp-users] ptp4l and SyncE
> 
> I have a pair of directly connected X550 cards, which behave a bit
> like SyncE with the frequency. Their clocks seem to be locked to each
> other or some other common source. I'm not sure how this works and I'd
> love to see some explanation.

If it's two different X550 cards, I don't know. If it's two ports of the same card, the devices *do* share the same clock source (but you can't atomically set the registers for them, so it's not feasible to set it up as a proper boundary clock)

Thanks,
Jake

Re: [Linuxptp-users] ptp4l and SyncE

[Miroslav L. <mli...@re...>]

On Tue, Apr 24, 2018 at 05:10:48PM +0000, Keller, Jacob E wrote:
> > -----Original Message-----
> > From: Miroslav Lichvar [mailto:mli...@re...]
> > Sent: Tuesday, April 24, 2018 12:16 AM
> > To: Cliff Spradlin <th...@go...>
> > Cc: lin...@li...
> > Subject: Re: [Linuxptp-users] ptp4l and SyncE
> > 
> > I have a pair of directly connected X550 cards, which behave a bit
> > like SyncE with the frequency. Their clocks seem to be locked to each
> > other or some other common source. I'm not sure how this works and I'd
> > love to see some explanation.
> 
> If it's two different X550 cards, I don't know. If it's two ports of the same card, the devices *do* share the same clock source (but you can't atomically set the registers for them, so it's not feasible to set it up as a proper boundary clock)

It's two different X550T cards. They have one port connected to the
same switch and one port connected directly between them. Maybe the
cards use one clock for both MAC and PHY layers? If the two ports of
one card shared the clock too, that wouldn't work with different ports
running at slightly different frequencies, right? (I don't know much
about Ethernet and whether any of this makes any sense).

-- 
Miroslav Lichvar

Re: [Linuxptp-users] ptp4l and SyncE

[Keller, J. E <jac...@in...>]

> -----Original Message-----
> From: Miroslav Lichvar [mailto:mli...@re...]
> Sent: Thursday, April 26, 2018 12:33 AM
> To: Keller, Jacob E <jac...@in...>
> Cc: Cliff Spradlin <th...@go...>; lin...@li...
> Subject: Re: [Linuxptp-users] ptp4l and SyncE
> 
> On Tue, Apr 24, 2018 at 05:10:48PM +0000, Keller, Jacob E wrote:
> > > -----Original Message-----
> > > From: Miroslav Lichvar [mailto:mli...@re...]
> > > Sent: Tuesday, April 24, 2018 12:16 AM
> > > To: Cliff Spradlin <th...@go...>
> > > Cc: lin...@li...
> > > Subject: Re: [Linuxptp-users] ptp4l and SyncE
> > >
> > > I have a pair of directly connected X550 cards, which behave a bit
> > > like SyncE with the frequency. Their clocks seem to be locked to each
> > > other or some other common source. I'm not sure how this works and I'd
> > > love to see some explanation.
> >
> > If it's two different X550 cards, I don't know. If it's two ports of the same card,
> the devices *do* share the same clock source (but you can't atomically set the
> registers for them, so it's not feasible to set it up as a proper boundary clock)
> 
> It's two different X550T cards. They have one port connected to the
> same switch and one port connected directly between them. Maybe the
> cards use one clock for both MAC and PHY layers? If the two ports of
> one card shared the clock too, that wouldn't work with different ports
> running at slightly different frequencies, right? (I don't know much
> about Ethernet and whether any of this makes any sense).
> 

Each port uses the same clock *source*, but uses its own set of registers to tune the frequency, so each one can run independently. You can't run them as a "2 ports on the same clock" boundary clock style setup, because it is not possible to atomically set all the registers to the same values. Thus, in practice they have separate clocks, but they *do* source from the same frequency.

In the case of two X550T cards, I would expect there to be *some* difference, but if the clock sources are high quality it is possible they are close enough that we can't detect any difference in them? (though, changing frequency in on PHC using the ptp API should not affect the other, unless something else is connecting them...)

Thanks,
Jake

> --
> Miroslav Lichvar

Re: [Linuxptp-users] ptp4l and SyncE

[Miroslav L. <mli...@re...>]

On Thu, Apr 26, 2018 at 06:24:39PM +0000, Keller, Jacob E wrote:
> In the case of two X550T cards, I would expect there to be *some* difference, but if the clock sources are high quality it is possible they are close enough that we can't detect any difference in them? (though, changing frequency in on PHC using the ptp API should not affect the other, unless something else is connecting them...)

As expected, changing the PHC frequency on one card doesn't seem to
change the frequency on the other.

I left them running for a couple of days, both frequencies set to 0
with phc_ctl, and logged the offset measured by ptp4l using the NTPSHM
servo (i.e. not making any adjustments). The offset between them is
not exactly zero, but it's actually about 0.46 ppb. It seems to be the
same with both 1Gb and 10Gb link speeds.

Here is a plot of the Allan deviation:
https://i.imgur.com/jp3ZE3Z.png

Unless they have a hidden chip-scale atomic clock, I think they must
be synchronized to each other in some way. Maybe something was
implemented in preparation for SyncE support?

-- 
Miroslav Lichvar

Re: [Linuxptp-users] ptp4l and SyncE

[Keller, J. E <jac...@in...>]

> -----Original Message-----
> From: Miroslav Lichvar [mailto:mli...@re...]
> Sent: Wednesday, May 02, 2018 6:46 AM
> To: Keller, Jacob E <jac...@in...>
> Cc: Cliff Spradlin <th...@go...>; lin...@li...
> Subject: Re: [Linuxptp-users] ptp4l and SyncE
> 
> On Thu, Apr 26, 2018 at 06:24:39PM +0000, Keller, Jacob E wrote:
> > In the case of two X550T cards, I would expect there to be *some* difference,
> but if the clock sources are high quality it is possible they are close enough that
> we can't detect any difference in them? (though, changing frequency in on PHC
> using the ptp API should not affect the other, unless something else is connecting
> them...)
> 
> As expected, changing the PHC frequency on one card doesn't seem to
> change the frequency on the other.
> 
> I left them running for a couple of days, both frequencies set to 0
> with phc_ctl, and logged the offset measured by ptp4l using the NTPSHM
> servo (i.e. not making any adjustments). The offset between them is
> not exactly zero, but it's actually about 0.46 ppb. It seems to be the
> same with both 1Gb and 10Gb link speeds.
> 
> Here is a plot of the Allan deviation:
> https://i.imgur.com/jp3ZE3Z.png
> 
> Unless they have a hidden chip-scale atomic clock, I think they must
> be synchronized to each other in some way. Maybe something was
> implemented in preparation for SyncE support?
> 
> --
> Miroslav Lichvar

Could be, I don't know enough about the clock setup in the X550T.

Thanks,
Jake

Re: [Linuxptp-users] ptp4l and SyncE

[Richard C. <ric...@gm...>]

On Tue, Apr 24, 2018 at 09:15:49AM +0200, Miroslav Lichvar wrote:
> I have a pair of directly connected X550 cards, which behave a bit
> like SyncE with the frequency. Their clocks seem to be locked to each
> other or some other common source. I'm not sure how this works and I'd
> love to see some explanation.

All gigabit link peers are synchronized.  There is a protocol that
determines which peer is master.

With SyncE, the user configures which peer is master on the switch
ports.  If you do it right, you can syntonize your entire network this
way.

HTH,
Richard

Re: [Linuxptp-users] ptp4l and SyncE

[Miroslav L. <mli...@re...>]

On Wed, May 02, 2018 at 06:58:56AM -0700, Richard Cochran wrote:
> On Tue, Apr 24, 2018 at 09:15:49AM +0200, Miroslav Lichvar wrote:
> > I have a pair of directly connected X550 cards, which behave a bit
> > like SyncE with the frequency. Their clocks seem to be locked to each
> > other or some other common source. I'm not sure how this works and I'd
> > love to see some explanation.
> 
> All gigabit link peers are synchronized.  There is a protocol that
> determines which peer is master.

Interesting. I assume that applies only to the PHY chip and the MAC
may be but doesn't have to be synchronized to the PHY? The X550 is the
only NIC I have noticed to have a "synchronized" PHC so far.

-- 
Miroslav Lichvar

Re: [Linuxptp-users] ptp4l and SyncE

[Richard C. <ric...@gm...>]

On Wed, May 02, 2018 at 04:17:38PM +0200, Miroslav Lichvar wrote:
> Interesting. I assume that applies only to the PHY chip and the MAC
> may be but doesn't have to be synchronized to the PHY? The X550 is the
> only NIC I have noticed to have a "synchronized" PHC so far.

Right, the PHYs are always synchronized.  If the MAC time stamper is
using another clock then you won't see it from the outside.

Thanks,
Richard