linuxptp-users

[Linuxptp-users] Building a PTP grandmaster with disciplined oscillator

[Andre P. <and...@sr...>]

Dear all,

we are trying to build a PTP grandmaster with linuxptp to provide sync 
signals to telco equipment.

So far we've only used the internal oscillators of the NIC for a bunch 
of cards, Intel based and also Microchip LAN7430.

I am now looking into disciplining the oscillators with an external PPS 
input to improve frequency stability. The EVB-LAN7430 eval kit has a PPS 
input exposed over a GPIO pin.

I was wondering if someone has used this successfully at all?

In general, what are the steps to set this up? Is this something managed 
in the NIC driver itself or do I have to (permanently) run the `ts2phc` 
to keep to PHC in sync?

Are there studies/measurements that compare the stability of such a 
system to e.g. commercial PTP grandmaster solutions?

Thanks
Andre

Re: [Linuxptp-users] Building a PTP grandmaster with disciplined oscillator

[Richard C. <ric...@gm...>]

On Tue, Mar 08, 2022 at 12:38:46PM +0100, Andre Puschmann wrote:
> Dear all,
> 
> we are trying to build a PTP grandmaster with linuxptp to provide sync
> signals to telco equipment.
> 
> So far we've only used the internal oscillators of the NIC for a bunch of
> cards, Intel based and also Microchip LAN7430.
> 
> I am now looking into disciplining the oscillators with an external PPS
> input to improve frequency stability. The EVB-LAN7430 eval kit has a PPS
> input exposed over a GPIO pin.
> 
> I was wondering if someone has used this successfully at all?

Yes, I would guess so.
 
> In general, what are the steps to set this up? Is this something managed in
> the NIC driver itself or do I have to (permanently) run the `ts2phc` to keep
> to PHC in sync?

You need to run ts2phc or similar.  In general, the NICs are dumb devices.
 
> Are there studies/measurements that compare the stability of such a system
> to e.g. commercial PTP grandmaster solutions?

I haven't seen any.  I've done my own reports showing, for example,
linuxptp + Intel210 + uBlox GPS radio performing comparably to
commercial GM solutions.

However, making a reliable GM product is quite a bit of work.  The PTP
software stack is only one piece of the puzzle.

HTH,
Richard

Re: [Linuxptp-users] Building a PTP grandmaster with disciplined oscillator

[Andre P. <and...@sr...>]

On 8/3/22 15:33, Richard Cochran wrote:
>> I was wondering if someone has used this successfully at all?
> 
> Yes, I would guess so.

Is there a write-up or something similar available you know?


>> In general, what are the steps to set this up? Is this something managed in
>> the NIC driver itself or do I have to (permanently) run the `ts2phc` to keep
>> to PHC in sync?
> 
> You need to run ts2phc or similar.  In general, the NICs are dumb devices.

Got you. Thanks.

>   
>> Are there studies/measurements that compare the stability of such a system
>> to e.g. commercial PTP grandmaster solutions?
> 
> I haven't seen any.  I've done my own reports showing, for example,
> linuxptp + Intel210 + uBlox GPS radio performing comparably to
> commercial GM solutions.

Same question, anyone aware of a write-up or blog post that gives some 
instructions on HW wiring and (if any) driver modifications?


> However, making a reliable GM product is quite a bit of work.  The PTP
> software stack is only one piece of the puzzle.

Yes of course. This effort is just to develop a poor man's GM as a PoC.

But nonetheless interesting to observe that there is practically no NIC 
out there that brings together all the pieces. Should be quite easy to 
do. Cards like the Intel XXV710-DA2T or E810-XXVDA4T with GPS input are 
hard to get and $$$.

Thanks
Andre

Re: [Linuxptp-users] Building a PTP grandmaster with disciplined oscillator

[Marco D. (SIDN) <mar...@si...>]

Richard,

Op 15-03-2022 om 15:37 schreef Richard Cochran:

> 2. Run NTP client on the Linux host, and use phc2sys to sync PHC with
>     Linux system time.  Might end up milliseconds offset from global
>     time due to NTP performance alone.

This one triggered a question;

I have a setup with an interface being a PTP client (HW timestamping, a 
nice Meinberg GM and such) and phc2sys supplying time from it to another 
interface's PHC in the same server.

Is there anything to be said about the accuracy/precision on the second 
interface's PHC with this approach?

-- 
Marco

Re: [Linuxptp-users] Building a PTP grandmaster with disciplined oscillator

[Richard C. <ric...@gm...>]

On Tue, Mar 15, 2022 at 04:06:31PM +0100, Marco Davids (SIDN) via Linuxptp-users wrote:
> I have a setup with an interface being a PTP client (HW timestamping, a nice
> Meinberg GM and such) and phc2sys supplying time from it to another
> interface's PHC in the same server.
> 
> Is there anything to be said about the accuracy/precision on the second
> interface's PHC with this approach?

The answer is, "it depends".

The timestamps used by phc2sys are SW, and a busy system might
preempt phc2sys a just the wrong times, causing time error.  You a
can mitigate by using a preempt_rt kernel and giving phc2sys
sched_fifo priority.

You can force artificial system load using `hackbench` `stress` and
friends.

You can characterize your performance by measuring end to end, like
from a PPS from a client on the second port back to the Meinberg's PPS.

HTH,
Richard

Re: [Linuxptp-users] Building a PTP grandmaster with disciplined oscillator

[Miroslav L. <mli...@re...>]

On Tue, Mar 15, 2022 at 04:06:31PM +0100, Marco Davids (SIDN) via Linuxptp-users wrote:
> I have a setup with an interface being a PTP client (HW timestamping, a nice
> Meinberg GM and such) and phc2sys supplying time from it to another
> interface's PHC in the same server.
> 
> Is there anything to be said about the accuracy/precision on the second
> interface's PHC with this approach?

The accuracy and stability is impacted by asymmetry and jitter of the
PCIe latency. Half of the delay printed by phc2sys gives you an upper
bound, assuming in one direction the latency is zero and the other
direction taking all the delay, similarly to the root delay in NTP.

There is a PCIe feature supported on some HW called Precision Time
Measurement (PTM) that uses HW timestamping of PCIe messages and
should at least in theory significantly improve the accuracy and
stability, but the support in the kernel doesn't seem to be finished
yet.

If you need best performance, you should synchronize the PHC through
the system clock. That is, instead of

phc2sys -s eth0 -c eth1

use

phc2sys -s eth0 -c CLOCK_REALTIME
phc2sys -s CLOCK_REALTIME -c eth1

Ideally, the NICs should be the same model. That will cancel out
asymmetries in the HW timestamping errors, which can be larger than
the PCIe asymmetry.

-- 
Miroslav Lichvar

Re: [Linuxptp-users] Building a PTP grandmaster with disciplined oscillator

[Keller, J. E <jac...@in...>]

> -----Original Message-----
> From: Miroslav Lichvar <mli...@re...>
> Sent: Wednesday, March 16, 2022 2:32 AM
> To: Marco Davids (SIDN) <mar...@si...>
> Cc: lin...@li...
> Subject: Re: [Linuxptp-users] Building a PTP grandmaster with disciplined
> oscillator
> 
> On Tue, Mar 15, 2022 at 04:06:31PM +0100, Marco Davids (SIDN) via Linuxptp-
> users wrote:
> > I have a setup with an interface being a PTP client (HW timestamping, a nice
> > Meinberg GM and such) and phc2sys supplying time from it to another
> > interface's PHC in the same server.
> >
> > Is there anything to be said about the accuracy/precision on the second
> > interface's PHC with this approach?
> 
> The accuracy and stability is impacted by asymmetry and jitter of the
> PCIe latency. Half of the delay printed by phc2sys gives you an upper
> bound, assuming in one direction the latency is zero and the other
> direction taking all the delay, similarly to the root delay in NTP.
> 
> There is a PCIe feature supported on some HW called Precision Time
> Measurement (PTM) that uses HW timestamping of PCIe messages and
> should at least in theory significantly improve the accuracy and
> stability, but the support in the kernel doesn't seem to be finished
> yet.
> 

The challenge here is that PCIe PTM also requires device support.

I know some variants of the ice hardware support this, but typically only the ones that are built onto motherboards.

> If you need best performance, you should synchronize the PHC through
> the system clock. That is, instead of
> 
> phc2sys -s eth0 -c eth1
> 
> use
> 
> phc2sys -s eth0 -c CLOCK_REALTIME
> phc2sys -s CLOCK_REALTIME -c eth1
> 
> Ideally, the NICs should be the same model. That will cancel out
> asymmetries in the HW timestamping errors, which can be larger than
> the PCIe asymmetry.
> 

This approach allows you to take advantage of PCIe PTM as well, if its supported. The other approach does not allow any use of PCIe PTM since its directly tied to using the platform clock.

> --
> Miroslav Lichvar
> 
> 
> 
> _______________________________________________
> Linuxptp-users mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linuxptp-users

Re: [Linuxptp-users] Building a PTP grandmaster with disciplined oscillator

[Andre P. <and...@sr...>]

Hi Axel,

On 10/3/22 21:23, Axel Simon wrote:
> Your subject has "disciplined oscillator" in it. That sounds like you want ptp4l to carry the time forward while there is no GPS reception. Thus, it seems that you have a setup where your GPS receive does not have a lock to GPS at all times.

Not quite. But perhaps my explanation wasn't sufficient. What I have is 
an external device that is GPS synced and has 10MHz/1PPS output. Even if 
it's not GPS locked, the oscillator is much better and so the 1PPS 
output will be stable.

What I am actually interested in is using the 1 PPS reference and 
discipline the NIC's oscillator.

I am still not quite sure to which extend this would improve the 
timestamp accuracy.


> We found that ptp4l can only synchronize NIC clocks (e.g. i210) to an accuracy of ~2µs due to the latency PCIe transactions. If you want better synchronization, ptp4l with off-the-shelf NICs is not a solution.

Interesting. For my application (telecom) I definitely need accuracy in 
the range of 1.5us.

In your statement, are you referring to off-the-shelf NICs acting as 
slaves or master?
Also, are you referring to unmodified off-the-shelf NICs or modified, 
i.e. disciplined or equipped with higher quality oscillators?

Thanks
Andre

> 
> Cheers,
> Axel
> 
>>
>>> However, making a reliable GM product is quite a bit of work.  The PTP
>>> software stack is only one piece of the puzzle.
>>
>> Yes of course. This effort is just to develop a poor man's GM as a PoC.
>>
>> But nonetheless interesting to observe that there is practically no NIC out there that brings together all the pieces. Should be quite easy to do. Cards like the Intel XXV710-DA2T or E810-XXVDA4T with GPS input are hard to get and $$$.
>>
>> Thanks
>> Andre
>>
>>
>> _______________________________________________
>> Linuxptp-users mailing list
>> Lin...@li...
>> https://lists.sourceforge.net/lists/listinfo/linuxptp-users
> 


-- 
Andre Puschmann

Software Radio Systems (SRS)
https://www.srs.io
an...@sr...

PGP/GnuPG key: 0x204A85DFEA324D58
fingerprint: 3924 1C60 D52E 81A2 1F2E 0C9D 204A 85DF EA32 4D58

Re: [Linuxptp-users] Building a PTP grandmaster with disciplined oscillator

[Richard C. <ric...@gm...>]

On Thu, Mar 10, 2022 at 09:47:21PM +0100, Andre Puschmann wrote:

> > We found that ptp4l can only synchronize NIC clocks (e.g. i210) to an accuracy of ~2µs due to the latency PCIe transactions. If you want better synchronization, ptp4l with off-the-shelf NICs is not a solution.

FUD

For PCIe MACs, their PTP synchronization performance is NOT affected by PCIe latency.

> Interesting. For my application (telecom) I definitely need accuracy in the
> range of 1.5us.

Easy to get the i210 to within a few dozen nanoseconds without doing
anything special.

Thanks,
Richard

Re: [Linuxptp-users] Building a PTP grandmaster with disciplined oscillator

[Richard C. <ric...@gm...>]

On Fri, Mar 11, 2022 at 06:30:49AM -0800, Axel Simon wrote:

> Sorry, I checked again and I stand corrected. We saw some offsets
> before we were able to monitor our system properly and it's still in
> my head that we can't get better than 2µs of noise. We are
> synchronizing the PTP clocks between different NICs using phc2sys
> and connect two x86 systems.

Right, using phc2sys is not a great design for a GM.

Why?

because it uses SW time stamping.

But that is not the fault of the i210 card.

If you were to feed a 1 PPS from a GPS radio into the SDP of the i210,
for example, then you get your time error well below 1 usec.

Thanks,
Richard

Re: [Linuxptp-users] Building a PTP grandmaster with disciplined oscillator

[Andre P. <and...@sr...>]

Hey,

On 11/3/22 15:51, Richard Cochran wrote:
> If you were to feed a 1 PPS from a GPS radio into the SDP of the i210,
> for example, then you get your time error well below 1 usec.

That's exactly what this thread is about. However, I am still not fully 
understanding what's needed for that, especially the update mechanism 
for the TXSTMP register.

Are you using the HW, i.e. the TSSDP register or configure the SDP as 
input and update the value in the ISR?

For example, taking a PPS output (in my case 5V), level shifted to 3.3 
and feeding it to, e.g. SDP3, of the i210-T1. How should I configure the 
TSSDP register to let the PPS pulse sync the TS value?

Are there any driver patches needed or are all necessary knobs exposed 
over the /sys interface?

Thanks
Andre

Re: [Linuxptp-users] Building a PTP grandmaster with disciplined oscillator

[Richard C. <ric...@gm...>]

On Sun, Mar 13, 2022 at 07:01:28PM +0100, Andre Puschmann wrote:

> Are you using the HW, i.e. the TSSDP register or configure the SDP as input
> and update the value in the ISR?
> 
> For example, taking a PPS output (in my case 5V), level shifted to 3.3 and
> feeding it to, e.g. SDP3, of the i210-T1. How should I configure the TSSDP
> register to let the PPS pulse sync the TS value?

The ts2phc program does all of that already.  It uses standard APIs,
and so you can also roll your own, using ts2phc as a learning tool for
the APIs.

HTH,
Richard

Re: [Linuxptp-users] Building a PTP grandmaster with disciplined oscillator

[Axel S. <axe...@go...>]

Hi Richard,

> On Mar 11, 2022, at 6:51 AM, Richard Cochran <ric...@gm...> wrote:
> 
> On Fri, Mar 11, 2022 at 06:30:49AM -0800, Axel Simon wrote:
> 
>> Sorry, I checked again and I stand corrected. We saw some offsets
>> before we were able to monitor our system properly and it's still in
>> my head that we can't get better than 2µs of noise. We are
>> synchronizing the PTP clocks between different NICs using phc2sys
>> and connect two x86 systems.
> 
> Right, using phc2sys is not a great design for a GM.
> 
> Why?
> 
> because it uses SW time stamping.
> 
> But that is not the fault of the i210 card.
> 
> If you were to feed a 1 PPS from a GPS radio into the SDP of the i210,
> for example, then you get your time error well below 1 usec.
> 

Earlier you wrote:

> 
> For PCIe MACs, their PTP synchronization performance is NOT affected by PCIe latency.
> 
>> Interesting. For my application (telecom) I definitely need accuracy in the
>> range of 1.5us.
> 
> Easy to get the i210 to within a few dozen nanoseconds without doing
> anything special.

I find these statements a bit confusing.
What do you mean by "PTP synchronization performance" if it is not the ability to synchronize to other NICs?
"Not doing anything special" sounds to me like using phc2sys to synchronize various clocks in an x86 system. But then you say "phc2sys is not a great design for a" grandmaster. The only way to synchronize several devices with an accuracy of ~100ns using ptp4l seems to be
(a) using several NICs, but synchronizing them via some PPS wires or
(b) using a single NIC, and a PTP-enabled Ethernet switch
I don't think that (a) classifies as "not doing anything special" and (b) is probably not what people think of naturally when they plan  on using ptp4l as a boundary clock or master.

Is there something I'm missing here? Is there any documentation or recommendation regarding the design of a high-accuracy linuxptp-based system?

Cheers,
Axel

Re: [Linuxptp-users] Building a PTP grandmaster with disciplined oscillator

[Richard C. <ric...@gm...>]

On Tue, Mar 15, 2022 at 06:11:26AM -0700, Axel Simon wrote:

> Is there something I'm missing here? Is there any documentation or recommendation regarding the design of a high-accuracy linuxptp-based system?

No, you stated the facts well enough.

If you want to make a single port GM, then you might want to provide
it with a global time source.  Let's assume you are using the i210.
You have some choices.

1. Configure the i210 as --free_running=1 (not a global time source at
   all.  But it might be okay to synchronize a "time island" depending
   on your application requirements).

2. Run NTP client on the Linux host, and use phc2sys to sync PHC with
   Linux system time.  Might end up milliseconds offset from global
   time due to NTP performance alone.

3. Attach GPS PPS to, say, Linux serial port, and use phc2sys to sync
   PHC with Linux system time.  Not a great solution, because of poor
   time stamps from the Linux kernel PPS subsystem.

4. Attach GPS PPS to i210 SDP.  This is the best solution.  With a
   healthy GPS fix, your GM will be easily within a few hundred
   nanoseconds of global time.  That is without any special tuning.

Now, if your GM has more than one port, then of course you will want
to use design #4 and feed the PPS to i210 cards in parallel.
(Alternately you could feed PPS into one card, then generate a PPS
from that card to the others.)

It is your design.  You have your application requirements.  You can
make whatever engineering decisions you wish.

But please don't spread FUD about linuxptp.

Thanks,
Richard