linuxptp-users — Open discussion on using the linuxptp software.

Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC

[Andre P. <and...@sr...>]

Hi,

this morning I tried with the Mellanox OFED drivers v4.9-7.1.0 on Ubuntu 
20.04 LTS. This was the last version I could get the Mellanox drivers 
compiled. Result, however, is the same:

ptp4l[283.357]: selected /dev/ptp0 as PTP clock
ptp4l[283.408]: port 1 (enp1s0): INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[283.408]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on 
INIT_COMPLETE
ptp4l[283.408]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on 
INIT_COMPLETE
ptp4l[283.505]: port 1 (enp1s0): new foreign master fcaf6a.fffe.02b447-1
ptp4l[283.758]: selected best master clock fcaf6a.fffe.02b447
ptp4l[283.758]: port 1 (enp1s0): LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[284.536]: port 1 (enp1s0): minimum delay request interval 2^-4
ptp4l[284.946]: port 1 (enp1s0): UNCALIBRATED to SLAVE on 
MASTER_CLOCK_SELECTED
ptp4l[285.508]: rms 23255986572 max 37976909340 freq +182135 +/- 122816 
delay -1439 +/- 1952
ptp4l[286.506]: rms 1677 max 6157 freq +267356 +/- 4283 delay   892 +/-  20
ptp4l[287.504]: rms 1590 max 6255 freq +268556 +/- 4011 delay   922 +/-   6
ptp4l[288.502]: rms  161 max  228 freq +268326 +/- 308 delay   936 +/-   7
ptp4l[289.500]: rms  208 max  231 freq +268746 +/-  29 delay   947 +/-   2
ptp4l[290.498]: rms 1570 max 6114 freq +268897 +/- 3989 delay   940 +/-  10
ptp4l[291.496]: rms   53 max   68 freq +268598 +/-  80 delay   944 +/-   3
ptp4l[292.502]: rms 1600 max 6172 freq +269013 +/- 4074 delay   947 +/-   4
ptp4l[293.507]: rms   99 max  235 freq +268431 +/- 173 delay   940 +/-   3
ptp4l[294.512]: rms   33 max   50 freq +268684 +/-  20 delay   946 +/-   2

Btw I am using a HP-branded MCX312B with FW version 2.42.5044


Thanks
Andre


On 19/11/23 22:07, Andre Puschmann wrote:
> Hey,
> 
> I've been able to get my hands on a ConnectX-3 Pro card and have done 
> some initial testing. The card indeed has a shared PHC for both ports so 
> running ptp4l as BC or TC does indeed work without the jbod option.
> 
> However, sync performance (i.e. rms values) for the downstream OCs isn't 
> great. And in fact, even the Mellanox as a OC isn't giving great results 
> - rms values jump a lot (and I've tried various PI value combinations).
> 
> Is anyone else seeing this with Mlx cards as well? Could it be my model 
> or the firmware?
> 
> Here is the output of a OC config with the card:
> 
> $ sudo /opt/linuxptp/ptp4l -i enp1s0 -f ~/configs/ptp/oc.cfg -m -l6
> ptp4l[12737.960]: selected /dev/ptp0 as PTP clock
> ptp4l[12738.012]: port 1 (enp1s0): INITIALIZING to LISTENING on 
> INIT_COMPLETE
> ptp4l[12738.012]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on 
> INIT_COMPLETE
> ptp4l[12738.012]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING 
> on INIT_COMPLETE
> ptp4l[12738.060]: port 1 (enp1s0): new foreign master fcaf6a.fffe.02b447-1
> ptp4l[12738.314]: selected best master clock fcaf6a.fffe.02b447
> ptp4l[12738.314]: port 1 (enp1s0): LISTENING to UNCALIBRATED on RS_SLAVE
> ptp4l[12740.148]: port 1 (enp1s0): minimum delay request interval 2^-4
> ptp4l[12740.512]: port 1 (enp1s0): UNCALIBRATED to SLAVE on 
> MASTER_CLOCK_SELECTED
> ptp4l[12741.138]: rms 1450 max 1934 freq +270168 +/- 1641 delay   951 
> +/-  14
> ptp4l[12742.139]: rms  129 max  179 freq +268843 +/- 296 delay   963 
> +/-  11
> ptp4l[12743.140]: rms  241 max  490 freq +268455 +/- 452 delay   948 
> +/-   1
> ptp4l[12744.141]: rms  135 max  180 freq +268381 +/-  25 delay   947 
> +/-   1
> ptp4l[12745.142]: rms 1357 max 5277 freq +269064 +/- 3459 delay   950 
> +/-   1
> ptp4l[12746.143]: rms 1397 max 5092 freq +268197 +/- 3539 delay   935 
> +/-   7
> ptp4l[12747.144]: rms  210 max  417 freq +268048 +/- 243 delay   942 
> +/-   3
> ptp4l[12748.145]: rms   15 max   32 freq +268415 +/-  29 delay   947 
> +/-   2
> ptp4l[12749.146]: rms 1430 max 5594 freq +269126 +/- 3617 delay   950 
> +/-   1
> ptp4l[12750.147]: rms 1391 max 5162 freq +268252 +/- 3543 delay   942 
> +/-   4
> 
> 
> Thanks
> Andre
> 
> 
> 
> 
> 
> On 2/11/23 17:37, Jacob Keller wrote:
>>
>>
>> On 11/2/2023 4:15 AM, Andre Puschmann wrote:
>>> Hi,
>>>
>>> On 2/11/23 4:11, James Clark wrote:
>>>> I have a dual-port Mellanox ConnectX-3 (specifically MCX312A-XCBT),
>>>> which has a shared PHC. You can get them for less than $50 on
>>>> eBay/AliExpress. I had to upgrade the firmware on mine to get PTP
>>>> support. I haven't yet tried it as a boundary clock.
>>>
>>> Excellent. This is very helpful James. I've ordered a MCX312A and B and
>>> will compare both here. I'll share my results here soon. If you have a
>>> chance please also share the firmware version you're currently using on
>>> your NIC.
>>>
>>> With my Intel NIC I could get the BC config working but I needed to set
>>> the twoStepFlag to 1. Otherwise I was getting this for both ports:
>>>
>>> ptp4l[1040.180]: ioctl SIOCSHWTSTAMP failed: Numerical result out of 
>>> range
>>>
>>
>> Yep, that would indicate the device doesn't support one-step mode.
>>
>>> Sync quality wasn't great as expected though. I'll repeat with the
>>> Mellanox once I have them here.
>>>
>>> Thanks
>>> Andre
>>>
>>
>> For Intel NICs, the only products I am aware of which share PHC across
>> the device are the E800 series devices. Prior devices (E500, and E700,
>> as well as the gigabit products) do share the same internal oscillator
>> but due to the register interface each function has to setup its own 
>> clock.
>>
>> Thanks,
>> Jake
>>
>>
>> _______________________________________________
>> Linuxptp-users mailing list
>> Lin...@li...
>> https://lists.sourceforge.net/lists/listinfo/linuxptp-users
> 

-- 
Andre Puschmann

Software Radio Systems (SRS)
https://www.srs.io
an...@sr...

PGP/GnuPG key: 0x204A85DFEA324D58
fingerprint: 3924 1C60 D52E 81A2 1F2E 0C9D 204A 85DF EA32 4D58

Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC

[Miroslav L. <mli...@re...>]

On Sun, Nov 19, 2023 at 10:07:50PM +0100, Andre Puschmann wrote:
> Hey,
> 
> I've been able to get my hands on a ConnectX-3 Pro card and have done some
> initial testing. The card indeed has a shared PHC for both ports so running
> ptp4l as BC or TC does indeed work without the jbod option.
> 
> However, sync performance (i.e. rms values) for the downstream OCs isn't
> great. And in fact, even the Mellanox as a OC isn't giving great results -
> rms values jump a lot (and I've tried various PI value combinations).
> 
> Is anyone else seeing this with Mlx cards as well? Could it be my model or
> the firmware?

Try the L2 transport. IIRC at least some Mellanox NICs performed
worse with UDP transport for some reason.

-- 
Miroslav Lichvar

Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC

[egg c. <egg...@gm...>]

Hi,

How the GM side is configured? Are you writing system time to PHC
every second? If so, you can try make the phc free run. Without 1PPS
signal connecting to the phc or PTM enabled, it's not recommended to
set pmc's time by software, the jitter is quite big.

Is the GM and the client connected directly or through a switch? Try
connect them directly with an utp or fiber.

Andre Puschmann <and...@sr...> 于2023年11月20日周一 05:21写道：
>
> Hey,
>
> I've been able to get my hands on a ConnectX-3 Pro card and have done
> some initial testing. The card indeed has a shared PHC for both ports so
> running ptp4l as BC or TC does indeed work without the jbod option.
>
> However, sync performance (i.e. rms values) for the downstream OCs isn't
> great. And in fact, even the Mellanox as a OC isn't giving great results
> - rms values jump a lot (and I've tried various PI value combinations).
>
> Is anyone else seeing this with Mlx cards as well? Could it be my model
> or the firmware?
>
> Here is the output of a OC config with the card:
>
> $ sudo /opt/linuxptp/ptp4l -i enp1s0 -f ~/configs/ptp/oc.cfg -m -l6
> ptp4l[12737.960]: selected /dev/ptp0 as PTP clock
> ptp4l[12738.012]: port 1 (enp1s0): INITIALIZING to LISTENING on
> INIT_COMPLETE
> ptp4l[12738.012]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on
> INIT_COMPLETE
> ptp4l[12738.012]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING
> on INIT_COMPLETE
> ptp4l[12738.060]: port 1 (enp1s0): new foreign master fcaf6a.fffe.02b447-1
> ptp4l[12738.314]: selected best master clock fcaf6a.fffe.02b447
> ptp4l[12738.314]: port 1 (enp1s0): LISTENING to UNCALIBRATED on RS_SLAVE
> ptp4l[12740.148]: port 1 (enp1s0): minimum delay request interval 2^-4
> ptp4l[12740.512]: port 1 (enp1s0): UNCALIBRATED to SLAVE on
> MASTER_CLOCK_SELECTED
> ptp4l[12741.138]: rms 1450 max 1934 freq +270168 +/- 1641 delay   951
> +/-  14
> ptp4l[12742.139]: rms  129 max  179 freq +268843 +/- 296 delay   963 +/-  11
> ptp4l[12743.140]: rms  241 max  490 freq +268455 +/- 452 delay   948 +/-   1
> ptp4l[12744.141]: rms  135 max  180 freq +268381 +/-  25 delay   947 +/-   1
> ptp4l[12745.142]: rms 1357 max 5277 freq +269064 +/- 3459 delay   950
> +/-   1
> ptp4l[12746.143]: rms 1397 max 5092 freq +268197 +/- 3539 delay   935
> +/-   7
> ptp4l[12747.144]: rms  210 max  417 freq +268048 +/- 243 delay   942 +/-   3
> ptp4l[12748.145]: rms   15 max   32 freq +268415 +/-  29 delay   947 +/-   2
> ptp4l[12749.146]: rms 1430 max 5594 freq +269126 +/- 3617 delay   950
> +/-   1
> ptp4l[12750.147]: rms 1391 max 5162 freq +268252 +/- 3543 delay   942
> +/-   4
>
>
> Thanks
> Andre
>
>
>
>
>
> On 2/11/23 17:37, Jacob Keller wrote:
> >
> >
> > On 11/2/2023 4:15 AM, Andre Puschmann wrote:
> >> Hi,
> >>
> >> On 2/11/23 4:11, James Clark wrote:
> >>> I have a dual-port Mellanox ConnectX-3 (specifically MCX312A-XCBT),
> >>> which has a shared PHC. You can get them for less than $50 on
> >>> eBay/AliExpress. I had to upgrade the firmware on mine to get PTP
> >>> support. I haven't yet tried it as a boundary clock.
> >>
> >> Excellent. This is very helpful James. I've ordered a MCX312A and B and
> >> will compare both here. I'll share my results here soon. If you have a
> >> chance please also share the firmware version you're currently using on
> >> your NIC.
> >>
> >> With my Intel NIC I could get the BC config working but I needed to set
> >> the twoStepFlag to 1. Otherwise I was getting this for both ports:
> >>
> >> ptp4l[1040.180]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range
> >>
> >
> > Yep, that would indicate the device doesn't support one-step mode.
> >
> >> Sync quality wasn't great as expected though. I'll repeat with the
> >> Mellanox once I have them here.
> >>
> >> Thanks
> >> Andre
> >>
> >
> > For Intel NICs, the only products I am aware of which share PHC across
> > the device are the E800 series devices. Prior devices (E500, and E700,
> > as well as the gigabit products) do share the same internal oscillator
> > but due to the register interface each function has to setup its own clock.
> >
> > Thanks,
> > Jake
> >
> >
> > _______________________________________________
> > Linuxptp-users mailing list
> > Lin...@li...
> > https://lists.sourceforge.net/lists/listinfo/linuxptp-users
>
> --
> Andre Puschmann
>
> Software Radio Systems (SRS)
> https://www.srs.io
> an...@sr...
>
> PGP/GnuPG key: 0x204A85DFEA324D58
> fingerprint: 3924 1C60 D52E 81A2 1F2E 0C9D 204A 85DF EA32 4D58
>
>
>
> _______________________________________________
> Linuxptp-users mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linuxptp-users

Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC

[Andre P. <and...@sr...>]

Hey,

I've been able to get my hands on a ConnectX-3 Pro card and have done 
some initial testing. The card indeed has a shared PHC for both ports so 
running ptp4l as BC or TC does indeed work without the jbod option.

However, sync performance (i.e. rms values) for the downstream OCs isn't 
great. And in fact, even the Mellanox as a OC isn't giving great results 
- rms values jump a lot (and I've tried various PI value combinations).

Is anyone else seeing this with Mlx cards as well? Could it be my model 
or the firmware?

Here is the output of a OC config with the card:

$ sudo /opt/linuxptp/ptp4l -i enp1s0 -f ~/configs/ptp/oc.cfg -m -l6
ptp4l[12737.960]: selected /dev/ptp0 as PTP clock
ptp4l[12738.012]: port 1 (enp1s0): INITIALIZING to LISTENING on 
INIT_COMPLETE
ptp4l[12738.012]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on 
INIT_COMPLETE
ptp4l[12738.012]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING 
on INIT_COMPLETE
ptp4l[12738.060]: port 1 (enp1s0): new foreign master fcaf6a.fffe.02b447-1
ptp4l[12738.314]: selected best master clock fcaf6a.fffe.02b447
ptp4l[12738.314]: port 1 (enp1s0): LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[12740.148]: port 1 (enp1s0): minimum delay request interval 2^-4
ptp4l[12740.512]: port 1 (enp1s0): UNCALIBRATED to SLAVE on 
MASTER_CLOCK_SELECTED
ptp4l[12741.138]: rms 1450 max 1934 freq +270168 +/- 1641 delay   951 
+/-  14
ptp4l[12742.139]: rms  129 max  179 freq +268843 +/- 296 delay   963 +/-  11
ptp4l[12743.140]: rms  241 max  490 freq +268455 +/- 452 delay   948 +/-   1
ptp4l[12744.141]: rms  135 max  180 freq +268381 +/-  25 delay   947 +/-   1
ptp4l[12745.142]: rms 1357 max 5277 freq +269064 +/- 3459 delay   950 
+/-   1
ptp4l[12746.143]: rms 1397 max 5092 freq +268197 +/- 3539 delay   935 
+/-   7
ptp4l[12747.144]: rms  210 max  417 freq +268048 +/- 243 delay   942 +/-   3
ptp4l[12748.145]: rms   15 max   32 freq +268415 +/-  29 delay   947 +/-   2
ptp4l[12749.146]: rms 1430 max 5594 freq +269126 +/- 3617 delay   950 
+/-   1
ptp4l[12750.147]: rms 1391 max 5162 freq +268252 +/- 3543 delay   942 
+/-   4


Thanks
Andre





On 2/11/23 17:37, Jacob Keller wrote:
> 
> 
> On 11/2/2023 4:15 AM, Andre Puschmann wrote:
>> Hi,
>>
>> On 2/11/23 4:11, James Clark wrote:
>>> I have a dual-port Mellanox ConnectX-3 (specifically MCX312A-XCBT),
>>> which has a shared PHC. You can get them for less than $50 on
>>> eBay/AliExpress. I had to upgrade the firmware on mine to get PTP
>>> support. I haven't yet tried it as a boundary clock.
>>
>> Excellent. This is very helpful James. I've ordered a MCX312A and B and
>> will compare both here. I'll share my results here soon. If you have a
>> chance please also share the firmware version you're currently using on
>> your NIC.
>>
>> With my Intel NIC I could get the BC config working but I needed to set
>> the twoStepFlag to 1. Otherwise I was getting this for both ports:
>>
>> ptp4l[1040.180]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range
>>
> 
> Yep, that would indicate the device doesn't support one-step mode.
> 
>> Sync quality wasn't great as expected though. I'll repeat with the
>> Mellanox once I have them here.
>>
>> Thanks
>> Andre
>>
> 
> For Intel NICs, the only products I am aware of which share PHC across
> the device are the E800 series devices. Prior devices (E500, and E700,
> as well as the gigabit products) do share the same internal oscillator
> but due to the register interface each function has to setup its own clock.
> 
> Thanks,
> Jake
> 
> 
> _______________________________________________
> Linuxptp-users mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linuxptp-users

-- 
Andre Puschmann

Software Radio Systems (SRS)
https://www.srs.io
an...@sr...

PGP/GnuPG key: 0x204A85DFEA324D58
fingerprint: 3924 1C60 D52E 81A2 1F2E 0C9D 204A 85DF EA32 4D58

Re: [Linuxptp-users] How to use common (v)clock for physically different PHCs?

[Vladimir O. <ol...@gm...>]

On Mon, Nov 13, 2023 at 10:43:01AM +0000, Osterried Markus (ETAS-DAP/XPC-Fe3) wrote:
> Hi Vladimir,
> 
> thanks for your answer.
> 
> We do not have a connection between EC1_1588_PULSE_OUT1 and SWITCH_1588_DAT0 on our board, so this option is not possible for us.
> 
> But we use this trigger signal, which actually is EC1_1588_TRIG_IN1 and SWITCH_1588_DAT1. Both pins are connected together to a GPIO pin.
> In ES_DEVCPU_PTP_PTP_PINS_PTP_PIN_CFG register, SWITCH_1588_DAT1 is configured for STORE action.
> Generating an edge on these trigger signals is latching a timestamp of the ENETC (aka /dev/ptp0) and SWITCH (aka /dev/ptp1).
> As the input clocks of ENETC and SWITCH is derived from the same clock source, the PHCs always run at the same speed (with period configured in ENETC TMR_CTRL and ES_DEVCPU_PTP_PTP_CFG_PTP_SYS_CLK_CFG).

Have you studied the clock distribution trees for the ENETC and the
Felix switch IPs before drawing the conclusion that their input clocks
are derived from the same source? As far as I see, ENETC, through
HWA_CGA_M4_CLK_SEL, uses Cluster Group A PLL 2 divided by 3 as clock
(400 MHz), whereas Felix has its sys_clk at 156.25 MHz and it comes from
one of the SerDes PLLs, depending on the particular RCW configuration.

> When latching the timestamps of ENETC and SWITCH, we can calculate the offset between them, which in turn we use to clock_adjtime(ADJ_SETOFFSET|ADJ_NANO) one PHC to the other.
> Doing this in the startup phase, we can guarantee that /dev/ptp0 and /dev/ptp1 is synchronized, but yes, PHC subsystem does not know that they are now actually equivalent.

Aha, so you've implemented the extts API in the ocelot ptp driver?
Do you plan to upstream that work?

A pulse output from the ENETC PHC which is an input to the Felix PHC is
not strictly necessary. Your way should be just fine - a GPIO output
which is input both to ENETC and to Felix is equally fine, and ts2phc
should be able to work with that. I guess you're looking at a "generic"
source of 1-PPS signal, when you open up the ts2phc manual.

> When later ptp4l wants to adjust the PHC according to an external PTP master, it will do this only on one of the two PHCs, therefore they will be no longer in sync.

This is a similar reason to the one for which I modified ts2phc to
monitor the direction of ptp4l's synchronization dynamically. If the
slave port of the ptp4l BC is an ENETC port, then the Felix PHC gets
synchronized to the ENETC PHC, otherwise the ENETC PHC gets synchronized
to the Felix PHC.
https://sourceforge.net/p/linuxptp/mailman/linuxptp-devel/thread/20200830234525.3311891-11-olteanv%40gmail.com/

I've only tried it with the "PHC" kind of sources of 1-PPS signals.
If it doesn't work with a "generic" source, it probably isn't too hard
to modify ts2phc to treat that case reasonably.

> To avoid this, the idea was to create a vclock /dev/ptp2 based on /dev/ptp0. And to let /dev/ptp0 and /dev/ptp1 free running and in sync.
> But because PHC subsystem does not know that /dev/ptp0 and /dev/ptp1 is actually equivalent, it forbids to bind /dev/ptp2 to the socket of swpX.

It's not that the PHC subsystem doesn't know that /dev/ptp0 and
/dev/ptp1 are "actually equivalent". They _aren't_ actually equivalent.
They are completely independent clocks, independently adjustable.
Synchronized once and then left alone, I don't see what will prevent
them from starting to drift. The sync process needs to be continuous,
like for any other independent hardware clocks.

That, plus the fact that vclocks don't do what you think they do.
Even if there were 2 PTP clocks which could be proven to be equivalent,
a vclock only has a single PTP clock as parent, not 2.

> If I understand your suggested options correctly, the ptp4l instance which handles the external front-facing interfaces, will just adjust one physical PHC, while the other PHC is re-adjusted by either internal ptp4l instance or ts2phc, but of course within some latency. Is this correct?

Yes.

> Back to our original problem:
> We want to use ptp4l between all front-facing ports, while physical PHCs of eno0-eno3 and swp0-swp5 are already in sync - known by us, but not known by PHC subsystem.
> For this, you have mentioned boundary_clock_jbod can be used.
> I do not know what boundary_clock_jbod is doing in detail and if and how this could be used to provide a solution in this use case.
> Could you please explain.

man ptp4l

       boundary_clock_jbod
              When running as a boundary clock (that is, when more than
              one network interface is configured), ptp4l performs a
              sanity check to make sure that all of the ports share the
              same  hardware clock  device. This option allows ptp4l to
              work as a boundary clock using "just a bunch of devices"
              that are not synchronized to each other. For this mode,
              the collection of clocks must be synchronized by an
              external program, for example phc2sys(8) in "automatic"
              mode.  The default is 0 (disabled).

"physical PHCs of eno0-eno3 and swp0-swp5 are already in sync - known by
us, but not known by PHC subsystem" -> TL;DR: this option tells ptp4l
"trust me, the clocks are in sync, within an unknown tolerance, despite
being different, you can make a boundary clock out of these ports".
They'd better be in robust sync, though, because ptp4l can make an
attempt to adjust any of those PHCs.

Re: [Linuxptp-users] Max number of PTP clients on a BC port ?

[Richard C. <ric...@gm...>]

On Mon, Nov 13, 2023 at 06:41:49AM +0100, Surya Arby wrote:
> Thanks for your quick reply!
> 
> Would you be ok to tell me which commercial switches are very slow to
> process ptp in the transparent mode ?

I've made it my policy not to shame bad products, even though there
are tons of them on the market.

My advice is, try before you buy!
 
> Our boundary clocks in each technical room will be based on mellanox
> hardware (but not mellanox branded, it's OEM provided by our partner) with
> one single Netgear 4350 attached to it, with a few devices connected to it
> audio devices, probably less than 20 clients)

ptp4l will easily scale to 20 clients.

Thanks,
Richard

Re: [Linuxptp-users] How to use common (v)clock for physically different PHCs?

[Osterried M. (ETAS-DAP/XPC-Fe3) <mar...@et...>]

Hi Vladimir,

thanks for your answer.

We do not have a connection between EC1_1588_PULSE_OUT1 and SWITCH_1588_DAT0 on our board, so this option is not possible for us.

But we use this trigger signal, which actually is EC1_1588_TRIG_IN1 and SWITCH_1588_DAT1. Both pins are connected together to a GPIO pin.
In ES_DEVCPU_PTP_PTP_PINS_PTP_PIN_CFG register, SWITCH_1588_DAT1 is configured for STORE action.
Generating an edge on these trigger signals is latching a timestamp of the ENETC (aka /dev/ptp0) and SWITCH (aka /dev/ptp1).
As the input clocks of ENETC and SWITCH is derived from the same clock source, the PHCs always run at the same speed (with period configured in ENETC TMR_CTRL and ES_DEVCPU_PTP_PTP_CFG_PTP_SYS_CLK_CFG).
When latching the timestamps of ENETC and SWITCH, we can calculate the offset between them, which in turn we use to clock_adjtime(ADJ_SETOFFSET|ADJ_NANO) one PHC to the other.
Doing this in the startup phase, we can guarantee that /dev/ptp0 and /dev/ptp1 is synchronized, but yes, PHC subsystem does not know that they are now actually equivalent.

When later ptp4l wants to adjust the PHC according to an external PTP master, it will do this only on one of the two PHCs, therefore they will be no longer in sync.
To avoid this, the idea was to create a vclock /dev/ptp2 based on /dev/ptp0. And to let /dev/ptp0 and /dev/ptp1 free running and in sync.
But because PHC subsystem does not know that /dev/ptp0 and /dev/ptp1 is actually equivalent, it forbids to bind /dev/ptp2 to the socket of swpX.

If I understand your suggested options correctly, the ptp4l instance which handles the external front-facing interfaces, will just adjust one physical PHC, while the other PHC is re-adjusted by either internal ptp4l instance or ts2phc, but of course within some latency. Is this correct?

Back to our original problem:
We want to use ptp4l between all front-facing ports, while physical PHCs of eno0-eno3 and swp0-swp5 are already in sync - known by us, but not known by PHC subsystem.
For this, you have mentioned boundary_clock_jbod can be used.
I do not know what boundary_clock_jbod is doing in detail and if and how this could be used to provide a solution in this use case.
Could you please explain.


Many thanks and regards
Markus







Hello Markus,

On Wed, Nov 08, 2023 at 04:58:54PM +0000, Osterried Markus (ETAS-DAP/XPC-Fe3) via Linuxptp-users wrote:
> Hi,
> 
> I am using a custom board with NXP LS1027A SoC which has an integrated 
> ethernet controller (which provides one front facing port, eno0) and 
> an integrated ethernet switch (which provides four front facing ports, 
> swp0-swp3).
> 
> I want to implement an IEEE1588 boundary clock supporting all five 
> ports. The ethernet controller has attached a different PHC as the 
> ports of the switch has, /dev/ptp0 resp. /dev/ptp1.
> 
> Nevertheless, with hardware trigger signal, I can ensure that both 
> clocks start at the same time and runs with same frequency, so they 
> will always have the same value (within granularity of about 5 ns) - 
> as long as they are free running, because frequency adjustments and 
> time jumps can not be done atomically for both clocks.
> 
> The idea was to create a virtual PHC (vclock, /dev/ptp2), based on 
> either one of the two physical PHCs.
> 
> But while ptp4l prints this info
> port 1 (eno0): /dev/ptp2 is virtual clock port 2 (swp0): taking 
> /dev/ptp2 from the command line, not the attached ptp1
> 
> it seems that swp0 still uses the PHC /dev/ptp1 and frames send on 
> this port contains timestamps derived from /dev/ptp1, while eno0 uses 
> /dev/ptp2, so timestamps on both ports are quite different.
> 
> I do not really understand what ptp4l is doing when p->phc_index and
> p->phc_from_cmdline is set - but it seems it is not as I have
> expected.
> 
> But I have also recognized that setsockopt(fd, SOL_SOCKET, 
> SO_TIMESTAMPING, ...) fails when used with SOF_TIMESTAMPING_BIND_PHC 
> when vclock is not derived from natively attached PHC.
> 
> What would be the best solution to use a common (v)clock for all the 
> interfaces?
> I want to avoid to synchronize the physical PHCs with phc2sys, because 
> that would be less accurate then using hardware trigger signal which 
> keeps (free running) PHCs on the same value.

Virtual clocks (which btw can have a single parent, not two as you seem to expect) don't work that way. The physical MACs always take timestamps in their physical PHC's time domain, and depending on which socket should receive the timestamped packets, the physical timestamps are translated to the time base of the vclock associated with that socket.
You cannot make the switch port MACs take timestamps that are translated into a time domain associated with a vclock of the ENETC. They don't seem to help you here. They are used if you want multiple ptp4l instances, associated with multiple domains, on top of the same Ethernet port.

If I understand your use case correctly, you are looking for ways to keep the /dev/ptp0 clock (shared by eno0, eno1, eno2, eno3) in sync with the /dev/ptp1 clock (shared by swp0, swp1, swp2, swp3, swp4, swp5), so that you can run ptp4l with the boundary_clock_jbod option between all front-facing Ethernet ports.

There are 2 ways in which you can do that.

First, here's a diagram of the Ethernet system on the LS1028A family, for the viewers not familiar with it:

     +-------------------------+------------------------------------------------+
     |             +--------+  |                                 LS1028A        |
     |             |  eno3  |--|-----------------------------+                  |
     |             +--------+  | internal port (1G)          |                  |
     |                         |                             |                  |
     |             +--------+  | internal port (2.5G)        |                  |
     |             |  eno2  |--|-----------------+           |                  |
     |             +--------+  |                 |           |                  |
     |                         +------------------------------------------------+
     |         ENETC           |                 |           |                  |
     |                         |   Felix     +--------+  +--------+  /dev/ptp1  |
     |       /dev/ptp0         |  switch     |  swp4  |  |  swp5  |             |
     |                         |             +--------+  +--------+             |
     |                         |                                                |
     | +--------+  +--------+  | +--------+  +--------+  +--------+  +--------+ |
     | |  eno0  |  |  eno1  |  | |  swp0  |  |  swp1  |  |  swp2  |  |  swp3  | |
     +-------------------------+------------------------------------------------+
            |           |             |           |          |            |
         SerDes       RGMII        SerDes      SerDes     SerDes        SerDes

The first option is to run a pair of ptp4l instances on a pair of Ethernet ports internal to the SoC: one on eno2 and another on swp4.
These internal Ethernet ports are capable of hardware timestamping, so the PHC of eno2 (aka /dev/ptp0) will be kept in sync within very low tolerances to the PHC of swp3 (aka /dev/ptp1).

The second option needs collaboration from your board. If you set the RCW field EC1_SAI4_5_PMUX to the value of 0b101, you make the pinmuxing of the SoC disable the eno1 RGMII port, and you make those pins available for the auxiliary PTP pins of /dev/ptp0 and /dev/ptp1.
The idea here is to connect (externally to the SoC) the EC1_1588_PULSE_OUT1 pin of /dev/ptp0 to the SWITCH_1588_DAT[0] pin of /dev/ptp1. Then you use the ts2phc program to keep in sync the 2 hardware clocks. This is way more precise than phc2sys and comparable to the ptp4l option, because the PPS signal is timestamped in hardware using the extts functionality of the Felix PHC. One small note is that I never had access to a board with the PPS layout I'm proposing, but based on what I know, it should work.

There is no option to connect a PPS signal from one PHC to the other internally to the SoC, unfortunately.

By the way, can you tell more about that hardware trigger signal? I am familiar with the LS1028A family but I don't know what you are talking about.

Re: [Linuxptp-users] Max number of PTP clients on a BC port ?

[Surya A. <sur...@gm...>]

Thanks for your quick reply!

Would you be ok to tell me which commercial switches are very slow to
process ptp in the transparent mode ?

Our boundary clocks in each technical room will be based on mellanox
hardware (but not mellanox branded, it's OEM provided by our partner) with
one single Netgear 4350 attached to it, with a few devices connected to it
audio devices, probably less than 20 clients)

Thanks

Surya


Le lun. 13 nov. 2023 à 02:40, Richard Cochran <ric...@gm...> a
écrit :

> On Sun, Nov 12, 2023 at 09:57:56PM +0100, Surya Arby wrote:
>
> > we are considering the use of linux-based switches (hardware timestamping
> > provided in the ASIC, linuxptp will be used), the GM will be connected to
> > the switch, and all ports will be BC,
>
> In PTP land, switches and Boundary Clocks are not the same (as you
> probably already know, but you seem to conflate the two in this mail)
>
> > I was wondering if there was any known
> > limit to the number of PTP clients behind a BC port in the code ? (an
> > intermediate PTP-transparent switch will be used to connect several
> clients
> > to the BC port)
>
> There are no hard coded limits in the code.  Because the program is
> single threaded, there is a practical upper limit somewhere.
>
> BC mode is essentially stateless WRT the number of clients, so that
> memory usage is not a concern.  The only limitation is the rate at
> which Delay Requests can be handled, as the responses are serialized
> by the single threaded ptp4l program.
>
> TC needs a bit of state for each message needing residence time
> correction, and so memory use will increase with the number of PTP
> clients.  Also, when ptp4l runs in TC mode, the frame residence time
> will likely be in the single digit millisecond range.  Reducing that
> might be challenging, but maybe nobody cares about the residence time
> duration?  (FWIW I've seen commercial Transparent Switches where the
> residence time is in the double digit millisecond range.)
>
> HTH,
> Richard
>
>
>

Re: [Linuxptp-users] Max number of PTP clients on a BC port ?

[Richard C. <ric...@gm...>]

On Sun, Nov 12, 2023 at 09:57:56PM +0100, Surya Arby wrote:

> we are considering the use of linux-based switches (hardware timestamping
> provided in the ASIC, linuxptp will be used), the GM will be connected to
> the switch, and all ports will be BC,

In PTP land, switches and Boundary Clocks are not the same (as you
probably already know, but you seem to conflate the two in this mail)

> I was wondering if there was any known
> limit to the number of PTP clients behind a BC port in the code ? (an
> intermediate PTP-transparent switch will be used to connect several clients
> to the BC port)

There are no hard coded limits in the code.  Because the program is
single threaded, there is a practical upper limit somewhere.

BC mode is essentially stateless WRT the number of clients, so that
memory usage is not a concern.  The only limitation is the rate at
which Delay Requests can be handled, as the responses are serialized
by the single threaded ptp4l program.

TC needs a bit of state for each message needing residence time
correction, and so memory use will increase with the number of PTP
clients.  Also, when ptp4l runs in TC mode, the frame residence time
will likely be in the single digit millisecond range.  Reducing that
might be challenging, but maybe nobody cares about the residence time
duration?  (FWIW I've seen commercial Transparent Switches where the
residence time is in the double digit millisecond range.)

HTH,
Richard

[Linuxptp-users] Max number of PTP clients on a BC port ?

[Surya A. <sur...@gm...>]

Hi linuxptp gurus !

I'm new on the list,

we are considering the use of linux-based switches (hardware 
timestamping provided in the ASIC, linuxptp will be used), the GM will 
be connected to the switch, and all ports will be BC, I was wondering if 
there was any known limit to the number of PTP clients behind a BC port 
in the code ? (an intermediate PTP-transparent switch will be used to 
connect several clients to the BC port)

Best Regards,

Surya

Re: [Linuxptp-users] How to use common (v)clock for physically different PHCs?

[Vladimir O. <ol...@gm...>]

Hello Markus,

On Wed, Nov 08, 2023 at 04:58:54PM +0000, Osterried Markus (ETAS-DAP/XPC-Fe3) via Linuxptp-users wrote:
> Hi,
> 
> I am using a custom board with NXP LS1027A SoC which has an integrated
> ethernet controller (which provides one front facing port, eno0) and
> an integrated ethernet switch (which provides four front facing ports,
> swp0-swp3).
> 
> I want to implement an IEEE1588 boundary clock supporting all five
> ports. The ethernet controller has attached a different PHC as the
> ports of the switch has, /dev/ptp0 resp. /dev/ptp1.
> 
> Nevertheless, with hardware trigger signal, I can ensure that both
> clocks start at the same time and runs with same frequency, so they
> will always have the same value (within granularity of about 5 ns) -
> as long as they are free running, because frequency adjustments and
> time jumps can not be done atomically for both clocks.
> 
> The idea was to create a virtual PHC (vclock, /dev/ptp2), based on
> either one of the two physical PHCs.
> 
> But while ptp4l prints this info
> port 1 (eno0): /dev/ptp2 is virtual clock
> port 2 (swp0): taking /dev/ptp2 from the command line, not the attached ptp1
> 
> it seems that swp0 still uses the PHC /dev/ptp1 and frames send on
> this port contains timestamps derived from /dev/ptp1, while eno0 uses
> /dev/ptp2, so timestamps on both ports are quite different.
> 
> I do not really understand what ptp4l is doing when p->phc_index and
> p->phc_from_cmdline is set - but it seems it is not as I have
> expected.
> 
> But I have also recognized that setsockopt(fd, SOL_SOCKET,
> SO_TIMESTAMPING, ...) fails when used with SOF_TIMESTAMPING_BIND_PHC
> when vclock is not derived from natively attached PHC.
> 
> What would be the best solution to use a common (v)clock for all the
> interfaces?
> I want to avoid to synchronize the physical PHCs with phc2sys, because
> that would be less accurate then using hardware trigger signal which
> keeps (free running) PHCs on the same value.

Virtual clocks (which btw can have a single parent, not two as you seem
to expect) don't work that way. The physical MACs always take timestamps
in their physical PHC's time domain, and depending on which socket
should receive the timestamped packets, the physical timestamps are
translated to the time base of the vclock associated with that socket.
You cannot make the switch port MACs take timestamps that are translated
into a time domain associated with a vclock of the ENETC. They don't
seem to help you here. They are used if you want multiple ptp4l instances,
associated with multiple domains, on top of the same Ethernet port.

If I understand your use case correctly, you are looking for ways to
keep the /dev/ptp0 clock (shared by eno0, eno1, eno2, eno3) in sync with
the /dev/ptp1 clock (shared by swp0, swp1, swp2, swp3, swp4, swp5), so
that you can run ptp4l with the boundary_clock_jbod option between all
front-facing Ethernet ports.

There are 2 ways in which you can do that.

First, here's a diagram of the Ethernet system on the LS1028A family,
for the viewers not familiar with it:

     +-------------------------+------------------------------------------------+
     |             +--------+  |                                 LS1028A        |
     |             |  eno3  |--|-----------------------------+                  |
     |             +--------+  | internal port (1G)          |                  |
     |                         |                             |                  |
     |             +--------+  | internal port (2.5G)        |                  |
     |             |  eno2  |--|-----------------+           |                  |
     |             +--------+  |                 |           |                  |
     |                         +------------------------------------------------+
     |         ENETC           |                 |           |                  |
     |                         |   Felix     +--------+  +--------+  /dev/ptp1  |
     |       /dev/ptp0         |  switch     |  swp4  |  |  swp5  |             |
     |                         |             +--------+  +--------+             |
     |                         |                                                |
     | +--------+  +--------+  | +--------+  +--------+  +--------+  +--------+ |
     | |  eno0  |  |  eno1  |  | |  swp0  |  |  swp1  |  |  swp2  |  |  swp3  | |
     +-------------------------+------------------------------------------------+
            |           |             |           |          |            |
         SerDes       RGMII        SerDes      SerDes     SerDes        SerDes

The first option is to run a pair of ptp4l instances on a pair of
Ethernet ports internal to the SoC: one on eno2 and another on swp4.
These internal Ethernet ports are capable of hardware timestamping,
so the PHC of eno2 (aka /dev/ptp0) will be kept in sync within very
low tolerances to the PHC of swp3 (aka /dev/ptp1).

The second option needs collaboration from your board. If you set the
RCW field EC1_SAI4_5_PMUX to the value of 0b101, you make the pinmuxing
of the SoC disable the eno1 RGMII port, and you make those pins available
for the auxiliary PTP pins of /dev/ptp0 and /dev/ptp1.
The idea here is to connect (externally to the SoC) the EC1_1588_PULSE_OUT1
pin of /dev/ptp0 to the SWITCH_1588_DAT[0] pin of /dev/ptp1. Then you
use the ts2phc program to keep in sync the 2 hardware clocks. This is
way more precise than phc2sys and comparable to the ptp4l option, because
the PPS signal is timestamped in hardware using the extts functionality
of the Felix PHC. One small note is that I never had access to a board
with the PPS layout I'm proposing, but based on what I know, it should
work.

There is no option to connect a PPS signal from one PHC to the other
internally to the SoC, unfortunately.

By the way, can you tell more about that hardware trigger signal? I am
familiar with the LS1028A family but I don't know what you are talking
about.

Re: [Linuxptp-users] The LinuxPTP-Users mailing list has moved

[Richard C. <ric...@gm...>]

On Wed, Nov 08, 2023 at 03:28:34PM -0800, Steve Sobol NTF wrote:

> The mailing lists and archives of the LinuxPTP Project are now fully
> hosted at Network Time Foundation. Direct posting to the Sourceforge
> lists has been disabled.

Small correction: Posting to the SF lists has not yet been disabled.
I will disable them on December 6, 2023 (one day after the next
release).

If you want to continue to participate on the mailing lists, please
subscribe to the new lists at the NTF.

Thanks,
Richard

[Linuxptp-users] The LinuxPTP-Users mailing list has moved

[Steve S. N. <sj...@nw...>]

Hello,

The mailing lists and archives of the LinuxPTP Project are now fully
hosted at Network Time Foundation. Direct posting to the Sourceforge
lists has been disabled. Content posted to the developer and user lists
at NTF is currently being forwarded to the Sourceforge lists.

Moving to the NTF list server will allow for easier searching of the
lists, with cleaner navigation of the list archives and none of the
telemetry or ad tracking Sourceforge uses on their websites. It is also
helpful to the LinuxPTP community if web traffic stays on LinuxPTP websites.

Please visit https://www.nwtime.org/privacy-policy/ to learn about how
we use your contact information.

For privacy reasons, we cannot see the list of Sourceforge subscribers.
LinuxPTP mailing list users are asked to subscribe to one or both of the
NTF-hosted mailing lists using the Subscribe option at these URLs:

  - LinuxPTP Users: https://lists.nwtime.org/sympa/info/linuxptp-users

  - LinuxPTP Developers: https://lists.nwtime.org/sympa/info/linuxptp-devel

Each mailing list provides an Archive link which can be used to search
that mailing list’s archives.

Network Time Foundation is an IRS-approved 501(c)3 non-profit
organization responsible for the stewardship and support of many network
time-related protocols and projects, including LinuxPTP and NTP. For
more information, or to donate or become a member, visit
https://www.nwtime.org.

Best regards
--Steve Sobol

[Linuxptp-users] How to use common (v)clock for physically different PHCs?

[Osterried M. (ETAS-DAP/XPC-Fe3) <mar...@et...>]

Hi,

I am using a custom board with NXP LS1027A SoC which has an integrated ethernet controller (which provides one front facing port, eno0) and an integrated ethernet switch (which provides four front facing ports, swp0-swp3).
I want to implement an IEEE1588 boundary clock supporting all five ports. The ethernet controller has attached a different PHC as the ports of the switch has, /dev/ptp0 resp. /dev/ptp1.
Nevertheless, with hardware trigger signal, I can ensure that both clocks start at the same time and runs with same frequency, so they will always have the same value (within granularity of about 5 ns) - as long as they are free running, because frequency adjustments and time jumps can not be done atomically for both clocks.
The idea was to create a virtual PHC (vclock, /dev/ptp2), based on either one of the two physical PHCs.
But while ptp4l prints this info
port 1 (eno0): /dev/ptp2 is virtual clock
port 2 (swp0): taking /dev/ptp2 from the command line, not the attached ptp1
it seems that swp0 still uses the PHC /dev/ptp1 and frames send on this port contains timestamps derived from /dev/ptp1, while eno0 uses /dev/ptp2, so timestamps on both ports are quite different.
I do not really understand what ptp4l is doing when p->phc_index and p->phc_from_cmdline is set - but it seems it is not as I have expected.
But I have also recognized that setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, ...) fails when used with SOF_TIMESTAMPING_BIND_PHC when vclock is not derived from natively attached PHC.

What would be the best solution to use a common (v)clock for all the interfaces?
I want to avoid to synchronize the physical PHCs with phc2sys, because that would be less accurate then using hardware trigger signal which keeps (free running) PHCs on the same value.

Many thanks
Markus

[Linuxptp-users] Fwd: Configuration for boundary clock with on two-port NIC

[egg c. <egg...@gm...>]

Hi,

As Miroslav said, 82599(and other intel nics) has separated phc for
each eth port, and pcie PTM are not supported.
The only way to sync each phc accurately is to use the SDP pins, one
port generate an 1PPS and others take the 1PPS to sync the phc, using
ts2phc.

Andre Puschmann <and...@sr...> 于2023年11月1日周三 16:08写道：
>
> Hi everyone,
>
> I am picking up this thread as we're still facing some issues with
> getting the BC mode working on this two-port Intel 82599ES NIC.
>
> Cabling is still the same as in the original post - one port receives
> PTP sync from an external GM and should redistribute it over the second
> port to a client.
>
> We've previously used two ptp4linux instances, one for each interface
> but I am not sure that is the way to go. Indeed the clock_mode setting
> in the config file allows for BC which in turn requires two interfaces
> to be specified. However, doing so results in a PHC device mismatch:
>
>   /opt/linuxptp/ptp4l -2 -i enp1s0f1 -i enp1s0f0 -f
> /opt/ptp4l_cfg/bc.cfg -m -l 6
> ptp4l[851.735]: selected /dev/ptp1 as PTP clock
> ptp4l[851.735]: port 1 (enp1s0f1): PHC device mismatch
> ptp4l[851.735]: port 1 (enp1s0f1): /dev/ptp1 requested, ptp0 attached
> ptp4l[851.735]: failed to open port enp1s0f1
> failed to create a clock
>
> I am not sure where this is coming from but indeed the first interface
> seems PHC 1 assigned and the second PHC 0. Not sure if this is an issue.
>
> $ ethtool -T enp1s0f0
> Time stamping parameters for enp1s0f0:
> Capabilities:
>          hardware-transmit
>          software-transmit
>          hardware-receive
>          software-receive
>          software-system-clock
>          hardware-raw-clock
> PTP Hardware Clock: 1
> Hardware Transmit Timestamp Modes:
>          off
>          on
> Hardware Receive Filter Modes:
>          none
>          ptpv1-l4-sync
>          ptpv1-l4-delay-req
>          ptpv2-event
>
>
> I've further noticed the phc_index config param but it doesn't make a
> difference, resulting in the same mismatch error.
>
> Specifying the -p param, however, helps in getting over (this) error but
> I still don't get PTP packets on the output interface (tcpdump not
> showing anything).
>
> /opt/linuxptp/ptp4l -2 -i enp1s0f1 -i enp1s0f0 -p /dev/ptp0 -f
> /opt/ptp4l_cfg/bc.cfg -m -l6
> ptp4l[1040.051]: selected /dev/ptp0 as PTP clock
> ptp4l[1040.051]: port 2 (enp1s0f0): taking /dev/ptp0 from the command
> line, not the attached ptp1
> ptp4l[1040.097]: driver rejected most general HWTSTAMP filter
> ptp4l[1040.097]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range
> ptp4l[1040.132]: port 1 (enp1s0f1): INITIALIZING to FAULTY on
> FAULT_DETECTED (FT_UNSPECIFIED)
> ptp4l[1040.180]: driver rejected most general HWTSTAMP filter
> ptp4l[1040.180]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range
> ptp4l[1040.220]: port 2 (enp1s0f0): INITIALIZING to FAULTY on
> FAULT_DETECTED (FT_UNSPECIFIED)
> ptp4l[1040.220]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on
> INIT_COMPLETE
> ptp4l[1040.220]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on
> INIT_COMPLETE
>
>
> My questions are now whether or not the BC clock_mode is the way to go?
> And, if so, what do you think might be causing the issues we are seeing
> with the config [1] or setup?
>
> Thanks
> Andre
>
>
> [1] https://pastes.io/epvudxplcz
>
>
>
> On 3/10/23 15:41, Nils Fuerste wrote:
> > Ah yeah, I didnt read well. That really improved the sync quality -
> > thanks a lot for your advice!
> >
> > On 3/10/23 14:38, Miroslav Lichvar wrote:
> >> On Tue, Oct 03, 2023 at 02:33:32PM +0200, Nils Fuerste wrote:
> >>> Thanks for your help! Unfortunately, setting those values doesn't
> >>> work for
> >>> me. I get the following errors:
> >>>
> >>> sudo ptp4l -2 -i enp1s0f1 -f ./default-new-master.cfg -m
> >>> -1.0 is an out of range value for option pi_proportional_const at
> >>> line 73
> >>> failed to parse configuration file ./default-new-master.cfg
> >> It should be pi_proportional_exponent, not pi_proportional_const.
> >>
> >>> $ sudo ptp4l -2 -i enp1s0f1 -f ./default-new-master.cfg -m
> >>> P is a malformed value for option pi_proportional_scale at line 75
> >>> failed to parse configuration file ./default-new-master.cfg
> >> P and I should be replaced with the values from the table.
> >>
> >
> >
> > _______________________________________________
> > Linuxptp-users mailing list
> > Lin...@li...
> > https://lists.sourceforge.net/lists/listinfo/linuxptp-users
>
>
> _______________________________________________
> Linuxptp-users mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linuxptp-users

Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC

[Jacob K. <jac...@in...>]

On 11/2/2023 4:15 AM, Andre Puschmann wrote:
> Hi,
> 
> On 2/11/23 4:11, James Clark wrote:
>> I have a dual-port Mellanox ConnectX-3 (specifically MCX312A-XCBT),
>> which has a shared PHC. You can get them for less than $50 on
>> eBay/AliExpress. I had to upgrade the firmware on mine to get PTP
>> support. I haven't yet tried it as a boundary clock.
> 
> Excellent. This is very helpful James. I've ordered a MCX312A and B and 
> will compare both here. I'll share my results here soon. If you have a 
> chance please also share the firmware version you're currently using on 
> your NIC.
> 
> With my Intel NIC I could get the BC config working but I needed to set 
> the twoStepFlag to 1. Otherwise I was getting this for both ports:
> 
> ptp4l[1040.180]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range
> 

Yep, that would indicate the device doesn't support one-step mode.

> Sync quality wasn't great as expected though. I'll repeat with the 
> Mellanox once I have them here.
> 
> Thanks
> Andre
> 

For Intel NICs, the only products I am aware of which share PHC across
the device are the E800 series devices. Prior devices (E500, and E700,
as well as the gigabit products) do share the same internal oscillator
but due to the register interface each function has to setup its own clock.

Thanks,
Jake

Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC

[Andre P. <and...@sr...>]

Hi,

On 2/11/23 4:11, James Clark wrote:
> I have a dual-port Mellanox ConnectX-3 (specifically MCX312A-XCBT),
> which has a shared PHC. You can get them for less than $50 on
> eBay/AliExpress. I had to upgrade the firmware on mine to get PTP
> support. I haven't yet tried it as a boundary clock.

Excellent. This is very helpful James. I've ordered a MCX312A and B and 
will compare both here. I'll share my results here soon. If you have a 
chance please also share the firmware version you're currently using on 
your NIC.

With my Intel NIC I could get the BC config working but I needed to set 
the twoStepFlag to 1. Otherwise I was getting this for both ports:

ptp4l[1040.180]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range

Sync quality wasn't great as expected though. I'll repeat with the 
Mellanox once I have them here.

Thanks
Andre

Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC

[James C. <jj...@jc...>]

On Thu, Nov 2, 2023 at 2:30 AM Andre Puschmann <and...@sr...> wrote:
>Is there any other NIC
> that you can recommend and/or know that has a shared PHC among it's ports?

I have a dual-port Mellanox ConnectX-3 (specifically MCX312A-XCBT),
which has a shared PHC. You can get them for less than $50 on
eBay/AliExpress. I had to upgrade the firmware on mine to get PTP
support. I haven't yet tried it as a boundary clock.

I also tried a secondhand dual-port Intel X550-T2, but that has a
separate PHC for each port.

James

Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC

[Andre P. <and...@sr...>]

On 1/11/23 9:28, Miroslav Lichvar wrote:
> On Wed, Nov 01, 2023 at 09:04:12AM +0100, Andre Puschmann wrote:
>>   /opt/linuxptp/ptp4l -2 -i enp1s0f1 -i enp1s0f0 -f /opt/ptp4l_cfg/bc.cfg -m
>> -l 6
>> ptp4l[851.735]: selected /dev/ptp1 as PTP clock
>> ptp4l[851.735]: port 1 (enp1s0f1): PHC device mismatch
>> ptp4l[851.735]: port 1 (enp1s0f1): /dev/ptp1 requested, ptp0 attached
>> ptp4l[851.735]: failed to open port enp1s0f1
>> failed to create a clock
>>
>> I am not sure where this is coming from but indeed the first interface seems
>> PHC 1 assigned and the second PHC 0. Not sure if this is an issue.
> 
> It is an issue. The two ports of the NIC don't share the same PHC, so
> there needs to be something keeping them in sync (phc2sys -a) and
> ptp4l needs to be told to not verify the PHC index numbers. That's the
> boundary_clock_jbod option. Don't expect best accuracy in this setup.

Thanks Miroslav. I'll give this a try tomorrow. Is there any other NIC 
that you can recommend and/or know that has a shared PHC among it's ports?

Thanks
Andre

Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC

[Miroslav L. <mli...@re...>]

On Wed, Nov 01, 2023 at 09:04:12AM +0100, Andre Puschmann wrote:
>  /opt/linuxptp/ptp4l -2 -i enp1s0f1 -i enp1s0f0 -f /opt/ptp4l_cfg/bc.cfg -m
> -l 6
> ptp4l[851.735]: selected /dev/ptp1 as PTP clock
> ptp4l[851.735]: port 1 (enp1s0f1): PHC device mismatch
> ptp4l[851.735]: port 1 (enp1s0f1): /dev/ptp1 requested, ptp0 attached
> ptp4l[851.735]: failed to open port enp1s0f1
> failed to create a clock
> 
> I am not sure where this is coming from but indeed the first interface seems
> PHC 1 assigned and the second PHC 0. Not sure if this is an issue.

It is an issue. The two ports of the NIC don't share the same PHC, so
there needs to be something keeping them in sync (phc2sys -a) and
ptp4l needs to be told to not verify the PHC index numbers. That's the
boundary_clock_jbod option. Don't expect best accuracy in this setup.

-- 
Miroslav Lichvar

Re: [Linuxptp-users] Configuration for boundary clock with on two-port NIC

[Andre P. <and...@sr...>]

Hi everyone,

I am picking up this thread as we're still facing some issues with 
getting the BC mode working on this two-port Intel 82599ES NIC.

Cabling is still the same as in the original post - one port receives 
PTP sync from an external GM and should redistribute it over the second 
port to a client.

We've previously used two ptp4linux instances, one for each interface 
but I am not sure that is the way to go. Indeed the clock_mode setting 
in the config file allows for BC which in turn requires two interfaces 
to be specified. However, doing so results in a PHC device mismatch:

  /opt/linuxptp/ptp4l -2 -i enp1s0f1 -i enp1s0f0 -f 
/opt/ptp4l_cfg/bc.cfg -m -l 6
ptp4l[851.735]: selected /dev/ptp1 as PTP clock
ptp4l[851.735]: port 1 (enp1s0f1): PHC device mismatch
ptp4l[851.735]: port 1 (enp1s0f1): /dev/ptp1 requested, ptp0 attached
ptp4l[851.735]: failed to open port enp1s0f1
failed to create a clock

I am not sure where this is coming from but indeed the first interface 
seems PHC 1 assigned and the second PHC 0. Not sure if this is an issue.

$ ethtool -T enp1s0f0
Time stamping parameters for enp1s0f0:
Capabilities:
         hardware-transmit
         software-transmit
         hardware-receive
         software-receive
         software-system-clock
         hardware-raw-clock
PTP Hardware Clock: 1
Hardware Transmit Timestamp Modes:
         off
         on
Hardware Receive Filter Modes:
         none
         ptpv1-l4-sync
         ptpv1-l4-delay-req
         ptpv2-event


I've further noticed the phc_index config param but it doesn't make a 
difference, resulting in the same mismatch error.

Specifying the -p param, however, helps in getting over (this) error but 
I still don't get PTP packets on the output interface (tcpdump not 
showing anything).

/opt/linuxptp/ptp4l -2 -i enp1s0f1 -i enp1s0f0 -p /dev/ptp0 -f 
/opt/ptp4l_cfg/bc.cfg -m -l6
ptp4l[1040.051]: selected /dev/ptp0 as PTP clock
ptp4l[1040.051]: port 2 (enp1s0f0): taking /dev/ptp0 from the command 
line, not the attached ptp1
ptp4l[1040.097]: driver rejected most general HWTSTAMP filter
ptp4l[1040.097]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range
ptp4l[1040.132]: port 1 (enp1s0f1): INITIALIZING to FAULTY on 
FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[1040.180]: driver rejected most general HWTSTAMP filter
ptp4l[1040.180]: ioctl SIOCSHWTSTAMP failed: Numerical result out of range
ptp4l[1040.220]: port 2 (enp1s0f0): INITIALIZING to FAULTY on 
FAULT_DETECTED (FT_UNSPECIFIED)
ptp4l[1040.220]: port 0 (/var/run/ptp4l): INITIALIZING to LISTENING on 
INIT_COMPLETE
ptp4l[1040.220]: port 0 (/var/run/ptp4lro): INITIALIZING to LISTENING on 
INIT_COMPLETE


My questions are now whether or not the BC clock_mode is the way to go? 
And, if so, what do you think might be causing the issues we are seeing 
with the config [1] or setup?

Thanks
Andre


[1] https://pastes.io/epvudxplcz



On 3/10/23 15:41, Nils Fuerste wrote:
> Ah yeah, I didnt read well. That really improved the sync quality - 
> thanks a lot for your advice!
> 
> On 3/10/23 14:38, Miroslav Lichvar wrote:
>> On Tue, Oct 03, 2023 at 02:33:32PM +0200, Nils Fuerste wrote:
>>> Thanks for your help! Unfortunately, setting those values doesn't 
>>> work for
>>> me. I get the following errors:
>>>
>>> sudo ptp4l -2 -i enp1s0f1 -f ./default-new-master.cfg -m
>>> -1.0 is an out of range value for option pi_proportional_const at 
>>> line 73
>>> failed to parse configuration file ./default-new-master.cfg
>> It should be pi_proportional_exponent, not pi_proportional_const.
>>
>>> $ sudo ptp4l -2 -i enp1s0f1 -f ./default-new-master.cfg -m
>>> P is a malformed value for option pi_proportional_scale at line 75
>>> failed to parse configuration file ./default-new-master.cfg
>> P and I should be replaced with the values from the table.
>>
> 
> 
> _______________________________________________
> Linuxptp-users mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linuxptp-users

Re: [Linuxptp-users] Hardware timestamping multiple domains.

[Miroslav L. <mli...@re...>]

On Mon, Oct 30, 2023 at 10:26:35PM +0000, Eric Decker wrote:
> My understanding is each domain has a set of PTP messages associated with it.  If we have two domains there would two sets of Sync, Follow Up and path delay messages on the network simultaneously.  Would the hardware stamping in the MAC timestamp frames/messages from both domains?  If the MAC does capture timestamps for the PTP frames in both domains, then does the PHC and vclocks process those timestamps?

Hardware doesn't care about domains. It timestamps all event
messages. With vclocks the hardware timestamp is interpreted as a
simple counter and the hardware timestamps are converted by vclocks to
timestamp that will be receive by applications. The application
selects the vclock by binding its socket. A single ptp4l instance
works in a single domain, so it needs just one vclock. It will receive
messages from other domains with timestamps converted by its vclock,
but it will ignore them. With multiple ptp4l instances you have
hardware timestamping enabled in multiple domains, each instance using
only timestamps of messages in its domain.

-- 
Miroslav Lichvar

[Linuxptp-users] Hardware timestamping multiple domains.

[Eric D. <eri...@to...>]

Hello,

I found the following conversations about PTP time sync in multiple time domains using vclocks.


https://sourceforge.net/p/linuxptp/mailman/linuxptp-users/thread/ZC5fs7FCI6qf3eLe%40localhost/#msg37800712

https://sourceforge.net/p/linuxptp/mailman/linuxptp-users/thread/20210806163702.GA15386%40hoboy.vegasvil.org/#msg37331851

In the second link it says,


"A virtual PHC is nothing more than a Linux kernel
timecounter/cyclecounter structure on top of a physical PHC which the
kernel keeps completely free-running. The timecounter/cyclecounter
structure provides 'virtual' time bases on top of the same physical
counter, i.e. when you adjust the frequency or the absolute time of the
virtual clock, only a software correction factor is modified, which is
applied to all raw clock values read from the physical PHC. So the
clock_gettime() returned by a virtual PHC is a simple "a * x + b"
translation of the clock_gettime() returned by the backing PHC. This is
also what allows you to have many virtual clocks on top of the same
physical one."


My understanding is each domain has a set of PTP messages associated with it.  If we have two domains there would two sets of Sync, Follow Up and path delay messages on the network simultaneously.  Would the hardware stamping in the MAC timestamp frames/messages from both domains?  If the MAC does capture timestamps for the PTP frames in both domains, then does the PHC and vclocks process those timestamps?

Thank you,


[cid:image001.png@01DA0B5D.67D77B10]
Eric Decker
TORC Robotics, Inc.
405 Partnership Drive, Blacksburg, VA 24060
Email:     eri...@to...<mailto:eri...@to...>
Website:  https://torc.ai<https://torc.ai/>

[cid:image002.png@01DA0B5D.67D77B10]<https://www.facebook.com/torcrobotics/> [cid:image003.png@01DA0B5D.67D77B10] <https://www.linkedin.com/company/torc-technologies/about/?viewAsMember=true>  [cid:image004.png@01DA0B5D.67D77B10] <https://twitter.com/torcrobotics>  [cid:image005.png@01DA0B5D.67D77B10] <https://www.youtube.com/channel/UCGKXBpisBMTJtshrr3KkOTg?view_as=subscriber>  [cid:image006.png@01DA0B5D.67D77B10] <https://www.instagram.com/torc_robotics/?hl=en>  [cid:image007.jpg@01DA0B5D.67D77B10] <https://www.glassdoor.com/Overview/Working-at-TORC-Robotics-EI_IE1027022.11,24.htm>


Confidentiality Notice: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

Re: [Linuxptp-users] Use NIC as PTP GM without external sync

[Nils F. <nil...@sr...>]

On 19/10/23 14:51, Miroslav Lichvar wrote:
> On Thu, Oct 19, 2023 at 02:40:30PM +0200, Nils Fuerste wrote:
> ptp4l[2023/10/19/12:28:10]: state=2 rms  222 max  518 freq  +1020 +/-   0
> delay 20436 +/-  13
> ptp4l[2023/10/19/12:28:11]: state=2 rms 968245340 max 999999669 freq   +992
> +/-   0 delay 20507 +/-  36
> ptp4l[2023/10/19/12:28:13]: state=2 rms 433012546 max 999999740 freq   +921
> +/-  28 delay 20559 +/-  31
> Those errors are almost exactly 1 second, which is very suspicious. It
> could be a driver bug. What kernel version do you use? It would help
> if you could get a third device and see if it works better as a server
> or client to determine on which side is the problem.
>
As proposed I have tested some other scenarios. During this process I 
have realized that the PTP version on the embedded device is probably 
not mainline. I have copied the config to another machine with mainline 
PTP and it was complaining about many parameters in the PTP client 
config of the embedded device.

I have used some other configuration to narrow down the issue. These are 
the results:

- Using other master, same client, same configs -> Same result

- Using other master, other client with  config from embedded device -> 
Results are worse than before (I had to comment out some config items 
that were not recognized by ptp4l.

- Using a proper PTP GM (Qulsar Q2), original client (embedded device) 
-> Good sync, around rms 20 +- 5

- I have tested different SFP to Ethernet adapters in all of those tests 
above, sync quality stayed the same


What can I do? Since a proper PTP GM can provide a good PTP sync to the 
client I guess my config is faulty? Or can it be something else? Can I 
maybe do some more tests?

Many thanks!
Nils

Re: [Linuxptp-users] Use NIC as PTP GM without external sync

[Miroslav L. <mli...@re...>]

On Thu, Oct 19, 2023 at 02:40:30PM +0200, Nils Fuerste wrote:
> Hello everyone!
> 
> I am trying to use a NIC as a PTP GM with any external sync sorce. I have a
> PC with a dual port NIC (82599ES 10-Gigabit SFI/SFP+) connected to an
> embedded device over Ethernet. I use a SFP-to-Ethernet adapter to translate
> from SFP+ to Ethernet. Both devices need to be synced but I cannot use GPS
> or other external clock sources. My idea is to use the NIC as a PTP GM and
> serve the embedded device with PTP. What I am not sure about is if I need to
> use phc2sys to sync NIC and the physical clock of the machine hosting the
> NIC? What quality of sync can I expect?

You can leave the clock of the NIC running free if you don't need the
client's clock to be accurate, only synchronized to the server.

> ptp4l[2023/10/19/12:28:10]: state=2 rms  222 max  518 freq  +1020 +/-   0
> delay 20436 +/-  13
> ptp4l[2023/10/19/12:28:11]: state=2 rms 968245340 max 999999669 freq   +992
> +/-   0 delay 20507 +/-  36
> ptp4l[2023/10/19/12:28:13]: state=2 rms 433012546 max 999999740 freq   +921
> +/-  28 delay 20559 +/-  31

Those errors are almost exactly 1 second, which is very suspicious. It
could be a driver bug. What kernel version do you use? It would help
if you could get a third device and see if it works better as a server
or client to determine on which side is the problem.

-- 
Miroslav Lichvar