linuxptp-users — Open discussion on using the linuxptp software.

Re: [Linuxptp-users] LinuxPTP on Zynq with DP83640

[Richard C. <ric...@gm...>]

On Tue, Apr 18, 2017 at 12:54:23PM -0700, Wolfgang Hennig wrote:
> I'm working on a project where a Xilinx Zynq uses a DP83640 (eval board) as
> the Ethernet PHY. I'm running Ubuntu/Linaro 15 on the Zynq's ARM processor
> and successfully modified kernel options and compiled/installed LinuxPTP to
> synchronize time over the network. Now I would like to use the DP83640's
> configurable reference clock (output on a GPIO pin) to clock logic in the
> FPGA section of the Zynq. To do that I would need to read/write the
> DP83640's internal registers, but I don't know how.
> 
> It seems that there is some functionality in setting the registers in
> dp83640.c. In an earlier post in this list:

We don't have any support for enabling the clock output.  You can hack
this in using something like this:

	ext_write(0, phydev, PAGE6, PTP_COC, coc);

> I should add that I did not (yet) find where/how to modify the xemacps
> driver per https://lwn.net/Articles/392150/
> >    1. Before mdio_register, add
> >		bus->locktype = MDIOBUS_ATOMIC_RW;
> >    2. In the .ndo_start_xmit function, add
> >		skb_tx_timestamp()
> >    3. In the NAPI poll function, add
> >		skb_rx_timestamp()

That info is obsolete.  All you need is a MAC driver that uses phylib
and calls skb_tx_timestamp().

> and so the HW timestamping might have been provided by the Zynq's build in
> TSU instead of the DP83640. How would you tell?

If your Zynq driver implements time stamping or PHC, then you should
disable or remove that code.

BTW you can check sysfs to see the PHC device names:

What:		/sys/class/ptp/ptpN/clock_name
Description:
		This file contains the name of the PTP hardware clock
		as a human readable string. The purpose of this
		attribute is to provide the user with a "friendly
		name" and to help distinguish PHY based devices from
		MAC based ones. The string does not necessarily have
		to be any kind of unique id.


> Does it matter where in the
> function one adds skb_tx_timestamp()?

See the comment in include/linux/skbuff.h.

Thanks,
Richard

[Linuxptp-users] LinuxPTP on Zynq with DP83640

[Wolfgang H. <wh...@xi...>]

Hi,

I'm working on a project where a Xilinx Zynq uses a DP83640 (eval board) 
as the Ethernet PHY. I'm running Ubuntu/Linaro 15 on the Zynq's ARM 
processor and successfully modified kernel options and 
compiled/installed LinuxPTP to synchronize time over the network. Now I 
would like to use the DP83640's configurable reference clock (output on 
a GPIO pin) to clock logic in the FPGA section of the Zynq. To do that I 
would need to read/write the DP83640's internal registers, but I don't 
know how.

It seems that there is some functionality in setting the registers in 
dp83640.c. In an earlier post in this list:
> >  (2) DP83640 Driver: I found several comments that the GPIOs of the
> >  PHYs should be connected together on order to be able to synchronize
> >  the two PHYs (recalibrate) but I did't find the detail information
> >  which GPIO of PHY 1should be connected to which GPIO of PHY 2.
>
> If does not matter which GPIO. You can choose any one of these.
>
> 	1, 2, 3, 4, 8, 9, 10, 11
>
> The dp83640 driver has a module parameter that lets you configure
> this.

  In the code I see statements like this (e.g. line 678)
> ext_write  <http://lxr.free-electrons.com/ident?i=ext_write>(0, master,PAGE4  <http://lxr.free-electrons.com/ident?i=PAGE4>,PTP_CTL  <http://lxr.free-electrons.com/ident?i=PTP_CTL>,val  <http://lxr.free-electrons.com/ident?i=val>);
which seems to write to the PTP control register in page 4 of the 
DP83640's memory. However I can't figure out how this would be used, 
either through LinuxPTP or a separate user program. Any help would be 
greatly appreciated.


Some more system info:
I'm using the xemacps driver for the Zynq MAC
> root@pixie-net:~# dmesg | grep b000
> [    0.867840] mdio_bus ef2df0a4: /amba/ethernet@e000b000/mdio has 
> invalid PHY address
> [    1.010225] xemacps e000b000.ethernet: pdev->id -1, baseaddr 
> 0xe000b000, irq 147
> [    3.486259] xemacps e000b000.ethernet eth1: renamed from eth0
> [    6.282066] xemacps e000b000.ethernet: GEM: phydev ef202c00, 
> phydev->phy_id 0x20005ce1, phydev->addr 0x0
> [    6.282097] xemacps e000b000.ethernet: phy_addr 0x0, phy_id 0x20005ce1
> [    6.282107] xemacps e000b000.ethernet: attach [NatSemi DP83640] phy 
> driver
> [    8.391352] xemacps e000b000.ethernet: Set clk to 0 Hz
> [    8.391367] xemacps e000b000.ethernet: link up (100/FULL)
Though there are some warnings, I get a working connection through our 
local network


ethtool reports
> root@pixie-net:~# ethtool -T eth1
> Time stamping parameters for eth1:
> Capabilities:
>         hardware-transmit     (SOF_TIMESTAMPING_TX_HARDWARE)
>         hardware-receive      (SOF_TIMESTAMPING_RX_HARDWARE)
>         hardware-raw-clock    (SOF_TIMESTAMPING_RAW_HARDWARE)
> PTP Hardware Clock: 0
> Hardware Transmit Timestamp Modes:
>         off                   (HWTSTAMP_TX_OFF)
>         on                    (HWTSTAMP_TX_ON)
>         one-step-sync         (HWTSTAMP_TX_ONESTEP_SYNC)
> Hardware Receive Filter Modes:
>         none                  (HWTSTAMP_FILTER_NONE)
>         ptpv1-l4-event        (HWTSTAMP_FILTER_PTP_V1_L4_EVENT)
>         ptpv2-l4-event        (HWTSTAMP_FILTER_PTP_V2_L4_EVENT)
>         ptpv2-l2-event        (HWTSTAMP_FILTER_PTP_V2_L2_EVENT)
>         ptpv2-event           (HWTSTAMP_FILTER_PTP_V2_EVENT)

Running ptp4l on 2 systems connected though the local network, I see 
them link up and adjust to an offset of a few hundred ns.

I should add that I did not (yet) find where/how to modify the xemacps 
driver per https://lwn.net/Articles/392150/
>     1. Before mdio_register, add
> 		bus->locktype = MDIOBUS_ATOMIC_RW;
>     2. In the .ndo_start_xmit function, add
> 		skb_tx_timestamp()
>     3. In the NAPI poll function, add
> 		skb_rx_timestamp()
and so the HW timestamping might have been provided by the Zynq's build 
in TSU instead of the DP83640. How would you tell? Does it matter where 
in the function one adds skb_tx_timestamp()?


Thanks for your help,

Wolfgang









-- 
Wolfgang Hennig
XIA LLC

Re: [Linuxptp-users] ptp4l stack integration via linux OS with least burden on the Firmware and major PTP offload to the hardware

[MSR, C. <cha...@in...>]

Hi Jake,

I appreciate your feedback. Yes - our hardware and most of other hardware supports 'timestamp all' - an easier option for the software as you mention. However, it is not always the case for different customers -
•       given the encapsulations such as MPLS-TP beyond the linux supported IPv4/6 UDP packets, we definitely need proprietary hardware for improved parsing capabilities.
•       Also, the 1-step implementation requires insertion of ToD and update of CF at specific locations while the packets are in transit (unlike 2-step mechanism, where the timestamps can be collected later/after the packets are egressed). These offsets are obtained by parsing the packets inside the hardware rather than blowing the size of the descriptors, which consumes many cpu cycles.
•       Most importantly we have witnessed two groups - generic OS based (Linux etc) group, which prefers 'timestamp all' and proprietary OS based (network switches, processors) group, which takes advantage of the underlying hardware to reap the best performance (timestamping as directed by stack a subset). Needless to say, the hardware has to support both the groups and we do that.

Thanking you in anticipation,
Regards,
Chandra

© : +60.175508142
(O): +60.4.636.6412

“Knowledge speaks, Wisdom listens”


-----Original Message-----
From: Keller, Jacob E
Sent: Saturday, April 15, 2017 12:12 AM
To: MSR, CHANDRASEKHAR <cha...@in...>; Richard Cochran <ric...@gm...>
Cc: lin...@li...
Subject: RE: [Linuxptp-users] ptp4l stack integration via linux OS with least burden on the Firmware and major PTP offload to the hardware



> -----Original Message-----
> From: MSR, CHANDRASEKHAR [mailto:cha...@in...]
> Sent: Friday, April 14, 2017 7:48 AM
> To: Richard Cochran <ric...@gm...<mailto:ric...@gm...>>
> Cc: lin...@li...<mailto:lin...@li...>
> Subject: Re: [Linuxptp-users] ptp4l stack integration via linux OS
> with least burden on the Firmware and major PTP offload to the
> hardware
>
>
>
> Hi Richard,
>
> Thank you a lot for responding to my question though you could have
> easily shut me off, given the 'off topic' nature of the question. I
> need some basic clarificaions from your response.
>
> You say " IMHO this is only reasonable approach for modern MAC hardware" & "
> Nonsense.  The driver must read the descriptor in any case, and it
> will only handle the time stamp if the option is enabled.". My
> confusion here is: You seem to support timestamping all the packets
> but driver will handle the timestamp if the option is enabled. If the
> driver knows whether to handle the timestamp or not through the flags
> in the DMA descriptor, then the underlying MAC hardware does take
> advantage of it so that the timestamping is carried out for the
> packets (any - ptp or nonptp), which has the option enabled. As you
> might know, the hardware usually stores the timestamps in a FIFO,
> which will be read out on polling the status or a suitable interrupt.
> With the distinction between packet needing timestamps (ptp or nonptp)
> and packet with no need of timestamps, the number of timestamps that
> can be pushed by the MAC hardware into the FIFO is limited, saving
> considerable silicon - especially the case with ASICs, where the
> depths are predetermined unlike FPGAs. Further, the driver has to wait for some more time after the packet is transmitted, to complete the dma ack with the timestamp (to be collected from the FIFO and put it back in the descriptor).
> However, for the packets not needing timestamps, the driver can give
> the ack quickly fetching the next set of descriptors. Thus,
> timestamping as needed by the MAC hardware is really the need of the hour than timestamping every packet.
> Please share your opinion.
>
> You say " It is just eight bytes, after all". Actually, it is not a PTM (PCIe) timestamp.
> In 1588v2, the timestamp is 10 bytes/80b. Further, our hardware
> provides even fractional nanoseconds of 16b (giving out 2^-16ns
> resolution). Understandably, the proprietary stacks have the
> capability to handle the complete 12B of timestamps than the public-domain stacks. Just FYI. Do share your opinion.
>
> Thank you again for sharing your expertise with us - it is very helpful.

It's certainly easier in software if all packets have a timestamp. Additionally, I imagine it's easier in hardware if there is no special casing since you wouldn't need any circuitry to build out how to decide which frames get timestamped and which do not. This should be simpler. It seems to be that yes you do pay the 8 or 16 bytes per frame if you have timestamps always, but that cost is smaller than the huge complexity required to have the hardware pre-determine which packet gets timestamped if it only supports a subset of "all packets" while also forcing software to now be aware of what the hardware will or will not timestamp.

It may be that specific hardware has limitations on how it timestamps for other reasons, but obviously we can't comment on how the hardware is designed, since we didn't design it. I dont think a FIFO for pushing timestamps by the MAC is necessarily the only way to do it, and certainly lots of hardware already suports "timestamp all packets" already today. This is a valuable feature to software already, so if you wanted to support it then you don't need support for any subset of all packets.

Thanks,
Jake

Re: [Linuxptp-users] ptp4l stack integration via linux OS with least burden on the Firmware and major PTP offload to the hardware

[Keller, J. E <jac...@in...>]

> -----Original Message-----
> From: MSR, CHANDRASEKHAR [mailto:cha...@in...]
> Sent: Friday, April 14, 2017 7:48 AM
> To: Richard Cochran <ric...@gm...>
> Cc: lin...@li...
> Subject: Re: [Linuxptp-users] ptp4l stack integration via linux OS with least
> burden on the Firmware and major PTP offload to the hardware
> 
> 
> 
> Hi Richard,
> 
> Thank you a lot for responding to my question though you could have easily shut
> me off, given the 'off topic' nature of the question. I need some basic clarificaions
> from your response.
> 
> You say " IMHO this is only reasonable approach for modern MAC hardware" & "
> Nonsense.  The driver must read the descriptor in any case, and it will only handle
> the time stamp if the option is enabled.". My confusion here is: You seem to
> support timestamping all the packets but driver will handle the timestamp if the
> option is enabled. If the driver knows whether to handle the timestamp or not
> through the flags in the DMA descriptor, then the underlying MAC hardware
> does take advantage of it so that the timestamping is carried out for the packets
> (any - ptp or nonptp), which has the option enabled. As you might know, the
> hardware usually stores the timestamps in a FIFO, which will be read out on
> polling the status or a suitable interrupt. With the distinction between packet
> needing timestamps (ptp or nonptp) and packet with no need of timestamps, the
> number of timestamps that can be pushed by the MAC hardware into the FIFO is
> limited, saving considerable silicon - especially the case with ASICs, where the
> depths are predetermined unlike FPGAs. Further, the driver has to wait for some
> more time after the packet is transmitted, to complete the dma ack with the
> timestamp (to be collected from the FIFO and put it back in the descriptor).
> However, for the packets not needing timestamps, the driver can give the ack
> quickly fetching the next set of descriptors. Thus, timestamping as needed by the
> MAC hardware is really the need of the hour than timestamping every packet.
> Please share your opinion.
> 
> You say " It is just eight bytes, after all". Actually, it is not a PTM (PCIe) timestamp.
> In 1588v2, the timestamp is 10 bytes/80b. Further, our hardware provides even
> fractional nanoseconds of 16b (giving out 2^-16ns resolution). Understandably,
> the proprietary stacks have the capability to handle the complete 12B of
> timestamps than the public-domain stacks. Just FYI. Do share your opinion.
> 
> Thank you again for sharing your expertise with us - it is very helpful.

It's certainly easier in software if all packets have a timestamp. Additionally, I imagine it's easier in hardware if there is no special casing since you wouldn't need any circuitry to build out how to decide which frames get timestamped and which do not. This should be simpler. It seems to be that yes you do pay the 8 or 16 bytes per frame if you have timestamps always, but that cost is smaller than the huge complexity required to have the hardware pre-determine which packet gets timestamped if it only supports a subset of "all packets" while also forcing software to now be aware of what the hardware will or will not timestamp.

It may be that specific hardware has limitations on how it timestamps for other reasons, but obviously we can't comment on how the hardware is designed, since we didn't design it. I dont think a FIFO for pushing timestamps by the MAC is necessarily the only way to do it, and certainly lots of hardware already suports "timestamp all packets" already today. This is a valuable feature to software already, so if you wanted to support it then you don't need support for any subset of all packets.

Thanks,
Jake

Re: [Linuxptp-users] ptp4l stack integration via linux OS with least burden on the Firmware and major PTP offload to the hardware

[MSR, C. <cha...@in...>]

Hi Richard,

Thank you a lot for responding to my question though you could have easily shut me off, given the 'off topic' nature of the question. I need some basic clarificaions from your response.

You say " IMHO this is only reasonable approach for modern MAC hardware" & " Nonsense.  The driver must read the descriptor in any case, and it will only handle the time stamp if the option is enabled.". My confusion here is: You seem to support timestamping all the packets but driver will handle the timestamp if the option is enabled. If the driver knows whether to handle the timestamp or not through the flags in the DMA descriptor, then the underlying MAC hardware does take advantage of it so that the timestamping is carried out for the packets (any - ptp or nonptp), which has the option enabled. As you might know, the hardware usually stores the timestamps in a FIFO, which will be read out on polling the status or a suitable interrupt. With the distinction between packet needing timestamps (ptp or nonptp) and packet with no need of timestamps, the number of timestamps that can be pushed by the MAC hardware into the FIFO is limited, saving considerable silicon - especially the case with ASICs, where the depths are predetermined unlike FPGAs. Further, the driver has to wait for some more time after the packet is transmitted, to complete the dma ack with the timestamp (to be collected from the FIFO and put it back in the descriptor). However, for the packets not needing timestamps, the driver can give the ack quickly fetching the next set of descriptors. Thus, timestamping as needed by the MAC hardware is really the need of the hour than timestamping every packet. Please share your opinion.

You say " It is just eight bytes, after all". Actually, it is not a PTM (PCIe) timestamp. In 1588v2, the timestamp is 10 bytes/80b. Further, our hardware provides even fractional nanoseconds of 16b (giving out 2^-16ns resolution). Understandably, the proprietary stacks have the capability to handle the complete 12B of timestamps than the public-domain stacks. Just FYI. Do share your opinion.

Thank you again for sharing your expertise with us - it is very helpful.

***
Having said that, I cannot resist offering an opinion...

> A simple solution is timestamping all the packets and making the dma 
> flow uniform to all the packets - ptp and non-ptp regular packets, 
> where each packet's timestamp whether needed or not for the ptp4l 
> stack will be stored in the hardware fifo.

IMHO this is only reasonable approach for modern MAC hardware.  Just write the time stamp into the packet descriptor and be done with it.
It is just eight bytes, after all.  No two threads, no packet parsing or alternate paths.  KISS.

Also, precise time stamps are useful to other applications beyond PTP, and so making PTP frames into a special case is artificially limiting the usefulness of your HW.

> It can lead to overflow due to slower turnaround from the driver or

Nonsense.  The driver must read the descriptor in any case, and it will only handle the time stamp if the option is enabled.
 
> in the worst case slow down the link losing the throughput.

again, you can have a descriptor bit that tells whether to copy the time stamp back or not.

Just my 2 cents,
***

Thanking you in anticipation,
Regards,
Chandra

(c) : +60.175508142
(O): +60.4.636.6412

"Knowledge speaks, Wisdom listens"


-----Original Message-----
From: Richard Cochran [mailto:ric...@gm...] 
Sent: Thursday, April 13, 2017 3:41 AM
To: MSR, CHANDRASEKHAR <cha...@in...>
Cc: lin...@li...
Subject: Re: [Linuxptp-users] ptp4l stack integration via linux OS with least burden on the Firmware and major PTP offload to the hardware

Chandra,

Your question has nothing to do with linuxptp (user space stack) or even with the Linux kernel, and as such it is off topic for this list.

Having said that, I cannot resist offering an opinion...

> A simple solution is timestamping all the packets and making the dma 
> flow uniform to all the packets - ptp and non-ptp regular packets, 
> where each packet's timestamp whether needed or not for the ptp4l 
> stack will be stored in the hardware fifo.

IMHO this is only reasonable approach for modern MAC hardware.  Just write the time stamp into the packet descriptor and be done with it.
It is just eight bytes, after all.  No two threads, no packet parsing or alternate paths.  KISS.

Also, precise time stamps are useful to other applications beyond PTP, and so making PTP frames into a special case is artificially limiting the usefulness of your HW.

> It can lead to overflow due to slower turnaround from the driver or

Nonsense.  The driver must read the descriptor in any case, and it will only handle the time stamp if the option is enabled.
 
> in the worst case slow down the link losing the throughput.

again, you can have a descriptor bit that tells whether to copy the time stamp back or not.

Just my 2 cents,

Richard

Re: [Linuxptp-users] ptp4l stack integration via linux OS with least burden on the Firmware and major PTP offload to the hardware

[Richard C. <ric...@gm...>]

Chandra,

Your question has nothing to do with linuxptp (user space stack) or
even with the Linux kernel, and as such it is off topic for this list.

Having said that, I cannot resist offering an opinion...

> A simple solution is timestamping all the packets and making the dma
> flow uniform to all the packets - ptp and non-ptp regular packets,
> where each packet's timestamp whether needed or not for the ptp4l
> stack will be stored in the hardware fifo. 

IMHO this is only reasonable approach for modern MAC hardware.  Just
write the time stamp into the packet descriptor and be done with it.
It is just eight bytes, after all.  No two threads, no packet parsing
or alternate paths.  KISS.

Also, precise time stamps are useful to other applications beyond PTP,
and so making PTP frames into a special case is artificially limiting
the usefulness of your HW.

> It can lead to overflow due to slower turnaround from the driver or

Nonsense.  The driver must read the descriptor in any case, and it
will only handle the time stamp if the option is enabled.
 
> in the worst case slow down the link losing the throughput.

again, you can have a descriptor bit that tells whether to copy the
time stamp back or not.

Just my 2 cents,

Richard

[Linuxptp-users] ptp4l stack integration via linux OS with least burden on the Firmware and major PTP offload to the hardware

[MSR, C. <cha...@in...>]

Hi Friends,

I am lost in visualizing a proper system with ptp4l on our hardware that supports linux along with hardware timestamping. Could you help bring some clarity on my following question? Please note that I am a systems architect and a linux expert and thus the following flow is a generic one. Please fill in the gaps from the linux perspective, if you see any. Appreciate your help on this long thread.

Ptp4l stack generates the sync packets - can be layer 2, layer3 (ipv4/6) with UDP etc. Each packet gets a set of skbufs + 1 more skbuf for associating the corresponding timestamp from the hardware to complete the dma flow for a ptp packet. Now, these sync packets come along with other regular Ethernet, IP packets, which donot need the extra skbuf for timestamp purpose. The hardware (DMA+PTP packet parser + MAC+Phy) has the capability to timestamp only the PTP packets. Now, WITHOUT LOADING THE DRIVER/FIRMWARE (the driver should not parse the packets to know whether they are ptp or non-ptp regular packets), can I have the capability to assign an extra skbuf for a ptp packet to associate the timestamp with the packet's DMA flow? In other words, the non-ptp packets will be parsed accordingly in the hardware without generating any timestamps and thus their DMA flow (considering tx) will terminate with no the extra skbuf (not needed as they donot generate timestamps in the hardware). Given this scenario,

How does the driver know for which packet, it should complete the dma flow by waiting for the timestamp and associating it with the extra skbuf? If we can have this driven by the respective application (ptp4l for ptp packets and tcp/udp apps for regular non-ptp traffic) in a way that the firmware/driver knows about it easily without any burdensome operation such as parsing, it is very efficient (more traffic on ethernet link). More importantly, the underlying hardware does the tougher job of producing timestamps for only the necessary packets.

A simple solution is timestamping all the packets and making the dma flow uniform to all the packets - ptp and non-ptp regular packets, where each packet's timestamp whether needed or not for the ptp4l stack will be stored in the hardware fifo. It can lead to overflow due to slower turnaround from the driver or in the worst case slow down the link losing the throughput.

My preferred solution: The DMA header has a control bit that clearly tells PTP or non-PTP packet. The firmware has two threads - mac and ptp. Now the contro bit determins which thread has to complete the dma flow. For example, for a non-ptp packet, the mac driver will complete the DMA flow by sending the status back without any timestamp in it (prefer to have no bytes allocated for the timestamp at all). For a ptp packet, the ptp driver will complete the DMA flow by sending the final status back to the DMA driver by inserting the timestamp (extra bytes present in the DMA status word) in the status word. The ptp driver knows that it has to read the status register to check whether the PTP packet's timestamp is ready in the FIFO. However, the mac driver does not even bother about this and will complete sending its status quickly. I see no sequential requirements of packet status for the DMA completion between ptp and non-ptp packets. I feel this is very efficient and faster, enabling the highest throughput on the Ethernet link and achieving the objective that the PTP flow should anyway utilize the link to the absolute minimum required and be least intrusive to the regular non-PTP flow.

Thanking you in anticipation,
Regards,
Chandra

(c) : +60.175508142
(O): +60.4.636.6412

"Knowledge speaks, Wisdom listens"

Re: [Linuxptp-users] [External] Re: PTP - MAC time

[Richard C. <ric...@gm...>]

On Wed, Apr 12, 2017 at 02:30:51PM +0000, Ian Thompson wrote:
> If I do "adjtimex -p", I get a raw time e.g.  1492005529.681665099.

I assume that is some wrapper program around the adjtimex system call?

> This is system time, but is it TAI, Unix time or UTC?

The system call...

ADJTIMEX(2)                Linux Programmer's Manual               ADJTIMEX(2)

NAME
       adjtimex - tune kernel clock

SYNOPSIS
       #define _BSD_SOURCE      /* See feature_test_macros(7) */
       #include <sys/timex.h>

       int adjtimex(struct timex *buf);

... returns UTC.

> I was assuming it was seconds from 00:00:00 UTC without leap seconds. 

UTC always includes leap seconds by definition.

> We are trying to timestamp some data collection with GPS time. 
> The stmicro MAC has an auxiliary register that can be updated with the clock registers on an external event.
> We clock that event at 4kHz and use the values in the auxiliary registers to form the timestamp, which is a single 64bit value in uSeconds.
> This value always seems to be 37 seconds off, even if I "correct" stmmac_hwtstamp_ioctl to not use UTC.
> 
> Is there something wrong with my thinking?

I am not sure.  Do you read /dev/ptpX to get those time stamps?
What does 'testptp -c' say about your device?

Thanks,
Richard

Re: [Linuxptp-users] [External] Re: PTP - MAC time

[Ian T. <Ian...@pg...>]

Richard

Sorry, more questions.
If I do "adjtimex -p", I get a raw time e.g.  1492005529.681665099.
This is system time, but is it TAI, Unix time or UTC?
I was assuming it was seconds from 00:00:00 UTC without leap seconds. 

We are trying to timestamp some data collection with GPS time. 
The stmicro MAC has an auxiliary register that can be updated with the clock registers on an external event.
We clock that event at 4kHz and use the values in the auxiliary registers to form the timestamp, which is a single 64bit value in uSeconds.
This value always seems to be 37 seconds off, even if I "correct" stmmac_hwtstamp_ioctl to not use UTC.

Is there something wrong with my thinking?

Thanks
Ian T.

-----Original Message-----
From: Richard Cochran [mailto:ric...@gm...] 
Sent: Tuesday, April 11, 2017 8:27 AM
To: Ian Thompson
Cc: lin...@li...
Subject: Re: [External] Re: [Linuxptp-users] PTP - MAC time

On Mon, Apr 10, 2017 at 03:18:32PM +0000, Ian Thompson wrote:
> I still see the issue with 3.18 but I haven't yet seen it on 4.9. 
> Unfortunately, we have a proprietary driver for a device on the pcie bus which doesn't yet support 4.x kernels and it is this that generates (via an application) most of the network traffic.
> I might have to port all of the stmmac changes back to 3.18.

Maybe I wasn't clear enough in my answer, but the loss of one time stamp is unfortunate but understandable, and it normally should be tolerable.

The resetting of the PHC time (the cause of the 37 second error) in the driver is just plain wrong, and you should fix that.
 
> If I add 37 seconds to getnstimeofday then the effect of the "glitch" is less pronounced. 
> Kernel 3.18 introduced timekeeping.c, with  timekeeping_get_tai_offset(), which I thought might give me the UTC offset but it returns 0 at the point I call it.
> Is there a call within the kernel to find the UTC offset?

The offset has to be provided to the kernel by user space by calling
adjtimex() with the ADJ_TAI mode set.

Thanks,
Richard

Re: [Linuxptp-users] CONFIG_NETWORK_PHY_TIMESTAMPING

[Richard C. <ric...@gm...>]

On Tue, Apr 11, 2017 at 12:05:07PM +0200, Axel Holzinger wrote:
> Or is it okay to leave CONFIG_NETWORK_PHY_TIMESTAMPING undefined in case the
> MAC is doing the timestamping (as for example in the Sitaras)?

Yes.

The only use for CONFIG_NETWORK_PHY_TIMESTAMPING is the phyter driver.

   drivers/net/phy/dp83640.c

Thanks,
Richard

Re: [Linuxptp-users] [External] Re: PTP - MAC time

[Richard C. <ric...@gm...>]

On Mon, Apr 10, 2017 at 03:18:32PM +0000, Ian Thompson wrote:
> I still see the issue with 3.18 but I haven't yet seen it on 4.9. 
> Unfortunately, we have a proprietary driver for a device on the pcie bus which doesn't yet support 4.x kernels and it is this that generates (via an application) most of the network traffic.
> I might have to port all of the stmmac changes back to 3.18.

Maybe I wasn't clear enough in my answer, but the loss of one time
stamp is unfortunate but understandable, and it normally should be
tolerable.

The resetting of the PHC time (the cause of the 37 second error) in the
driver is just plain wrong, and you should fix that.
 
> If I add 37 seconds to getnstimeofday then the effect of the "glitch" is less pronounced. 
> Kernel 3.18 introduced timekeeping.c, with  timekeeping_get_tai_offset(), which I thought might give me the UTC offset but it returns 0 at the point I call it.
> Is there a call within the kernel to find the UTC offset?

The offset has to be provided to the kernel by user space by calling
adjtimex() with the ADJ_TAI mode set.

Thanks,
Richard

[Linuxptp-users] CONFIG_NETWORK_PHY_TIMESTAMPING

[Axel H. <aho...@gm...>]

Hi,

with regards to my other post regarding AM5728 the question came up, if
CONFIG_NETWORK_PHY_TIMESTAMPING has to be defined, even if timestamping is
done in the MAC (and not in the PHY)?

Or is it okay to leave CONFIG_NETWORK_PHY_TIMESTAMPING undefined in case the
MAC is doing the timestamping (as for example in the Sitaras)?

Thank you very much
Axel

Re: [Linuxptp-users] [External] Re: PTP - MAC time

[Ian T. <Ian...@pg...>]

Richard

Thanks for the response.
I now have one board running kernel 3.18 and another running kernel 4.9.

I still see the issue with 3.18 but I haven't yet seen it on 4.9. 
Unfortunately, we have a proprietary driver for a device on the pcie bus which doesn't yet support 4.x kernels and it is this that generates (via an application) most of the network traffic.
I might have to port all of the stmmac changes back to 3.18.

If I add 37 seconds to getnstimeofday then the effect of the "glitch" is less pronounced. 
Kernel 3.18 introduced timekeeping.c, with  timekeeping_get_tai_offset(), which I thought might give me the UTC offset but it returns 0 at the point I call it.
Is there a call within the kernel to find the UTC offset?

Regards
Ian T. 

-----Original Message-----
From: Richard Cochran [mailto:ric...@gm...] 
Sent: Thursday, April 06, 2017 12:49 AM
To: Ian Thompson
Cc: lin...@li...
Subject: [External] Re: [Linuxptp-users] PTP - MAC time

On Wed, Apr 05, 2017 at 02:34:14PM +0000, Ian Thompson wrote:
> Why is the time that gets put into the PTP registers in the STM MAC, Unix time rather than PTP time?

See below.

To you question from the other thread:

On Tue, Apr 04, 2017 at 03:45:16PM +0000, Ian Thompson wrote:
> Possibly following on from David’s post.
> 
> We have a system with 18 boards in a rack, each board has a Altera SoC with the STM Ethernet MAC connected via gigabit Ethernet to an Arista ptp-aware switch and then a Spectracom GrandMaster.
> The boards are running Linux kernel 3.15.0.

That HW puts the time stamps into the buffer descriptor, and so in theory it should never miss a time stamp.  This is most likely a driver bug.  Looking at the git log I see:

 v4.11-rc1~124^2~171^2~12 deeb637 net: stmmac: remove freesoftware address
     v4.9-rc7~33^2~33^2~1 ba1ffd7 stmmac: fix PTP support for GMAC4
     v4.9-rc7~33^2~33^2~2 d204205 stmmac: update the PTP header file
         v4.9-rc4~28^2~68 c30a70d stmmac: fix and review the ptp registration.
         v4.9-rc4~28^2~96 50756eb stmmac: fix an error code in stmmac_ptp_register()
         v4.9-rc1~28^2~10 7086605 stmmac: fix error check when init ptp
       v4.9-rc1~127^2~108 efee95f ptp_clock: future-proofing drivers against PTP subsystem becoming optional
   v4.1-rc1~128^2~100^2~5 e7ea55b ptp: stmmac: use helpers for converting ns to timespec.
   v4.1-rc1~128^2~119^2~6 3f6c465 ptp: stmmac: convert to the 64 bit get/set time methods.
        v3.17-rc5~41^2~38 5566401 stmmac: ptp: fix the reference clock
        v3.17-rc5~41^2~50 f95f404 stmmac: set ptp_clock to NULL while unregister
  v3.15-rc1~113^2~108^2~5 4986b4f0 ptp: drivers: set the number of programmable pins.
           v3.13-rc7~13^2 7cd0139 stmmac: Fix incorrect spinlock release and PTP cap detection.
       v3.10-rc1~66^2~195 32ceabc stmmac: improve/review and fix kernel-doc
   v3.10-rc1~66^2~327^2~1 92ba688 stmmac: add the support for PTP hw clock driver
   v3.10-rc1~66^2~327^2~2 891434b stmmac: add IEEE PTPv1 and PTPv2 support.

Especially ba1ffd7 looks suspicious.

> Apr  4 13:42:04 localhost user.info   ptp4l: [537.164] rms  123 max  599 freq   +255 +/-  39 delay  7362 +/-  48
> Apr  4 13:42:29 localhost user.err    ptp4l: [561.387] timed out while polling for tx timestamp
> Apr  4 13:42:29 localhost user.err    ptp4l: [561.387] increasing tx_timestamp_timeout may correct this issue, but it is likely caused by a driver bug
> Apr  4 13:42:29 localhost user.err    ptp4l: [561.387] port 1: send delay request failed
> Apr  4 13:42:29 localhost user.notice ptp4l: [561.387] port 1: SLAVE 
> to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED) Apr  4 13:42:45 localhost user.notice ptp4l: [577.388] port 1: FAULTY to LISTENING on FAULT_CLEARED
> Apr  4 13:42:45 localhost user.warn   ptp4l: [577.414] clockcheck: clock jumped backward or running slower than expected!
> Apr  4 13:42:45 localhost user.notice ptp4l: [577.414] port 1: new 
> foreign master 000cec.fffe.0a085d-1 Apr  4 13:42:47 localhost 
> user.notice ptp4l: [579.414] selected best master clock 
> 000cec.fffe.0a085d Apr  4 13:42:47 localhost user.notice ptp4l: [579.414] port 1: LISTENING to UNCALIBRATED on RS_SLAVE Apr  4 13:42:54 localhost user.notice ptp4l: [587.164] port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
> Apr  4 13:46:46 localhost user.info   ptp4l: [818.414] rms 2312500092 max 37000001557 freq   +246 +/- 250 delay  7358 +/-  46
> Apr  4 13:51:02 localhost user.info   ptp4l: [1074.413] rms  116 max  681 freq   +256 +/-  48 delay  7373 +/-  88
> 
> Does this imply that one lost delay request can do this, or is there a retry mechanism?

One lost delay request shouldn't introduct such a large error.  This is a driver bug.  Notice that the time error is 37 seconds, or the UTC/TAI offset.

When resetting the fault, ptp4l re-initializes HW time stamping.

The funtion, stmmac_hwtstamp_ioctl(), in

    drivers/net/ethernet/stmicro/stmmac/stmmac_main.c 

programs the system time (UTC) into the PHC every time HW time stamping is enabled.  It shouldn't do that.

> We have a lot of traffic leaving the boards but only PTP traffic 
> coming in. As we increase the off board transfer rates the problem 
> seems to occur more often.

That could indicate a driver or a HW issue, or both.

HTH,
Richard

Re: [Linuxptp-users] Strange behaviour on AM5728 board

[Axel H. <aho...@gm...>]

Hi Richard,

On Sat, Apr 08, 2017 at 12:57:44PM +0200, Richard Cochran wrote: 
> Or you can just hack the driver to print clk_get_rate(cpts->refclk).

Cool idea, did that and added this line to drivers/net/ethernet/ti/cpts.c
(and noticed who wrote the file) as last line in
static void cpts_clk_init() {
   ...
  dev_err(dev, "clk_get_rate(cpts->refclk) = %lu",
clk_get_rate(cpts->refclk));
}

Output:
[   12.012718] cpsw 48484000.ethernet: clk_get_rate(cpts->refclk) =
125000000

It's strange, because from reading CM_GMAC_GMAC_CLKCTRL (L3_CLK is selected
as a clock source) and studying the TRM of the AM5728 regarding what is
doing the boot ROM code I learned that the value should be 133MHz, but what
is printed in the output is 125MHz. Hum.

Okay, so I changed cpts_clock_shift = <29>; to cpts_clock_shift = <28>; in
arch/arm/boot/dts/dra7.dtsi.

Unfortunately that didn't help either, while it changed the ptp4l output:
user@arm:~/linuxptp-1.8$ sudo ./ptp4l -i eth1 -s -m -q
[sudo] password for user:
ptp4l[76.441]: selected /dev/ptp0 as PTP clock
ptp4l[76.463]: port 1: INITIALIZING to LISTENING on INITIALIZE
ptp4l[76.463]: port 0: INITIALIZING to LISTENING on INITIALIZE
ptp4l[76.463]: port 1: link up
ptp4l[77.442]: port 1: new foreign master 0050c2.fffe.39e7e1-8
ptp4l[79.442]: selected best master clock 000b72.fffe.0587fa
ptp4l[79.442]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[80.345]: port 1: minimum delay request interval 2^3
ptp4l[80.942]: rms 9552511550405 max 9552793492018 freq +500000 +/- 500000
delay -227782976 +/- 776530
ptp4l[81.942]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[82.442]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[82.941]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[83.442]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[83.941]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[84.441]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[84.941]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[85.441]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[85.941]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[86.441]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[86.941]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[87.441]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[87.941]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[88.441]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[88.941]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[89.441]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[89.941]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[90.441]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[90.941]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[91.441]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[91.941]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[92.441]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[92.941]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[93.441]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[93.941]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[94.441]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[94.941]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[95.441]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[95.441]: rms 16042529614 max 16321529908 freq +1000000 +/-   0 delay
-227006446 +/-   0
ptp4l[95.941]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[96.441]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[96.441]: rms 17168088472 max 17447241720 freq +1000000 +/-   0
ptp4l[96.941]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[97.441]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[97.441]: rms 18293668079 max 18572965183 freq +1000000 +/-   0
ptp4l[97.941]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[98.441]: clockcheck: clock jumped forward or running faster than
expected!
ptp4l[98.441]: rms 19419257020 max 19698670451 freq +1000000 +/-   0

and so on...

I will now again contact the manufacturer and tell him what I found out,
hoping that the manufacturer can fix the issue or at least shed more light
on what's going on.

Many many thanks and best regards
Axel

Re: [Linuxptp-users] Strange behaviour on AM5728 board

[Richard C. <ric...@gm...>]

On Sat, Apr 08, 2017 at 11:39:06AM +0200, Axel Holzinger wrote:
> I dived into device tree files and couldn't yet find out how the
> manufacturer configures CPTS_RFT_CLK. I have the impression that it's not
> running at all. So I contacted them and asked if they can point me in the
> right direction. I'll report back as soon as I have further information.

Or you can just hack the driver to print clk_get_rate(cpts->refclk).

Thanks,
Richard

Re: [Linuxptp-users] Strange behaviour on AM5728 board

[Axel H. <aho...@gm...>]

Hi Richard,

> On Thu, Apr 06, 2017 at 09:30:27PM +0200, Richard Cochran wrote:
> Sorry, I didn't mean that those are the correct values.  Those two
> properties define the clock constants:
> 
> 	nanoseconds = ticks * cpts_clock_mult / 2^cpts_clock_shift
> 
> The values from the beaglebone are
> 
> 	cpts_clock_mult = <0x80000000>;
> 	cpts_clock_shift = <29>;
> 
> and so
> 
> 	nanoseconds = ticks * 2^31 / 2^29
> 	nanoseconds = ticks * 4
> 
> which is correct if the input clock is 250 MHz.  If your SoC uses a
> different clock, then the values in the DTS are wrong.

Than you very much for the detailed explanation. Very helpful!

I dived into device tree files and couldn't yet find out how the
manufacturer configures CPTS_RFT_CLK. I have the impression that it's not
running at all. So I contacted them and asked if they can point me in the
right direction. I'll report back as soon as I have further information.

> > Is there anything else I can check?
> 
> The two values are calculated automatically as of commit:
> 
>  v4.10-rc1~202^2~61^2~1 88f0f0b net: ethernet: ti: cpts: calc mult and
shift
> from refclk freq
> 
> You can try kernel v4.10 and remove those two properties from the DTS.
> It should "just work" then.

That's very good news. I cloned Tis v4.10.8, but currently struggle with the
manufacturers way of configuring (which is only supported by the
manufacturer for the v4.1.6). I'm also going to report back if I managed to
get v4.10.8 running.

Tnak you very much for your support!
Axel

Re: [Linuxptp-users] Strange behaviour on AM5728 board

[Richard C. <ric...@gm...>]

On Thu, Apr 06, 2017 at 07:01:47PM +0200, Axel Holzinger wrote:
> So the values for cpts_clock_mult and cpts_clock_shift are configured like
> you proposed.

Sorry, I didn't mean that those are the correct values.  Those two
properties define the clock constants:

	nanoseconds = ticks * cpts_clock_mult / 2^cpts_clock_shift

The values from the beaglebone are

	cpts_clock_mult = <0x80000000>;
	cpts_clock_shift = <29>;

and so

	nanoseconds = ticks * 2^31 / 2^29
	nanoseconds = ticks * 4

which is correct if the input clock is 250 MHz.  If your SoC uses a
different clock, then the values in the DTS are wrong.
 
> Is there anything else I can check?

The two values are calculated automatically as of commit:

 v4.10-rc1~202^2~61^2~1 88f0f0b net: ethernet: ti: cpts: calc mult and shift from refclk freq

You can try kernel v4.10 and remove those two properties from the DTS.
It should "just work" then.

HTH,
Richard

Re: [Linuxptp-users] Strange behaviour on AM5728 board

[Axel H. <aho...@gm...>]

Hi Richard,

thank you for your prompt reply.

> On Thu, Apr 06, 2017 at 04:58:18PM +0200, Richard Cochran wrote:
> > 
> > On Thu, Apr 06, 2017 at 01:45:19PM +0200, Axel Holzinger wrote:
> > To me it looks like there is trouble correctly adjusting the frequency
of
> > the PHC, it's always remaining +1000000, but also delays are ridicously
high
> > and negative (that doesn't make sense, does it?).
> 
> Does that SoC use the CPTS?

I do think so.

> Probably the input clock is wrong.  Check the data sheet and the device
tree
> values for:
> 
> 	cpts_clock_mult = <0x80000000>;
> 	cpts_clock_shift = <29>;

Well I'm no dts expert at all. I found in
ti-linux-kernel/arch/arm/boot/dts/am57xx-titanium.dts:

#include "dra74x.dtsi"

In dra74x.dtsi:

#include "dra7.dtsi"

In dra7.dtsi:

/ {
    #address-cells = <1>;
    #size-cells = <1>;

    compatible = "ti,dra7xx";
    interrupt-parent = <&crossbar_mpu>;

    /*
        * XXX: Use a flat representation of the SOC interconnect.
        * The real OMAP interconnect network is quite complex.
        * Since it will not bring real advantage to represent that in DT for
        * the moment, just use a fake OCP bus entry to represent the whole
bus
        * hierarchy.
        */
    ocp {
        compatible = "ti,dra7-l3-noc", "simple-bus";
        #address-cells = <1>;
        #size-cells = <1>;
        ranges;
        ti,hwmods = "l3_main_1", "l3_main_2";
        reg = <0x44000000 0x1000000>,
                <0x45000000 0x1000>;
        interrupts-extended = <&crossbar_mpu GIC_SPI 4 IRQ_TYPE_LEVEL_HIGH>,
                        <&wakeupgen GIC_SPI 10 IRQ_TYPE_LEVEL_HIGH>;

        ...

        mac: ethernet@48484000 {
            compatible = "ti,dra7-cpsw","ti,cpsw";
            ti,hwmods = "gmac";
            clocks = <&dpll_gmac_ck>, <&gmac_gmii_ref_clk_div>;
            clock-names = "fck", "cpts";
            cpdma_channels = <8>;
            ale_entries = <1024>;
            bd_ram_size = <0x2000>;
            no_bd_ram = <0>;
            rx_descs = <64>;
            mac_control = <0x20>;
            slaves = <2>;
            active_slave = <0>;
            cpts_clock_mult = <0x80000000>;
            cpts_clock_shift = <29>;
            reg = <0x48484000 0x1000
                    0x48485200 0x2E00>;
            #address-cells = <1>;
            #size-cells = <1>;

            /*
                * Do not allow gating of cpsw clock as workaround
                * for errata i877. Keeping internal clock disabled
                * causes the device switching characteristics
                * to degrade over time and eventually fail to meet
                * the data manual delay time/skew specs.
                */
            ti,no-idle;

            /*
                * rx_thresh_pend
                * rx_pend
                * tx_pend
                * misc_pend
                */
            interrupts = <GIC_SPI 334 IRQ_TYPE_LEVEL_HIGH>,
                        <GIC_SPI 335 IRQ_TYPE_LEVEL_HIGH>,
                        <GIC_SPI 336 IRQ_TYPE_LEVEL_HIGH>,
                        <GIC_SPI 337 IRQ_TYPE_LEVEL_HIGH>;
            ranges;
            status = "disabled";

            davinci_mdio: mdio@48485000 {
                compatible = "ti,davinci_mdio";
                #address-cells = <1>;
                #size-cells = <0>;
                ti,hwmods = "davinci_mdio";
                bus_freq = <1000000>;
                reg = <0x48485000 0x100>;
                clocks = <&gmac_gmii_ref_clk_div>;
                clock-names = "fck";
            };

            cpsw_emac0: slave@48480200 {
                /* Filled in by U-Boot */
                mac-address = [ 00 00 00 00 00 00 ];
            };

            cpsw_emac1: slave@48480300 {
                /* Filled in by U-Boot */
                mac-address = [ 00 00 00 00 00 00 ];
            };

            phy_sel: cpsw-phy-sel@4a002554 {
                compatible = "ti,dra7xx-cpsw-phy-sel";
                reg= <0x4a002554 0x4>;
                reg-names = "gmii-sel";
            };
        };

        ...
    };

    ...
};

So the values for cpts_clock_mult and cpts_clock_shift are configured like
you proposed.

Is there anything else I can check?

Thanks
Axel

Re: [Linuxptp-users] Strange behaviour on AM5728 board

[Richard C. <ric...@gm...>]

On Thu, Apr 06, 2017 at 01:45:19PM +0200, Axel Holzinger wrote:
> To me it looks like there is trouble correctly adjusting the frequency of
> the PHC, it's always remaining +1000000, but also delays are ridicously high
> and negative (that doesn't make sense, does it?).

Does that SoC use the CPTS?  Probably the input clock is wrong.  Check
the data sheet and the device tree values for:

	cpts_clock_mult = <0x80000000>;
	cpts_clock_shift = <29>;

Thanks,
Richard

[Linuxptp-users] Strange behaviour on AM5728 board

[Axel H. <aho...@gm...>]

Hi linuxptp community,

I could put my hands on a TI AM5728 box (Elesar Titanium) running Linux arm
4.1.6-01294--g8406366 #6 SMP PREEMPT armv7l (Debian Jessy) and ethtool is
reporting this:
user@arm:~/linuxptp-1.8$ sudo ethtool -T eth1
Time stamping parameters for eth1:
Capabilities:
        hardware-transmit     (SOF_TIMESTAMPING_TX_HARDWARE)
        software-transmit     (SOF_TIMESTAMPING_TX_SOFTWARE)
        hardware-receive      (SOF_TIMESTAMPING_RX_HARDWARE)
        software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)
        software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
        hardware-raw-clock    (SOF_TIMESTAMPING_RAW_HARDWARE)
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
        off                   (HWTSTAMP_TX_OFF)
        on                    (HWTSTAMP_TX_ON)
Hardware Receive Filter Modes:
        none                  (HWTSTAMP_FILTER_NONE)
        ptpv2-event           (HWTSTAMP_FILTER_PTP_V2_EVENT)

I'm observing strange behaviour of ptp4l:
user@arm:~/linuxptp-1.8$ sudo ./ptp4l -i eth1 -s -m -q
ptp4l[296.574]: selected /dev/ptp0 as PTP clock
ptp4l[296.593]: port 1: INITIALIZING to LISTENING on INITIALIZE
ptp4l[296.593]: port 0: INITIALIZING to LISTENING on INITIALIZE
ptp4l[296.593]: port 1: link up
ptp4l[297.202]: port 1: new foreign master 0050c2.fffe.39e7e1-8
ptp4l[299.202]: selected best master clock 000b72.fffe.0587fa
ptp4l[299.202]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[299.301]: port 1: minimum delay request interval 2^3
ptp4l[300.202]: rms 9169381730255 max 9169405598321 freq +500000 +/- 500000
delay -15782702 +/-   0
ptp4l[302.202]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[302.702]: rms 140849798 max 155677350 freq +1000000 +/-   0 delay
-14387879 +/-   0
ptp4l[303.702]: rms 202019230 max 217113647 freq +1000000 +/-   0 delay
-12993056 +/-   0
ptp4l[304.702]: rms 264705365 max 279944980 freq +1000000 +/-   0
ptp4l[305.702]: rms 322793382 max 336658462 freq +1000000 +/-   0 delay
-8404950 +/- 1529368
ptp4l[306.702]: rms 384104196 max 399489358 freq +1000000 +/-   0
ptp4l[307.702]: rms 446890503 max 462320408 freq +1000000 +/-   0
ptp4l[308.702]: rms 509687509 max 525151217 freq +1000000 +/-   0
ptp4l[309.702]: rms 572309716 max 587627646 freq +1000000 +/-   0 delay
-6521028 +/-   0
ptp4l[310.702]: rms 634949321 max 650458738 freq +1000000 +/-   0
ptp4l[311.702]: rms 697764821 max 713293606 freq +1000000 +/-   0
ptp4l[312.702]: rms 760581017 max 776124411 freq +1000000 +/-   0
ptp4l[313.702]: rms 823400790 max 838959051 freq +1000000 +/-   0
ptp4l[314.701]: rms 886218407 max 901783909 freq +1000000 +/-   0
ptp4l[315.701]: rms 949038000 max 964613935 freq +1000000 +/-   0
ptp4l[316.701]: rms 1011862805 max 1027444991 freq +1000000 +/-   0
ptp4l[317.701]: rms 1074684725 max 1090275855 freq +1000000 +/-   0
ptp4l[318.701]: rms 1137696718 max 1153460190 freq +1000000 +/-   0 delay
-6875582 +/-   0
ptp4l[319.701]: rms 1200688978 max 1216292064 freq +1000000 +/-   0
ptp4l[320.701]: rms 1263514719 max 1279122947 freq +1000000 +/-   0
ptp4l[321.701]: rms 1326341090 max 1341953920 freq +1000000 +/-   0
ptp4l[322.701]: rms 1389167792 max 1404784827 freq +1000000 +/-   0
ptp4l[323.701]: rms 1451994939 max 1467616429 freq +1000000 +/-   0
ptp4l[324.701]: rms 1514822314 max 1530446698 freq +1000000 +/-   0

To me it looks like there is trouble correctly adjusting the frequency of
the PHC, it's always remaining +1000000, but also delays are ridicously high
and negative (that doesn't make sense, does it?).

Has anybody ever seen such behaviour and/or has experience with the AM5728
(or similar)?

Thank you!
Axel

Re: [Linuxptp-users] clockcheck jumps forwards and backwards

[David M. <dav...@gm...>]

Gah, sorry - I goofed and pulled the 'ps' output from a different box than
the config. That indeed is the domain and does match the conf for any given
machine, but not when I mix'n'match (ttl, domain, and transport are
site-specific).

>From the problematic box:

bash-4.3# cat /etc/ptp4l.conf
[global]
slaveOnly 1
summary_interval 6
priority1 255
udp_ttl 16

[ma2]

and

S root     20462     1  0  80   0  2452  2920 poll_s   1 Mar03 ?
 00:58:13 /usr/sbin/ptp4l -f /etc/ptp4l.conf
S root     20468     1  0  80   0  1068  2902 hrtime   0 Mar03 ?
 00:01:17 /usr/sbin/phc2sys -a -r -u 64

So the config file and cmdlines look when the default 0 domain is
configured (in this case taken from the problematic box)

Apologies for the mixup,
Dave

On Thu, Apr 6, 2017 at 4:01 PM, Richard Cochran <ric...@gm...>
wrote:

> On Wed, Apr 05, 2017 at 09:18:09AM +1000, David Mirabito wrote:
> > The config is:
> >
> > [global]
> > slaveOnly 1
> > summary_interval 6
> > priority1 255
> >
> > [ma2]
> >
> > And running as:
> >    /usr/sbin/ptp4l -f /etc/ptp4l.conf
> >    /usr/sbin/phc2sys -a -r -u 64 -n 5
>
> Why are you using domain number 5 for phc2sys (-n 5) ?
>
> That would cause ptp4l to ignore all of phc2sys's queries, unless
> ptp4l were also using that domain.
>
> Or did you actually use -N 5 ?
>
> Thanks,
> Richard
>
>

Re: [Linuxptp-users] clockcheck jumps forwards and backwards

[Richard C. <ric...@gm...>]

On Wed, Apr 05, 2017 at 09:18:09AM +1000, David Mirabito wrote:
> The config is:
> 
> [global]
> slaveOnly 1
> summary_interval 6
> priority1 255
> 
> [ma2]
> 
> And running as:
>    /usr/sbin/ptp4l -f /etc/ptp4l.conf
>    /usr/sbin/phc2sys -a -r -u 64 -n 5

Why are you using domain number 5 for phc2sys (-n 5) ?

That would cause ptp4l to ignore all of phc2sys's queries, unless
ptp4l were also using that domain.

Or did you actually use -N 5 ?

Thanks,
Richard

Re: [Linuxptp-users] PTP - MAC time

[Richard C. <ric...@gm...>]

On Wed, Apr 05, 2017 at 02:34:14PM +0000, Ian Thompson wrote:
> Why is the time that gets put into the PTP registers in the STM MAC, Unix time rather than PTP time?

See below.

To you question from the other thread:

On Tue, Apr 04, 2017 at 03:45:16PM +0000, Ian Thompson wrote:
> Possibly following on from David’s post.
> 
> We have a system with 18 boards in a rack, each board has a Altera SoC with the STM Ethernet MAC connected via gigabit Ethernet to an Arista ptp-aware switch and then a Spectracom GrandMaster.
> The boards are running Linux kernel 3.15.0.

That HW puts the time stamps into the buffer descriptor, and so in
theory it should never miss a time stamp.  This is most likely a
driver bug.  Looking at the git log I see:

 v4.11-rc1~124^2~171^2~12 deeb637 net: stmmac: remove freesoftware address
     v4.9-rc7~33^2~33^2~1 ba1ffd7 stmmac: fix PTP support for GMAC4
     v4.9-rc7~33^2~33^2~2 d204205 stmmac: update the PTP header file
         v4.9-rc4~28^2~68 c30a70d stmmac: fix and review the ptp registration.
         v4.9-rc4~28^2~96 50756eb stmmac: fix an error code in stmmac_ptp_register()
         v4.9-rc1~28^2~10 7086605 stmmac: fix error check when init ptp
       v4.9-rc1~127^2~108 efee95f ptp_clock: future-proofing drivers against PTP subsystem becoming optional
   v4.1-rc1~128^2~100^2~5 e7ea55b ptp: stmmac: use helpers for converting ns to timespec.
   v4.1-rc1~128^2~119^2~6 3f6c465 ptp: stmmac: convert to the 64 bit get/set time methods.
        v3.17-rc5~41^2~38 5566401 stmmac: ptp: fix the reference clock
        v3.17-rc5~41^2~50 f95f404 stmmac: set ptp_clock to NULL while unregister
  v3.15-rc1~113^2~108^2~5 4986b4f0 ptp: drivers: set the number of programmable pins.
           v3.13-rc7~13^2 7cd0139 stmmac: Fix incorrect spinlock release and PTP cap detection.
       v3.10-rc1~66^2~195 32ceabc stmmac: improve/review and fix kernel-doc
   v3.10-rc1~66^2~327^2~1 92ba688 stmmac: add the support for PTP hw clock driver
   v3.10-rc1~66^2~327^2~2 891434b stmmac: add IEEE PTPv1 and PTPv2 support.

Especially ba1ffd7 looks suspicious.

> Apr  4 13:42:04 localhost user.info   ptp4l: [537.164] rms  123 max  599 freq   +255 +/-  39 delay  7362 +/-  48
> Apr  4 13:42:29 localhost user.err    ptp4l: [561.387] timed out while polling for tx timestamp
> Apr  4 13:42:29 localhost user.err    ptp4l: [561.387] increasing tx_timestamp_timeout may correct this issue, but it is likely caused by a driver bug
> Apr  4 13:42:29 localhost user.err    ptp4l: [561.387] port 1: send delay request failed
> Apr  4 13:42:29 localhost user.notice ptp4l: [561.387] port 1: SLAVE to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
> Apr  4 13:42:45 localhost user.notice ptp4l: [577.388] port 1: FAULTY to LISTENING on FAULT_CLEARED
> Apr  4 13:42:45 localhost user.warn   ptp4l: [577.414] clockcheck: clock jumped backward or running slower than expected!
> Apr  4 13:42:45 localhost user.notice ptp4l: [577.414] port 1: new foreign master 000cec.fffe.0a085d-1
> Apr  4 13:42:47 localhost user.notice ptp4l: [579.414] selected best master clock 000cec.fffe.0a085d
> Apr  4 13:42:47 localhost user.notice ptp4l: [579.414] port 1: LISTENING to UNCALIBRATED on RS_SLAVE
> Apr  4 13:42:54 localhost user.notice ptp4l: [587.164] port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
> Apr  4 13:46:46 localhost user.info   ptp4l: [818.414] rms 2312500092 max 37000001557 freq   +246 +/- 250 delay  7358 +/-  46
> Apr  4 13:51:02 localhost user.info   ptp4l: [1074.413] rms  116 max  681 freq   +256 +/-  48 delay  7373 +/-  88
> 
> Does this imply that one lost delay request can do this, or is there a retry mechanism?

One lost delay request shouldn't introduct such a large error.  This
is a driver bug.  Notice that the time error is 37 seconds, or the
UTC/TAI offset.

When resetting the fault, ptp4l re-initializes HW time stamping.

The funtion, stmmac_hwtstamp_ioctl(), in

    drivers/net/ethernet/stmicro/stmmac/stmmac_main.c 

programs the system time (UTC) into the PHC every time HW time
stamping is enabled.  It shouldn't do that.

> We have a lot of traffic leaving the boards but only PTP traffic
> coming in. As we increase the off board transfer rates the problem
> seems to occur more often.

That could indicate a driver or a HW issue, or both.

HTH,
Richard

Re: [Linuxptp-users] [External] Re: clockcheck jumps forwards and backwards

[Richard C. <ric...@gm...>]

Ian,

The problem you are seeing shares the same symptom as in the original
post, but the root cause is different because you have totally
different HW.  Let's take this discussion into its own thread, please.
I'll start by answering your other recent post.

Thanks,
Richard

[Linuxptp-users] PTP - MAC time

[Ian T. <Ian...@pg...>]

All

I have another question where I reveal my ignorance of IEEE-1588.

Why is the time that gets put into the PTP registers in the STM MAC, Unix time rather than PTP time?
Am I missing a configuration setting for this?

Regards
Ian T.