Menu
▾
▴
linuxptp-devel
|
From: Miroslav L. <mli...@re...> - 2021-03-15 16:56:49
|
As I understand it, there is currently no explicit support for SyncE in the mainline kernel and linuxptp. For example, there is no access to the Synchronization Status Message (SSM), so ptp4l cannot know if SyncE is active on a port. IIRC some people use SyncE with linuxptp by disabling the frequency control. I assume they set up everything manually with some 3rd party HW-specific tools using custom ioctls, or maybe it's just a client-only use case for linuxptp. There seems to be a growing interest for SyncE, but it's not clear to me how the functionality needed for controlling and monitoring SyncE is supposed to be split between the firmware, driver, kernel, and ptp4l. In 1588-2019 there is an L1_SYNC TLV specified. Does anyone know what is actually needed for the kernel and linuxptp to support it properly, e.g. on a boundary clock? Any pointers? -- Miroslav Lichvar |
|
From: Keller, J. E <jac...@in...> - 2021-03-15 17:14:04
|
> -----Original Message----- > From: Miroslav Lichvar <mli...@re...> > Sent: Monday, March 15, 2021 9:57 AM > To: lin...@li... > Subject: [Linuxptp-devel] SyncE support > > As I understand it, there is currently no explicit support for SyncE > in the mainline kernel and linuxptp. For example, there is no access > to the Synchronization Status Message (SSM), so ptp4l cannot know if > SyncE is active on a port. > Right, this appears to be the general case today > IIRC some people use SyncE with linuxptp by disabling the frequency > control. I assume they set up everything manually with some 3rd party > HW-specific tools using custom ioctls, or maybe it's just a > client-only use case for linuxptp. > I think the only examples I've seen were client only... > There seems to be a growing interest for SyncE, but it's not clear to > me how the functionality needed for controlling and monitoring SyncE > is supposed to be split between the firmware, driver, kernel, and > ptp4l. Right. I've heard a few folks here talking about it, but I don't know the plans off hand. Given that some parts of it are device specific, it would seem to me like we need some extensions of the PTP interface that drivers which support SyncE hardware could implement, and then we could extend the LinuxPTP stack to include monitoring synce, perhaps as part of ptp4l itself or as part of a separate application? > > In 1588-2019 there is an L1_SYNC TLV specified. Does anyone know what > is actually needed for the kernel and linuxptp to support it properly, > e.g. on a boundary clock? > > Any pointers? > > -- > Miroslav Lichvar > > > > _______________________________________________ > Linuxptp-devel mailing list > Lin...@li... > https://lists.sourceforge.net/lists/listinfo/linuxptp-devel |
|
From: Frantisek R. <Fra...@po...> - 2021-03-15 21:53:05
|
Dear gentlemen, thanks for opening this interesting topic... Is there any stock NIC (silicon, board), with an open-source Linux driver, that would support SyncE off the shelf in the NIC hardware? Other than that... SyncE sounds like a pretty self-contained L1 feature, at least considering just a single NIC port. Obviously the routing of clock signals among possibly several ports in a system and a local master oscillator - that would be a more complicated, proprietary story. But at the level of a single port... what do you need? SyncE enable / disable, and select master vs. slave role? It would take a fairly simple extension to some existing API - makes me wonder which one: MII, Ethtool, Netlink, or maybe the timestamping API? I'd guess Ethtool or Netlink would be the right level of abstraction. Or some nodes in procfs or sysfs :-) MII is too HW-specific and the timestamping stuff operating on top of setsockopt() feels a little too detached from hardware. And yes there is some additional messaging, I understand in-band in Ethernet, which would probably have to be handled by ptp4l software, unless offloaded to the kernel-space driver or hardware... Frank Rysanek |
|
From: Miroslav L. <mli...@re...> - 2021-03-16 10:25:18
|
On Mon, Mar 15, 2021 at 10:40:08PM +0100, Frantisek Rysanek wrote: > Dear gentlemen, > thanks for opening this interesting topic... > > Is there any stock NIC (silicon, board), with an open-source Linux > driver, that would support SyncE off the shelf in the NIC hardware? I didn't find anything in the current mainline kernel. There may be 3rd party open source drivers. In the Intel's igc driver I saw few SYNCE registers defined, but no code using them. > But at the level of a single port... what do you need? > SyncE enable / disable, and select master vs. slave role? At minimum, get the current state of the link with information like whether the port is TX/RX coherent to be able to provide the L1_SYNC TLV. > And yes there is some additional messaging, I understand in-band in > Ethernet, which would probably have to be handled by ptp4l software, > unless offloaded to the kernel-space driver or hardware... I'd expect the OSSP/ESMC part to be handled in the firmware. I think ptp4l should just be able to see the current status and maybe flip some bits to configure it. -- Miroslav Lichvar |
|
From: Keller, J. E <jac...@in...> - 2021-03-18 15:17:17
|
> -----Original Message----- > From: Frantisek Rysanek <Fra...@po...> > Sent: Monday, March 15, 2021 2:40 PM > To: lin...@li... > Subject: Re: [Linuxptp-devel] SyncE support > > Dear gentlemen, > thanks for opening this interesting topic... > > Is there any stock NIC (silicon, board), with an open-source Linux > driver, that would support SyncE off the shelf in the NIC hardware? > I believe that SyncE is supported by the E800 series 100GB hardware from Intel, but there's no existing driver/software support. > Other than that... SyncE sounds like a pretty self-contained L1 > feature, at least considering just a single NIC port. Obviously the > routing of clock signals among possibly several ports in a system and > a local master oscillator - that would be a more complicated, > proprietary story. > But at the level of a single port... what do you need? > SyncE enable / disable, and select master vs. slave role? > It would take a fairly simple extension to some existing API - makes > me wonder which one: MII, Ethtool, Netlink, or maybe the timestamping > API? I'd guess Ethtool or Netlink would be the right level of > abstraction. Or some nodes in procfs or sysfs :-) > MII is too HW-specific and the timestamping stuff operating on top of > setsockopt() feels a little too detached from hardware. > > And yes there is some additional messaging, I understand in-band in > Ethernet, which would probably have to be handled by ptp4l software, > unless offloaded to the kernel-space driver or hardware... > > Frank Rysanek > > > _______________________________________________ > Linuxptp-devel mailing list > Lin...@li... > https://lists.sourceforge.net/lists/listinfo/linuxptp-devel |
|
From: Frantisek R. <Fra...@po...> - 2021-03-16 15:06:24
|
On 16 Mar 2021 at 11:25, Miroslav Lichvar wrote: > > In the Intel's igc driver I saw few SYNCE registers defined, > but no code using them. > Whoa. igc ? oh there's an i225... didn't know about this one, thanks for the pointer, this definitely looks promising :-) Looks like a successor to the i210 generation. If I read the gossip correctly, the first generation of i225 are in circulation on some gaming motherboards from the first half of 2020, and have a hardware bug, something to do with the inter-packet gap. https://www.techpowerup.com/266335/intel-i225-foxville-2-5gbe-phy-has- a-flaw-affecting-performance-rocket-lake-s-2h-2020-production-confirme d A version 2 of the silicon, supposedly produced since Q4/2020, has this bug fixed. There's no public datasheet, just a data brief and a spec update (a list of SKU's + the erratum for v1 inter-packet gap bug.) https://www.intel.com/content/www/us/en/design/products-and-solutions/ networking-and-io/ethernet-controller-i225/technical-library.html And, I can actually see a stand-alone PCI-e board mentioned at ark.intel.com - the i225-T1 : https://ark.intel.com/content/www/us/en/ark/products/211808/intel-ethe rnet-network-adapter-i225-t1.html I cannot see this in the price lists at disties yet, will keep an eye on that one :-) The board may surface sooner from HP, Lenovo or Dell... There are no photoes yet - I'm curious if it has an SDP pin header, similar to the i210-T1. And this one looks cool too: http://www.commell.com.tw/Product/Peripheral/PCI%20Express%20mini%20ca rd/MPX-225.HTM Frank |
|
From: Miroslav L. <mli...@re...> - 2021-03-16 15:59:36
|
On Tue, Mar 16, 2021 at 03:52:36PM +0100, Frantisek Rysanek wrote: > On 16 Mar 2021 at 11:25, Miroslav Lichvar wrote: > > > > In the Intel's igc driver I saw few SYNCE registers defined, > > but no code using them. > > > Whoa. igc ? oh there's an i225... didn't know about this one, thanks > for the pointer, this definitely looks promising :-) > Looks like a successor to the i210 generation. I'm sorry for misleading you. I meant the ice driver (speeds up to 100Gb). I didn't see an out-of-tree version of the igc driver. The I225 might be an interesting NIC for experiments for a different reason. It seems it supports the PCIe Precision Time Measurement (PTM) feature. It is a hardware implementation of an NTP-like protocol over PCIe, which should allow a highly accurate synchronization of the system clock, avoiding the asymmetry on PCIe. It is not supported in the igc driver yet, but there were some patches submitted for it. I measured the PCIe asymmetry with the I210 on few different boards+CPUs and it changed a lot (between about -100ns and 100ns), so I think PTM could make a significant improvement. -- Miroslav Lichvar |
|
From: Geva, E. <ere...@si...> - 2021-03-16 17:47:03
|
On 16/03/2021 16:59, Miroslav Lichvar wrote: > On Tue, Mar 16, 2021 at 03:52:36PM +0100, Frantisek Rysanek wrote: >> On 16 Mar 2021 at 11:25, Miroslav Lichvar wrote: >>> >>> In the Intel's igc driver I saw few SYNCE registers defined, >>> but no code using them. >>> >> Whoa. igc ? oh there's an i225... didn't know about this one, thanks >> for the pointer, this definitely looks promising :-) >> Looks like a successor to the i210 generation. > > I'm sorry for misleading you. I meant the ice driver (speeds up to > 100Gb). I didn't see an out-of-tree version of the igc driver. > > The I225 might be an interesting NIC for experiments for a different > reason. It seems it supports the PCIe Precision Time Measurement (PTM) > feature. It is a hardware implementation of an NTP-like protocol over > PCIe, which should allow a highly accurate synchronization of the > system clock, avoiding the asymmetry on PCIe. It is not supported in > the igc driver yet, but there were some patches submitted for it. > > I measured the PCIe asymmetry with the I210 on few different > boards+CPUs and it changed a lot (between about -100ns and 100ns), so > I think PTM could make a significant improvement. > The "Precision Time Measurement" sound a cool feature. The PCI is probably asymmetric. As long as the traffic is not overflow (less then 99%), The transfer time of the PCI have a limit of ~120 ms (on my PC). Using hardware timestamps, the timestamps are always taken on the NIC hardware, before the PCI. So, the PCI should not change the measurement. The i210 does have inter gap issue, but it should not be relevant for PTP measure. I did some testing with TSN streaming with i210. By adding guards on the i210 drivers to over come the inter gap issue, I manage to get a precision of less then 20 ns on the streams. My team did notice that the oscillator of the i210 was not precise enough. We use the i210 as client (slave mode), with a good master. The results should be higher. Erez |
|
From: Keller, J. E <jac...@in...> - 2021-03-18 15:19:54
|
> -----Original Message----- > From: Miroslav Lichvar <mli...@re...> > Sent: Tuesday, March 16, 2021 8:59 AM > To: lin...@li... > Subject: Re: [Linuxptp-devel] SyncE support > > On Tue, Mar 16, 2021 at 03:52:36PM +0100, Frantisek Rysanek wrote: > > On 16 Mar 2021 at 11:25, Miroslav Lichvar wrote: > > > > > > In the Intel's igc driver I saw few SYNCE registers defined, > > > but no code using them. > > > > > Whoa. igc ? oh there's an i225... didn't know about this one, thanks > > for the pointer, this definitely looks promising :-) > > Looks like a successor to the i210 generation. > > I'm sorry for misleading you. I meant the ice driver (speeds up to > 100Gb). I didn't see an out-of-tree version of the igc driver. Yep, the ice hardware is supposed to have SyncE support, though no software here yet :( Not sure about igc. > > The I225 might be an interesting NIC for experiments for a different > reason. It seems it supports the PCIe Precision Time Measurement (PTM) > feature. It is a hardware implementation of an NTP-like protocol over > PCIe, which should allow a highly accurate synchronization of the > system clock, avoiding the asymmetry on PCIe. It is not supported in > the igc driver yet, but there were some patches submitted for it. > > I measured the PCIe asymmetry with the I210 on few different > boards+CPUs and it changed a lot (between about -100ns and 100ns), so > I think PTM could make a significant improvement. > igc should support PTM, but I'm not sure why the patches haven't landed. I'm not very well connected to the igc team. > -- > Miroslav Lichvar > > > > _______________________________________________ > Linuxptp-devel mailing list > Lin...@li... > https://lists.sourceforge.net/lists/listinfo/linuxptp-devel |
|
From: Frantisek R. <Fra...@po...> - 2021-03-16 22:38:40
|
On 16 Mar 2021 at 16:59, Miroslav Lichvar wrote: > > I'm sorry for misleading you. I meant the ice driver (speeds up to > 100Gb). I didn't see an out-of-tree version of the igc driver. > Okay, thanks for that update... Still I'm curious what news the i225 may bring in terms of e.g. GPIO timestamping resolution. Does it get better than those 8 ns I got used to with the i210 etc. > The I225 might be an interesting NIC for experiments for a different > reason. It seems it supports the PCIe Precision Time Measurement (PTM) > feature. It is a hardware implementation of an NTP-like protocol over > PCIe, which should allow a highly accurate synchronization of the > system clock, avoiding the asymmetry on PCIe. It is not supported in > the igc driver yet, but there were some patches submitted for it. > > I measured the PCIe asymmetry with the I210 on few different > boards+CPUs and it changed a lot (between about -100ns and 100ns), so > I think PTM could make a significant improvement. > Erez Geva was quicker to respond, and I agree with him: the packet timestamping is done by the NIC HW, so unless you have an application where you need accurate timing all the way to the software-based system clock, the determinism of PCIe bus latencies should not matter much... I understand that you being a RedHat insider, a principal maintainer of Chrony and a maintainer of the RedHat linuxptp RPM, your application is precisely getting the system clock as solid as theoretically possible :-) So far, I've only ever considered PTP and its breathtaking accuracy to be any use in dedicated hardware: switches with HW PTP support including a central stable oscillator, or pure slaves that syntonize a *stable* local hardware clock to the NIC's PHC and use that precise HW clock for some further, thoroughly HW-implemented purpose... As for getting the Linux system timebase precise, I guess the random component to ISR latency has several contributing factors, the PCI-e bus latency being only one of them... also, the crystal oscillator serving as a "stable" local reference for the PC chipset clock synth is in reality not very stable. I don't have a precise figure, but if I take the typical i210 NIC 25 MHz xtals for a benchmark, I've seen instability translating into ppb deviations of about +/- 10 to +/- 20 ppb, within the 1s period of ptp4l sampling - and a PC chipset is likely to have an even worse oscillator... BTW, speaking of NICs with PTP and SyncE support, the only one I know is a custom FPGA-based design by Oregano: https://www.oreganosystems.at/products/syn1588/hardware/syn1588r-pcie- nic Obviously that NIC is irrelevant to Linuxptp, as the FPGA-based Oregano stack implements the whole protocol... and that NIC is obviously not cheap :-) Thanks for spending your time, responding to my novice questions... Frank |
|
From: Miroslav L. <mli...@re...> - 2021-03-17 12:04:32
|
On Tue, Mar 16, 2021 at 11:38:17PM +0100, Frantisek Rysanek wrote: > Still I'm curious what news the i225 may bring in terms of e.g. GPIO > timestamping resolution. Does it get better than those 8 ns > I got used to with the i210 etc. I have not had a chance to play with one yet. If the frequency of the clock followed the increase in the symbol rate of 2.5GBASE-T, it would have a ~4.8ns resolution. > the packet timestamping is done by the NIC HW, so unless you have an > application where you need accurate timing all the way to the > software-based system clock, the determinism of PCIe bus latencies > should not matter much... Right, but in most cases I see people are interested in the accuracy and stability of the system clock. There don't seem to be many applications using hardware timestamps of the PHC. In some cases a sub-microsecond accuracy is required. The CPU<->NIC connection is the most problematic link in the chain. You can buy switches with good PTP support and compensate for asymmetric errors in hardware timestamping to get the PHC accurate to few nanoseconds, but due to the huge PCIe latency with an unknown asymmetry it's difficult to tell how accurate the system clock actually is. With the same NIC but different CPUs/boards you can get very different results. > As for getting the Linux system timebase precise, I guess the random > component to ISR latency has several contributing factors, the PCI-e > bus latency being only one of them... Normally there should be no interrupts involved. The time critical part is just a single read of a 32-bit PCI register. With the I210 that takes about 1700 ns and the asymmetry in my measurements was up to about 100 ns. The shortest delay I saw with faster NICs was about 450 ns, asymmetry unknown. > also, the crystal oscillator > serving as a "stable" local reference for the PC chipset clock > synth is in reality not very stable. I don't have a precise figure, > but if I take the typical i210 NIC 25 MHz xtals for a benchmark, > I've seen instability translating into ppb deviations of about +/- 10 > to +/- 20 ppb, within the 1s period of ptp4l sampling - and a PC > chipset is likely to have an even worse oscillator... I'd say that's normal for an uncompensated XO in a computer case where the temperature can change rapidly. The PTP and phc2sys update rate need to be higher to get better stability. Some PTP profiles use 128 messages/second. -- Miroslav Lichvar |
|
From: Frantisek R. <Fra...@po...> - 2021-03-17 15:49:21
|
On 17 Mar 2021 at 13:04, Miroslav Lichvar wrote: > > Right, but in most cases I see people are interested in the accuracy > and stability of the system clock. There don't seem to be many > applications using hardware timestamps of the PHC. > > In some cases a sub-microsecond accuracy is required. The CPU<->NIC > connection is the most problematic link in the chain. You can buy > switches with good PTP support and compensate for asymmetric errors in > hardware timestamping to get the PHC accurate to few nanoseconds, but > due to the huge PCIe latency with an unknown asymmetry it's difficult > to tell how accurate the system clock actually is. With the same NIC > but different CPUs/boards you can get very different results. > > > As for getting the Linux system timebase precise, I guess the random > > component to ISR latency has several contributing factors, the PCI-e > > bus latency being only one of them... > > Normally there should be no interrupts involved. The time critical > part is just a single read of a 32-bit PCI register. With the I210 > that takes about 1700 ns and the asymmetry in my measurements was up > to about 100 ns. The shortest delay I saw with faster NICs was about > 450 ns, asymmetry unknown. > Thanks for all the data and clarifications :-) That's interesting... I'm not sure the PTM (that you have mentioned previously) will help with asymmetry. You characterize the mechanism as "NTP-like". I'm not a member of the the PCI-SIG, and the only clear text about the PTM that can be found in Google is some change notice by the PCI-SIG from back in 2013, referring to PTM v1.0a. https://fdocuments.in/document/precision-time-measurement-ptm.html That paper clearly says (I'm quoting): "Therefore ((t4 - t1) - (t3 - t2)) effectively gives the round trip message transit time between the two components, and that quantity divided by 2 approximates the Link delay - the time difference between t1 and t2. It is assumed that the Link transit times from PTM Requester to PTM Responder and back again are symmetric, which is typically a good assumption." The current revision listed at the PCI-SIG website is v4.0. Which makes me wonder if this assumption is still considered valid :-) I understand that apart from the NIC (peripheral device / endpoint function in general), PTM must also be supported by the root complex and any PCI-e switches along the path... so a PTM-capable NIC alone will not even suffice for this simple NTP-style mechanism to function, if the host chipset is unaware of PTM. To account for asymmetry, the mechanism would have to resemble PTP = there would have to be a correction field in the PCI-e messages and the PCI-e switches would have to work like PTP TC switches :-) Frank |
|
From: Miroslav L. <mli...@re...> - 2021-03-17 17:01:28
|
On Wed, Mar 17, 2021 at 04:49:01PM +0100, Frantisek Rysanek wrote: > I understand that apart from the NIC (peripheral device / endpoint > function in general), PTM must also be supported by the root complex > and any PCI-e switches along the path... so a PTM-capable NIC alone > will not even suffice for this simple NTP-style mechanism to > function, if the host chipset is unaware of PTM. Right, all those components are expected to support PTM. It's implemented in the hardware. My understanding is that (at least some) existing CPUs already support it. I have no idea about switches. I guess even if they didn't support it, it could still perform better than what we currently have. As a workaround, at least for testing, the NIC can be inserted in the 16x slot which is connected directly to the CPU. In servers with fast NICs and CPUs with large number of PCIe lanes that could be the usual case. > To account for asymmetry, the mechanism would have to resemble PTP = > there would have to be a correction field in the PCI-e messages and > the PCI-e switches would have to work like PTP TC switches :-) The switches are supposed to work like NTP server and client at the same time (boundary clock in the PTP terminology), so all PCIe links have hardware timestamping on both ends. BTW, at least in theory, a network using boundary clocks should perform better than a similar network using transparent clocks, assuming the servos are well configured and the sync interval is short enough to minimize the time errors in the chain. Divide and conquer :). I think transparent clocks are meant to be the simpler and cheaper variant. -- Miroslav Lichvar |
|
From: Frantisek R. <Fra...@po...> - 2021-03-18 10:57:42
|
On 17 Mar 2021 at 18:01, Miroslav Lichvar wrote: [on PCI-e PTM] > > The switches are supposed to work like NTP server and client at the > same time (boundary clock in the PTP terminology), so all PCIe links > have hardware timestamping on both ends. > > BTW, at least in theory, a network using boundary clocks should > perform better than a similar network using transparent clocks, > assuming the servos are well configured and the sync interval is short > enough to minimize the time errors in the chain. Divide and conquer > :). I think transparent clocks are meant to be the simpler and cheaper > variant. > I recall some history around the debate on BC vs TC. >From my subjective perspective anyway. PTP v1 used to be all about BC's. Then in 2008 came PTP v2, and suddenly TC was all the rage, the only right way for PTP-capable switches to function from now on. Except that, as years were passing by, PTP-aware switches did not exactly spawn in flocks everywhere. Rather, they remained few and far between. And, especially the telco industry would've loved to deploy PTP, but there was no way for their active elements in the networks to support PTP (to turn them all into TC's). So after some years since 2008, suddenly the Telecom Profiles came out, and were all about BC's and partial on-path support (or rather: no on-path support). The TC idea has a stronghold in the electric power industry, at least on paper - practical compatibility of implementations remains a fun topic, to use a euphemism, especially where multi-vendor heterogeneous setups are attempted... this is where I have a limited by ongoing practical experience :-) On the topic of BC's and their apparent renaissance, I've had a brief debate about this with someone from Meinberg, and the message was: BC's or TC's, either of them works, if done properly. Reportedly there have been tests of up to 16 BC's in a cascade, and if the local oscillators were stable, and the feedback loops were tuned appropriately, the cascade just worked, there were no ill effects (excessive residual wander at the tail end, or whatever). So it's all down to oscillator quality and feedback loop parameters = control theory and the respective math. Laplace, Nyquist, whatever... As for TC's, from my practical observations (and maybe a bit of gossip), switch chipset vendors have a hard time implementing the on-the-fly corrections in the hardware. Something that would work "out of the box" for the switch box makers. And because the silicon vendors struggle, some switch box vendors implement PTP on their own, mangling MII on the fly using custom FPGA designs etc. You probably know the ugly details an order of magnitude better than I do, i.e. what it takes to make a TC switch, for P2P mode vs. E2E mode... The "TC support in a switch" is likely never going to be pure hardware, there's always an element of switch firmware that e.g. does the talking with P2P peers... While E2E might look easier to support in a switch (than P2P), I've heard a credible opinion that in E2E mode, a switch port needs to keep track of delay transactions passing through (stateful style - ORLY?) and making E2E work for a single slave is not the same as making it work for many slaves :-) etc. I've wandered even further off topic here... In PTM, having a servoed local oscillator in every PCI-e switch (BC style) seems complicated, to the point of being impractical / difficult to implement, if the oscillator should be half decent / stable... In my wildest dreams, it would be beautiful to have a 1PPS output out of the CPU TSC. Plug that into some SDP input on an i210, and you'd have an idea, where the CPU's PPS edge is located, relative to the i210 PHC. Which can in turn be disciplined by PTP, or by another external 1PPS reference. Just a single output wire, coming from the TSC - it wouldn't even need to be adjustable. Having a timestamp for that CPU 1PPS event from the i210, you'd be able to calculate an offset in TSC granularity, and subtract that offset from every timestamp obtained via rdtsc (at the cost of a single integer sub() instruction). A single signal, for 1PPS, coming out of the CPU socket. Is that too much to ask? Compared to the rather complex and imperfect PTM implementation? Actually this is not my wildest dream. I can extrapolate this further. Such as: After the all, the TSC stands for a "Time Stamp Counter" - isn't that ironic? As it is, the TSC is just a register counting steadily forward. What reference does it actually have to the wall time in the outside world, apart from the CPU core clock? (Yes I know - now with TurboBoost, even the notion of a CPU clock is hazy.) I mean wouldn't it be nice if the CPU would have a discrete event *input* into the TSC? Typically useable as 1PPS *input*. Having a timestamping register (MSR?) on the inside, a neighbor to the TSC, which would always contain a timestamp of the last exterior 1PPS edge, in the TSC's "time domain". And maybe have another register yet, clocked the same way as the TSC, but which would always reset at every 1PPS edge. Should I patent this as the TSC-ng ? :-) Next up: make it easier to reference the relevant clock signals in a PC system to a common local oscillator. If the system builder (or even integrator) so desires, so that he can drop a VC-OCXO module on the board, and have synchronous-referenced clocks for the CPU TSC, the NIC ports, and whatever have you. This, combined with a 1PPS output or input in the CPU TSC, would allow for some nasty timing performance :-) And would make passing EMC tests more complicated, without "spread spectrum" tricks. I've tried PLL-railroading the 25MHz i210 NIC clock, I've heard about people who swap/discipline their PC motherboard's central clock XTAL. To me a problem with the motherboard's existing central oscillator is, that its frequency is not standardized, and can be "the most abundant ugly binary multiple available on the XTAL market at the moment", difficult to co-divide with 10 MHz, or 25 MHz, or whatever external reference would seem practical in a lab environment. There's obviously a theoretical chance to just plug in a "nicer" frequency reference (instead of the original crystal) and reprogram the existing clock synth - but unless you can mod the BIOS (or flash coreboot, or some such) the machine will have to boot far enough with a skewed set of system clock frequencies, for you to be able to finally reprogram the synth (and with a bit of luck the PC would not hang while you're tampering with the synth registers). And obviously this is not generic / reproducible across the PC HW market. The way it is, it would probably be easiest to equip your custom VC-OCXO timebase with a multi-channel synth output that would produce the right frequencies to "nail" all the various crystals in the system. I myself have not gone that far yet :-) in my garagey DIY experiments. Oh we still have SyncE in the subject... Yes if you provide your own external stable clock to the 25MHz clock input of your NIC, you can as well configure the PPB frequency offset to 0 in your NIC's internal digital synth, and re-wire your ptp4l servo loop to discipline your local VC-OCXO out of band / in a proprietary way. Now to combine this with SyncE, you'd need to solve the clock recovery at the "SyncE slave" end. Low-level mixed-signal electronics. How to properly hook up a PLL to the raw Ethernet signals. Even a vanilla analog PLL can weed out a lot of ugly random mess out of a noisy signal... In descriptions in the public interwebs, I haven't found information about the details of SyncE-enabled layer 1, e.g. if the inter-frame filling looks different, or what differences there are for SERDES, vs. for N-Base metallic (especially gigabit and above). Apparently it's plausible to combine SyncE with vanilla SFP transceivers, i.e. to recover clock from the SERDES RX coming out the back of an SFP socket towards the MAC block... which is making me wonder if SyncE is possible with multi-rate SGMII-based RJ45 SFP transceivers :-) Or, are there multi-rate RJ45 SFP's with a SERDES back-end? And as you have already explained, there's also a bit of messaging taking place in SyncE, in addition to the raw PLL stuff... Frank |
|
From: Geva, E. <ere...@si...> - 2021-03-18 13:04:27
|
On 18/03/2021 11:57, Frantisek Rysanek wrote: > On 17 Mar 2021 at 18:01, Miroslav Lichvar wrote: > > [on PCI-e PTM] >> >> The switches are supposed to work like NTP server and client at the >> same time (boundary clock in the PTP terminology), so all PCIe links >> have hardware timestamping on both ends. >> >> BTW, at least in theory, a network using boundary clocks should >> perform better than a similar network using transparent clocks, >> assuming the servos are well configured and the sync interval is short >> enough to minimize the time errors in the chain. Divide and conquer >> :). I think transparent clocks are meant to be the simpler and cheaper >> variant. >> > > I recall some history around the debate on BC vs TC. > From my subjective perspective anyway. > > PTP v1 used to be all about BC's. > Then in 2008 came PTP v2, and suddenly TC was all the rage, the only > right way for PTP-capable switches to function from now on. > Except that, as years were passing by, PTP-aware switches did not > exactly spawn in flocks everywhere. Rather, they remained few and far > between. And, especially the telco industry would've loved to deploy > PTP, but there was no way for their active elements in the networks > to support PTP (to turn them all into TC's). So after some years > since 2008, suddenly the Telecom Profiles came out, and were all > about BC's and partial on-path support (or rather: no on-path > support). The TC idea has a stronghold in the electric power > industry, at least on paper - practical compatibility of > implementations remains a fun topic, to use a euphemism, especially > where multi-vendor heterogeneous setups are attempted... this is > where I have a limited by ongoing practical experience :-) > > On the topic of BC's and their apparent renaissance, I've had a brief > debate about this with someone from Meinberg, and the message was: > BC's or TC's, either of them works, if done properly. Reportedly > there have been tests of up to 16 BC's in a cascade, and if the local > oscillators were stable, and the feedback loops were tuned > appropriately, the cascade just worked, there were no ill effects > (excessive residual wander at the tail end, or whatever). So it's all > down to oscillator quality and feedback loop parameters = control > theory and the respective math. Laplace, Nyquist, whatever... > > As for TC's, from my practical observations (and maybe a bit of > gossip), switch chipset vendors have a hard time implementing the > on-the-fly corrections in the hardware. Something that would work > "out of the box" for the switch box makers. And because the silicon > vendors struggle, some switch box vendors implement PTP on their own, > mangling MII on the fly using custom FPGA designs etc. > You probably know the ugly details an order of magnitude better than > I do, i.e. what it takes to make a TC switch, for P2P mode vs. E2E > mode... The "TC support in a switch" is likely never going to be pure > hardware, there's always an element of switch firmware that e.g. does > the talking with P2P peers... While E2E might look easier to support > in a switch (than P2P), I've heard a credible opinion that in E2E > mode, a switch port needs to keep track of delay transactions passing > through (stateful style - ORLY?) and making E2E work for a single > slave is not the same as making it work for many slaves :-) etc. > > > I've wandered even further off topic here... > > In PTM, having a servoed local oscillator in every PCI-e switch (BC > style) seems complicated, to the point of being impractical / > difficult to implement, if the oscillator should be half decent / > stable... > > In my wildest dreams, it would be beautiful to have a 1PPS output out > of the CPU TSC. Plug that into some SDP input on an i210, and you'd > have an idea, where the CPU's PPS edge is located, relative to the > i210 PHC. Which can in turn be disciplined by PTP, or by another > external 1PPS reference. Just a single output wire, coming from the > TSC - it wouldn't even need to be adjustable. Having a timestamp for > that CPU 1PPS event from the i210, you'd be able to calculate an > offset in TSC granularity, and subtract that offset from every > timestamp obtained via rdtsc (at the cost of a single integer sub() > instruction). A single signal, for 1PPS, coming out of the CPU > socket. Is that too much to ask? Compared to the rather complex and > imperfect PTM implementation? > > Actually this is not my wildest dream. I can extrapolate this > further. Such as: > > After the all, the TSC stands for a "Time Stamp Counter" - isn't that > ironic? As it is, the TSC is just a register counting steadily > forward. What reference does it actually have to the wall time in the > outside world, apart from the CPU core clock? (Yes I know - now with > TurboBoost, even the notion of a CPU clock is hazy.) I mean wouldn't > it be nice if the CPU would have a discrete event *input* into the > TSC? Typically useable as 1PPS *input*. Having a timestamping > register (MSR?) on the inside, a neighbor to the TSC, which would > always contain a timestamp of the last exterior 1PPS edge, in the > TSC's "time domain". And maybe have another register yet, clocked the > same way as the TSC, but which would always reset at every 1PPS edge. > Should I patent this as the TSC-ng ? :-) > > Next up: make it easier to reference the relevant clock signals in a > PC system to a common local oscillator. If the system builder (or > even integrator) so desires, so that he can drop a VC-OCXO module on > the board, and have synchronous-referenced clocks for the CPU TSC, > the NIC ports, and whatever have you. This, combined with a 1PPS > output or input in the CPU TSC, would allow for some nasty timing > performance :-) And would make passing EMC tests more complicated, > without "spread spectrum" tricks. > > I've tried PLL-railroading the 25MHz i210 NIC clock, I've heard about > people who swap/discipline their PC motherboard's central clock XTAL. > To me a problem with the motherboard's existing central oscillator > is, that its frequency is not standardized, and can be "the most > abundant ugly binary multiple available on the XTAL market at the > moment", difficult to co-divide with 10 MHz, or 25 MHz, or whatever > external reference would seem practical in a lab environment. There's > obviously a theoretical chance to just plug in a "nicer" frequency > reference (instead of the original crystal) and reprogram the > existing clock synth - but unless you can mod the BIOS (or flash > coreboot, or some such) the machine will have to boot far enough with > a skewed set of system clock frequencies, for you to be able to > finally reprogram the synth (and with a bit of luck the PC would not > hang while you're tampering with the synth registers). And obviously > this is not generic / reproducible across the PC HW market. > The way it is, it would probably be easiest to equip your custom > VC-OCXO timebase with a multi-channel synth output that would > produce the right frequencies to "nail" all the various crystals in > the system. I myself have not gone that far yet :-) in my garagey > DIY experiments. > > > Oh we still have SyncE in the subject... > > Yes if you provide your own external stable clock to the 25MHz > clock input of your NIC, you can as well configure the PPB frequency > offset to 0 in your NIC's internal digital synth, and re-wire your > ptp4l servo loop to discipline your local VC-OCXO out of band / in a > proprietary way. Now to combine this with SyncE, you'd need to solve > the clock recovery at the "SyncE slave" end. Low-level mixed-signal > electronics. How to properly hook up a PLL to the raw Ethernet > signals. Even a vanilla analog PLL can weed out a lot of ugly random > mess out of a noisy signal... > > In descriptions in the public interwebs, I haven't found information > about the details of SyncE-enabled layer 1, e.g. if the inter-frame > filling looks different, or what differences there are for SERDES, > vs. for N-Base metallic > (especially gigabit and above). > > Apparently it's plausible to combine SyncE with vanilla SFP > transceivers, i.e. to recover clock from the SERDES RX coming out the > back of an SFP socket towards the MAC block... which is making me > wonder if SyncE is possible with multi-rate SGMII-based RJ45 SFP > transceivers :-) Or, are there multi-rate RJ45 SFP's with a SERDES > back-end? > > And as you have already explained, there's also a bit of messaging > taking place in SyncE, in addition to the raw PLL stuff... > > Frank > Your reply is extensive and I don't want to elaborate on most. I would like to add 2 points that I think you passed Switches that need to support TSN streams, need more then TC. TSN is only begining. How well would it be accepted by the market is yet be to seen. TSN enabled switches would be probably need to support IEEE 802.1AS. Erez > > > _______________________________________________ > Linuxptp-devel mailing list > Lin...@li... > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Flinuxptp-devel&data=04%7C01%7Cerez.geva.ext%40siemens.com%7C9c722dd44da3434a4f5008d8e9fcc1f2%7C38ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C637516619095431899%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=uQdS5qfxbjCdDaDQRFLxKAyYQotQSWz%2B8Ts8YSKnzXo%3D&reserved=0 > |
|
From: Miroslav L. <mli...@re...> - 2021-03-22 15:40:24
|
(Just few comments on some topics from this long post) On Thu, Mar 18, 2021 at 11:57:20AM +0100, Frantisek Rysanek wrote: > in a switch (than P2P), I've heard a credible opinion that in E2E > mode, a switch port needs to keep track of delay transactions passing > through (stateful style - ORLY?) and making E2E work for a single > slave is not the same as making it work for many slaves :-) etc. I think that depends on whether the switch is a one-step clock or two-step clock. In the one-step mode it shouldn't need to track individual requests and responses. It should be stateless, but it may be more difficult to implement in the silicon. I don't know if there are any difficulties specific to the E2E vs P2P implementation. To me it seems a big issue with PTP support in switches is that there is very little information available about their performance, e.g. resolution and typical accuracy. No public benchmarks. The vendors just want all the features marked as "supported". I heard about cases where it was so bad, that it was better disabled. > TSC? Typically useable as 1PPS *input*. Having a timestamping > register (MSR?) on the inside, a neighbor to the TSC, which would > always contain a timestamp of the last exterior 1PPS edge, in the > TSC's "time domain". And maybe have another register yet, clocked the > same way as the TSC, but which would always reset at every 1PPS edge. > Should I patent this as the TSC-ng ? :-) Yes, that would be great. Some small computers like the BeagleBone Black can timestamp PPS directly without interrupts. If nothing else, at least a fast GPIO suitable for polling would be a great improvement on x86. FWIW, some onboard NICs supported by the e1000e driver can "cross-timestamp" using the Always Running Timer (ART), which should avoid the asymmetry of PCIe. I have not seen any detailed description of how it actually works. -- Miroslav Lichvar |
|
From: Jacob K. <jac...@in...> - 2021-03-30 18:40:55
|
On 3/22/2021 8:40 AM, Miroslav Lichvar wrote: > FWIW, some onboard NICs supported by the e1000e driver can > "cross-timestamp" using the Always Running Timer (ART), which should > avoid the asymmetry of PCIe. I have not seen any detailed description > of how it actually works. > I'm not sure in the e1000e case but in general this is done via PCIe Precision Time Measurement extension which allows an end-point device that supports the function to setup a transaction that captures the device clock and ART value near-simultaneously. I think the e1000e support might be a variation of this which pre-dated the PTM standard and works when the device is a LAN-on-motherboard. |
|
From: Richard C. <ric...@gm...> - 2021-03-30 20:14:03
|
On Tue, Mar 30, 2021 at 11:40:42AM -0700, Jacob Keller wrote: > On 3/22/2021 8:40 AM, Miroslav Lichvar wrote: > > FWIW, some onboard NICs supported by the e1000e driver can > > "cross-timestamp" using the Always Running Timer (ART), which should > > avoid the asymmetry of PCIe. I have not seen any detailed description > > of how it actually works. > > > > I'm not sure in the e1000e case but in general this is done via PCIe > Precision Time Measurement extension which allows an end-point device > that supports the function to setup a transaction that captures the > device clock and ART value near-simultaneously. > > I think the e1000e support might be a variation of this which pre-dated > the PTM standard and works when the device is a LAN-on-motherboard. My understanding was that the e1000e did in fact implement the newfangled PCIe thingy. But I could be wrong. Thanks, Richard |
|
From: Machnikowski, M. <mac...@in...> - 2021-03-30 21:04:22
|
> -----Original Message----- > From: Richard Cochran <ric...@gm...> > Sent: Tuesday, March 30, 2021 10:14 PM > To: Keller, Jacob E <jac...@in...> > Cc: lin...@li... > Subject: Re: [Linuxptp-devel] SyncE support > > On Tue, Mar 30, 2021 at 11:40:42AM -0700, Jacob Keller wrote: > > On 3/22/2021 8:40 AM, Miroslav Lichvar wrote: > > > FWIW, some onboard NICs supported by the e1000e driver can > > > "cross-timestamp" using the Always Running Timer (ART), which should > > > avoid the asymmetry of PCIe. I have not seen any detailed > > > description of how it actually works. > > > > > > > I'm not sure in the e1000e case but in general this is done via PCIe > > Precision Time Measurement extension which allows an end-point device > > that supports the function to setup a transaction that captures the > > device clock and ART value near-simultaneously. > > > > I think the e1000e support might be a variation of this which > > pre-dated the PTM standard and works when the device is a LAN-on- > motherboard. > > My understanding was that the e1000e did in fact implement the newfangled > PCIe thingy. But I could be wrong. Some Intel I219 adapters embedded in the Sunrise Point PCH (and newer) have the ability to capture the ART along with the PTP clock. But that's not based on PTM, but on the proprietary Hammock Harbor IP triggered from the e1000. You can read about it here: https://www.freepatentsonline.com/y2020/0127861.html Regards Maciek > > Thanks, > Richard > > > _______________________________________________ > Linuxptp-devel mailing list > Lin...@li... > https://lists.sourceforge.net/lists/listinfo/linuxptp-devel Intel Technology Poland sp. z o.o. ul. Sowackiego 173 | 80-298 Gdask | Sd Rejonowy Gdask Pnoc | VII Wydzia Gospodarczy Krajowego Rejestru Sdowego - KRS 101882 | NIP 957-07-52-316 | Kapita zakadowy 200.000 PLN. Ta wiadomo wraz z zacznikami jest przeznaczona dla okrelonego adresata i moe zawiera informacje poufne. W razie przypadkowego otrzymania tej wiadomoci, prosimy o powiadomienie nadawcy oraz trwae jej usunicie; jakiekolwiek przegldanie lub rozpowszechnianie jest zabronione. This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited. |
