Re: [Linuxptp-users] offset explosion and irregular phc offsets

[Richard C. <ric...@gm...>]

Julien,

You have a very interesting and challenging application. You provided
lots of detail, and so I won't try to make very specific comments
right away. Please find a few general comments, below.

On Sun, Aug 11, 2013 at 10:54:56PM +0100, Julien Houles wrote:
> Configuration:
> Linux 3.5.3
> Intel 82574l with e1000e-2.3.2 drivers
> xenomai-2.6.2.1

Xenomai is great technology, but using it will introduce a *third*
clock into every host: the PHC, the Linux system clock, and the
Xenomai system clock. Probably you will want to directly synchronize
the Xenomai time with the PHC (= development effort).

It sounds like you are using Xenomai for real time Ethernet packet
scheduling. An alternative solution would be to use the Intel i210
PCIe card. It is not expensive, and it has a special feature, namely
real time transmission scheduling (but you would have to write a
driver for this yourself). You can synchronize the i210 using PTP and
then transmit during a globally synchronized time slot.

> The first solution we imagine is making a chain. We bought 3 boards and
> built the following setup:
> - port 5 (p37p1, ptp5) board 1 linked to port 5 (p37p1, ptp5) board 2
> port 5 board 1 is master and port 5 board 2 is slave
> - port 4 (p36p1, ptp4) board 2 linked to port 4 (p36p1, ptp4) board 3
> port 4 board 2 is master and port 4 board 3 is slave

This chain will not work very well. The multi-port nodes (like board 2)
do not have their port clocks synchronized. I guess that using phc2sys
to synchronize them with introduce several hundred nanoseconds time
error per node. For good chaining performance, you really want to have
the PHC clocks synchronized in hardware.

> General questions:
> What do you think of such a chain ? Do you think the offset between board 1
> and board 50 will be acceptable (~ 1µs)

I don't think this is possible without special measures.

To quote 802.1AS-2011 (AVB):

   B.3 End-to-end time-synchronization performance

   The requirements of this standard and of standards referenced for
   each medium ensure that any two time-aware systems separated by
   six or fewer time-aware systems (i.e., seven or fewer hops) will be
   synchronized to within 1 μs peak-to-peak of each other during
   steady-state operation (i.e., each time-aware system receives
   time-synchronization information every sync interval).

In AVB, in order to avoid gain peaking, the clocks are left free
running, but the phase and frequency offsets are calculated,
accumulated, and propagated through the network. You might consider
using this method in your system (but it means calculating global time
in your application on each node).

You might be able to break your network into ten chains of five,
connected by a 16 port transparent switch, for example.

There have been studies (especially WRT AVB) on using long chains. I
haven't really looked at this myself, but I have seen papers on the
AVB working group's website.

Have you looked at the white rabbit project?

> Could we use a mechanism built on a ethernet hub or switch (regular switch,
> not a IEEE1588 one because too expensive)? We’ll only use these links for
> a synchronization purpose. Can we expect a constant delay with a store and
> forward switch? Or should we use a hub?

You can try a normal switch, but they will introduce variable packet
delays. I have read about using a managed switch with prioritized PTP
queues instead of a PTP switch.  However, I expect that you will need
special PTP hardware in order to meet your requirements.

HTH,
Richard