linuxptp-users

[Linuxptp-users] UTC offset change when loosing grand master

[Boris B. <bor...@gm...>]

Hi all,

I have a system with a grand master clock that is GPS synchronized (secure
sync),
a SERVER machine running Linux/ptp4l acting as boundary clock for set of
CLIENTS systems.

The SERVER and the CLIENTS behind it requires precise time synchronization,
hence
the usage of PTP.

The grand master is aware that the current UTC offset is 37s today.

My SERVER on the other hand, is built with a code base from 2016 and think
the UTC offset is 36s. But as long as the grand master is present, the
boundary clock
distribute time with the correct offset.

The problem is, if the grand master is lost (power failure, network
issue....), my boundary
clock promotes itself as grand master, and uses its own knowledge of the
UTC offset instead
of the last known offset from the secure sync, and this cause a 1s step on
the UTC time !

This change propagate to the CLIENTS and also to the system clock of the
SERVER via phc2sys.
And during 1 or 2 minutes, the system clock of the SERVER and the CLIENTS
are not fully synchronized,
causing the system to fail to provide its service.

Looking at the documentation, I can't find any way of updating the UTC
offset of the SERVER in a simple
manner.

Rebuilding the SERVER with a new code base each time a new leap second is
inserted is not an option (the system
once deployed can not be updated easily).

What are my options ?

Thank you in advance for your advice !

Boris Boucher

Re: [Linuxptp-users] UTC offset change when loosing grand master

[Richard C. <ric...@gm...>]

On Wed, Jun 21, 2017 at 07:57:21PM +0200, Boris Boucher wrote:
> The problem is, if the grand master is lost (power failure, network
> issue....), my boundary
> clock promotes itself as grand master, and uses its own knowledge of the
> UTC offset instead
> of the last known offset from the secure sync, and this cause a 1s step on
> the UTC time !

Your are correct.  After the GM goes away, the ptp4l program does not
remember the UTC offset learned as a slave.  It makes sense that we
should store the learned value in this case.

> Looking at the documentation, I can't find any way of updating the UTC
> offset of the SERVER in a simple
> manner.

You can set the UTC offset at run time using the
GRANDMASTER_SETTINGS_NP management message.
 
> What are my options ?

The best solution is to patch the program to store the UTC offset from
the GM, provided that the currentUtcOffsetValid is set.

One work around that you could use without patching is to monitor the
status (by scripting the pmc program, for example) using GET
GRANDMASTER_SETTINGS_NP.  Then, when currentUtcOffsetValid = 1, write
the value of currentUtcOffset back into ptp4l using SET
GRANDMASTER_SETTINGS_NP.

Thanks,
Richard

Re: [Linuxptp-users] UTC offset change when loosing grand master

[Scott S. <ssi...@si...>]

I don't know if it's the recommended solution, but I encountered a similar
problem a while back. My solution was to use pmc as follows:

# pmc -u -b 0 'set GRANDMASTER_SETTINGS_NP clockClass 248 clockAccuracy
0xfe offsetScaledLogVariance 0xffff currentUtcOffset 37 leap61 0 leap59 0
currentUtcOffsetValid 1 ptpTimescale 1 timeTraceable 0 frequencyTraceable 0
timeSource 0xa0'

Hopefully someone more knowledgeable will have something better to suggest,
but this does work.





Thanks,

Scott Silverman | IT | Simplex | 312-360-2444
230 S. LaSalle St., Suite 4-100, Chicago, IL 60604

On Wed, Jun 21, 2017 at 12:57 PM, Boris Boucher <bor...@gm...>
wrote:

> Hi all,
>
> I have a system with a grand master clock that is GPS synchronized (secure
> sync),
> a SERVER machine running Linux/ptp4l acting as boundary clock for set of
> CLIENTS systems.
>
> The SERVER and the CLIENTS behind it requires precise time
> synchronization, hence
> the usage of PTP.
>
> The grand master is aware that the current UTC offset is 37s today.
>
> My SERVER on the other hand, is built with a code base from 2016 and think
> the UTC offset is 36s. But as long as the grand master is present, the
> boundary clock
> distribute time with the correct offset.
>
> The problem is, if the grand master is lost (power failure, network
> issue....), my boundary
> clock promotes itself as grand master, and uses its own knowledge of the
> UTC offset instead
> of the last known offset from the secure sync, and this cause a 1s step on
> the UTC time !
>
> This change propagate to the CLIENTS and also to the system clock of the
> SERVER via phc2sys.
> And during 1 or 2 minutes, the system clock of the SERVER and the CLIENTS
> are not fully synchronized,
> causing the system to fail to provide its service.
>
> Looking at the documentation, I can't find any way of updating the UTC
> offset of the SERVER in a simple
> manner.
>
> Rebuilding the SERVER with a new code base each time a new leap second is
> inserted is not an option (the system
> once deployed can not be updated easily).
>
> What are my options ?
>
> Thank you in advance for your advice !
>
> Boris Boucher
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Linuxptp-users mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linuxptp-users
>
>

-- 
DISCLAIMER: NOTICE REGARDING PRIVACY AND CONFIDENTIALITY 

The information contained in and/or accompanying this communication is 
intended only for use by the addressee(s) named herein and may contain 
legally privileged and/or confidential information. If you are not the 
intended recipient of this e-mail, you are hereby notified that any 
dissemination, distribution or copying of this information, and any 
attachments thereto, is strictly prohibited. If you have received this 
e-mail in error, please immediately notify the sender and permanently 
delete the original and any copy of any e-mail and any printout thereof. 
Electronic transmissions cannot be guaranteed to be secure or error-free. 
The sender therefore does not accept liability for any errors or omissions 
in the contents of this message which arise as a result of e-mail 
transmission. Simplex Trading, LLC and its affiliates reserves the right to 
intercept, monitor, and retain electronic communications to and from its 
system as permitted by law. Simplex Trading, LLC is a registered Broker 
Dealer with CBOE and a Member of SIPC.

[Linuxptp-users] Fwd: UTC offset change when loosing grand master

[Boris B. <bor...@gm...>]

Thank you for your responses.

I have tested the GRANDMASTER_SETTINGS_NP solution, it may work but I dont
think the solution is very elegant.

I also swear that I may also have a problem during system startup, if the
serveur is started before the master clock, it may
start to broadcast a time with the wrong UTC offset until the GM is ready.

Richard, I understand that the daemon should write the received offset on a
persistant storage on use it on startup, regardless
of the actual time zone or hardcoded info ? Is it a difficult modification
to implement in the daemon ?

Regards,

Boris.

Re: [Linuxptp-users] Fwd: UTC offset change when loosing grand master

[Richard C. <ric...@gm...>]

On Thu, Jun 22, 2017 at 09:18:34AM +0200, Boris Boucher wrote:
> I also swear that I may also have a problem during system startup, if the
> serveur is started before the master clock, it may
> start to broadcast a time with the wrong UTC offset until the GM is ready.

Be sure to check the value of currentUtcOffsetValid.  If zero, then
the offset should be ignored.
 
> Richard, I understand that the daemon should write the received offset on a
> persistant storage on use it on startup, regardless
> of the actual time zone or hardcoded info ? Is it a difficult modification
> to implement in the daemon ?

No, it doesn't make sense to automatically store the offset.  Why?
Because when the program starts again, there is no guarantee that one
or more new leap seconds have occurred, and there is no way to know,
except via the network.

Having said that, if you can verify the offset before starting ptp4l,
then you can provide it via the utc_offset configuration file option
or the --utc_offset=x command line switch.

Thanks,
Richard

Re: [Linuxptp-users] UTC offset change when loosing grand master

[Richard C. <ric...@gm...>]

On Wed, Jun 21, 2017 at 07:57:21PM +0200, Boris Boucher wrote:
> This change propagate to the CLIENTS and also to the system clock of the
> SERVER via phc2sys.
> And during 1 or 2 minutes, the system clock of the SERVER and the CLIENTS
> are not fully synchronized,
> causing the system to fail to provide its service.

Are the CLIENTS also running ptp4l?

Thanks,
Richard

Re: [Linuxptp-users] UTC offset change when loosing grand master

[Boris B. <bor...@gm...>]

2017-06-28 11:33 GMT+02:00 Richard Cochran <ric...@gm...>:
>
> On Wed, Jun 21, 2017 at 07:57:21PM +0200, Boris Boucher wrote:
> > This change propagate to the CLIENTS and also to the system clock of the
> > SERVER via phc2sys.
> > And during 1 or 2 minutes, the system clock of the SERVER and the CLIENTS
> > are not fully synchronized,
> > causing the system to fail to provide its service.
>
> Are the CLIENTS also running ptp4l?

No, the CLIENTS are running ptpd.

>
> Thanks,
> Richard

Re: [Linuxptp-users] Fwd: UTC offset change when loosing grand master

[Boris B. <bor...@gm...>]

2017-06-28 11:31 GMT+02:00 Richard Cochran <ric...@gm...>:
> On Thu, Jun 22, 2017 at 09:18:34AM +0200, Boris Boucher wrote:
>> I also swear that I may also have a problem during system startup, if the
>> serveur is started before the master clock, it may
>> start to broadcast a time with the wrong UTC offset until the GM is ready.
>
> Be sure to check the value of currentUtcOffsetValid.  If zero, then
> the offset should be ignored.
>
>> Richard, I understand that the daemon should write the received offset on a
>> persistant storage on use it on startup, regardless
>> of the actual time zone or hardcoded info ? Is it a difficult modification
>> to implement in the daemon ?
>
> No, it doesn't make sense to automatically store the offset.  Why?
> Because when the program starts again, there is no guarantee that one
> or more new leap seconds have occurred, and there is no way to know,
> except via the network.

I agree that on startup, we have no guarantee that no leap second was
added or removed
since the last shutdown.
But, in a global system that is powered up, the GPS helped master
clock will take some some
time to start, eventually more time than my SERVER machine. In between
the SERVER will
assume grand master role and start broadcast the wrong time (because
its utc offset knowledge
is old).
So, on one hand, we face the risk of having an offset error each time
the system is started, on the
other hand, we have e very little change of starting  up the system
just after a leap second
update and have the wrong time until the GPS helped master clock is available.
>From my point of view, having a risk of temporary false time after
each startup is more problematic.


>
> Having said that, if you can verify the offset before starting ptp4l,
> then you can provide it via the utc_offset configuration file option
> or the --utc_offset=x command line switch.
>

Happy to know about this option. This may be the solution in my case
to start the daemon
with the correct offset each time. Much better than having to update
the offset afterward with
the pmc command while the daemon is already broadcasting a bad offset.
I while try this ASAP.

By the way, this configuration option seams not to be documented
(never seen it before). I just
looked in the source code to retrieve it. But I didn't see it as a
command line switch in the code.
I will check that at work tomorow.

Thank you very much,

Boris.

> Thanks,
> Richard

Re: [Linuxptp-users] Fwd: UTC offset change when loosing grand master

[Richard C. <ric...@gm...>]

On Wed, Jun 28, 2017 at 11:38:48PM +0200, Boris Boucher wrote:
> But, in a global system that is powered up, the GPS helped master
> clock will take some some time to start, eventually more time than
> my SERVER machine. In between the SERVER will assume grand master
> role and start broadcast the wrong time (because its utc offset
> knowledge is old).

As long as currentUtcOffsetValid=0, then the behavior of the SERVER is
correct according to the standard.

Still, that is little consolation to you, when your clients ignore the
currentUtcOffsetValid flag.

> So, on one hand, we face the risk of having an offset error each
> time the system is started, on the other hand, we have e very little
> change of starting up the system just after a leap second update and
> have the wrong time until the GPS helped master clock is available.
> From my point of view, having a risk of temporary false time after
> each startup is more problematic.

I disagree.  If I were to implement this "magic" saving of the offset,
then it would appear to work, even though it is in fact broken.  Then
users would rely on this, until the day comes when someone hits our
latent leap second bug.

I prefer to force users to come to grips with their configuration WRT
leap second handling early on, rather than waiting until the rare
event causes very unpleasant surprises.

> By the way, this configuration option seams not to be documented
> (never seen it before). I just looked in the source code to retrieve
> it. But I didn't see it as a command line switch in the code.

This is a new option, post version 1.8 but in the latest git head.

    From 33e62f992542ac5ce6bdbb8ae6c34dec7011b543 Mon Sep 17 00:00:00 2001
    From: Viliam Lejcik <vil...@ki...>
    Date: Tue, 17 Jan 2017 18:25:26 +0100
    Subject: ptp4l: Make UTC offset configurable.

    Currently UTC offset is defined as a constant - CURRENT_UTC_OFFSET, and if
    a leap second is added, that constant is no longer valid. Ptp4l was
    updated to read the UTC offset from configuration instead.

    Signed-off-by: Viliam Lejcik <vil...@ki...>

Thanks,
Richard

Re: [Linuxptp-users] UTC offset change when loosing grand master

[Richard C. <ric...@gm...>]

On Wed, Jun 28, 2017 at 11:08:04PM +0200, Boris Boucher wrote:
> No, the CLIENTS are running ptpd.

Ok, then ptpd has a bug.  It ignores the currentUtcOffsetValid member
of the TIME_PROPERTIES data set.  You will need to fix that in order
to solve your start up problem, or you could instead work around it by
setting the offset in the ptp4l configuration or in its command line.

Looking at ptp4l, sending an incorrect currentUtcOffset with
currentUtcOffsetValid=0 is technically not wrong, but I agree that we
should retain the correct offset when provided by a GM, even after
that GM disappears.

I will review the ptp4l client behavior when using SW time stamps.
Maybe there is a bug there as well.

Thinking out loud, the rational for the current behavior is the common
use case of an isolated LAN without any global time source.  In this
case, users don't care what the offset is.  They simply want the nodes
in the system to be synchronized to each other.

Thanks,
Richard

Re: [Linuxptp-users] UTC offset change when loosing grand master

[Richard C. <ric...@gm...>]

On Thu, Jun 29, 2017 at 10:17:35AM +0200, Richard Cochran wrote:
> Looking at ptp4l, sending an incorrect currentUtcOffset with
> currentUtcOffsetValid=0 is technically not wrong, but I agree that we
> should retain the correct offset when provided by a GM, even after
> that GM disappears.

Over on the linuxptp-devel list, I posted a series that should fix
this issue.  Please see if it fixes the problem for you.

In order to test it, you can force the TAI-UTC offset to an out of
date value (like 35) using the configuration file or the command line.

Thanks,
Richard