The good thing about NTP is that it protects (to some degree) against single point of failure and false-ticking: whenever a server becomes unreachable, or unreliable (outlier), or honestly announcing absence of good time, - it's excluded from peer selection procedure, even if it's a local reference clock.
What about PTP? The only kind of protection I found concerns server availability. As long as the elected BMC stays online, it's considered an unquestionable reference for all other nodes, - no matter how bad time it keeps. Even when BMC's time source is specifically NTP (yes, I know that ptp_timesource is purely informational) and NTPd integration is enabled, this node continues as the master even when NTP is not running at all.
Is there any standard means to tell PTPd that it should enter the faulty state once its own clock is not reliable? The only method that comes to mind is a standalone supervising process that monitors NTPd (or other time source) and kills PTPd to restart it with another config (with lower or higher priority or announced clock accuracy). Or am I missing something?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
That is the inherent "fault" of the PTP protocol. Remember that it originated from measurement and control systems. UTC transfer is of less importance - frequency transfer and tight sync between all clocks in the domain are the top priorities. John Eidson has a great example in his book on PTP (http://www.amazon.co.uk/Measurement-Control-Communication-Advances-Industrial/dp/1846282500/) where a number of separate car assembly robots have to move in perfect sync because they are holding a car body together. They won't mind if they're off from the wall clock by a millisecond, but they do mind if they rip the thing apart because one moved too soon. I mean the robots don't mind - people do :)
On the slave side, PTP will obey whatever the master tells it. The term "best master" is purely based on protocol flags. No qualification is done by the slave on the basis of time behaviour, although there is some provision from the GM side. The offsetScaledLogVariance (local oscillator's Allan Variance) and clockAccuracy should be dynamic and reflect the GM clock's performance. So far I've seen almost no implementations that actually do that, they usually output a flat value. This is a good thing and a bad thing. If a GM starts going wild and those values will start changing, it may cause flapping in the timing domain. However, NTP populates some of this information in the timex structure if it's running on the local system. In other words, there are ways to establish the local clock's performance and tell the slaves about it - but stability of the timing domain is also a very important factor.
It's up to the slave implementation if it wishes do do some extra observation. PTPd can protect a slave from clock jumps ("panic mode" and NTP failover during panic mode), but not much more. We could do something non-standard (and I did consider this) like observing timing from multiple GMs and choosing the most stable one. But this is a vicious circle; without a tie-breaker reference, how does the slave know if it itself is off and that's why it shows variation against this one GM, but doesn't against another one. You need a quorum - minimum 3 GMs. You end up re-inventing NTP. Notably, TimeKeeper does this, it's able to observe multiple sources, but I'm not sure if it's able to qualify them. Unfortunately I haven't tested TimeKeeper.
ClockClass evaluation in the BMC order is between priority1 and priority2, so if you define priority1 to be the same value, when you're running a hardware GM and it's lost its reference and goes via holdover and freerun, it will degrade the clockClass accordingly (and other flags), causing it to drop its overal priority below other "healthy" GMs. This is what I do in production.
Even if PTPd master's NTP is off, it will still keep its slaves in sync with itself which, I agree, may be worse than other masters.
What you're suggesting is a worthy feature addition, but it will currently only work with NTPd - and not for long, because the NTPd integration I wrote for my production use, uses NTP mode 7 packets and with the latest stink around DDoS attacks resulting from people using NTPD in default configuration, it may not be possible to use mode 7 for much longer, we'll probably have to rewrite this for mode 6.
Anyhow, in general terms, what's ultimately needed is a way of talking to PTPD master's time source, whether NTP or PTP, and an ability to provide a clockClass degradation alternative value. In the short term, this could be talking to NTPd and / or monitoring the timex flags for degradation.
Most of the work is going into slave operation of ptpd, partially due to project members' interest and use cases. In my production environments I have hardware GMs with GPS timing and Rubidium oscillators so I don't have to deal with issues like this. A lot of progress has been made in GM operation of ptpd though.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Wojciech, your profound answers deserve to be included in the FAQ permanently! :-)
I didn't really suggest to include some new mechanism into PTPd, just wanted to make sure I didn't miss something already implemented. My planned configuration is 1+1 reserved masters that receive their reference time from a single custom source (thus implied quorum), with a custom driver for NTPd. Whenever a source line fails, the master using this line should step down. I think that a simple standalone supervisor will be sufficient, without modifying PTPd itself.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I am indeed treating this forum as a scrap space and some of this text will end up in the FAQ / book.
Ptpd aims to "minimise the damage" when reloading configuration, so it will restart the protocol only if absolutely necessary, so changing most clock quality description / priority settings is seamless.
I would advise that your supervisor add-on does what a typical hardware GM would do. The standard behaviour is that when the GM's clock source status changes, the clockClass property degrades (or raises) accordingly. The BMC algorithm is nothing but a series of comparisons of different values. Priority1 is compared first, clockClass effectively second, and Priority2 is the penultimate condition, just before the tie-breaking clock ID comparison. Therefore, if Priority1 is set to equal values on all of your multiple identical (parameter wise) GMs, clockClass effectively decides on the priority, and if all GMs are healthy, Priority2 decides on the order of selection. Priority1 can still be used to pre-empt the best master if needed (for this purpose, keep Priority1 at the same, non-zero value, so you can make something best master instantly by setting priority1 to zero).
You would need a copy of the standard (table 5) to know what values are typically used, so here's a breakdown:
clockClass ranges (in ptpd, the "preset" setting regulates the allowed ranges):
0-127: clock can only be: master when best, passive otherwise (master only)
128-254: clock can be master when best, otherwise slave (master or slave)
255: clock is slave-only
clockClass values:
For clocks with a primary reference such as GPS:
6 - master only, PTP timescale
7 - master only, clockClass 6 gone into holdover, PTP timescale
52 - master only, degradation alternative A for clock class 7: gone past holdover tolerance
187 - slave when not best, degradation alternative B for clock class 7: gone past holdover tolerance
For application-specific clock sources (like PTPd with NTPd):
13 - master only, ARB timescale (default in "masteronly" preset)
14 - master only, 13 gone into holdover, ARB timescale
58 - master only, degradation alt. for clock class 14: gone past holdover
193 - slave when not best, degradation alt. for clock class 14: gone past holdover
248 - used when no other definitions apply (default in "masterslave" preset)
So, you may choose to use the "masterslave" behaviour and your GM can become slave to the other PTP GMs when it's NTP reference is gone.
Last edit: Wojciech Owczarek 2014-02-18
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The good thing about NTP is that it protects (to some degree) against single point of failure and false-ticking: whenever a server becomes unreachable, or unreliable (outlier), or honestly announcing absence of good time, - it's excluded from peer selection procedure, even if it's a local reference clock.
What about PTP? The only kind of protection I found concerns server availability. As long as the elected BMC stays online, it's considered an unquestionable reference for all other nodes, - no matter how bad time it keeps. Even when BMC's time source is specifically NTP (yes, I know that ptp_timesource is purely informational) and NTPd integration is enabled, this node continues as the master even when NTP is not running at all.
Is there any standard means to tell PTPd that it should enter the faulty state once its own clock is not reliable? The only method that comes to mind is a standalone supervising process that monitors NTPd (or other time source) and kills PTPd to restart it with another config (with lower or higher priority or announced clock accuracy). Or am I missing something?
That is the inherent "fault" of the PTP protocol. Remember that it originated from measurement and control systems. UTC transfer is of less importance - frequency transfer and tight sync between all clocks in the domain are the top priorities. John Eidson has a great example in his book on PTP (http://www.amazon.co.uk/Measurement-Control-Communication-Advances-Industrial/dp/1846282500/) where a number of separate car assembly robots have to move in perfect sync because they are holding a car body together. They won't mind if they're off from the wall clock by a millisecond, but they do mind if they rip the thing apart because one moved too soon. I mean the robots don't mind - people do :)
On the slave side, PTP will obey whatever the master tells it. The term "best master" is purely based on protocol flags. No qualification is done by the slave on the basis of time behaviour, although there is some provision from the GM side. The offsetScaledLogVariance (local oscillator's Allan Variance) and clockAccuracy should be dynamic and reflect the GM clock's performance. So far I've seen almost no implementations that actually do that, they usually output a flat value. This is a good thing and a bad thing. If a GM starts going wild and those values will start changing, it may cause flapping in the timing domain. However, NTP populates some of this information in the timex structure if it's running on the local system. In other words, there are ways to establish the local clock's performance and tell the slaves about it - but stability of the timing domain is also a very important factor.
It's up to the slave implementation if it wishes do do some extra observation. PTPd can protect a slave from clock jumps ("panic mode" and NTP failover during panic mode), but not much more. We could do something non-standard (and I did consider this) like observing timing from multiple GMs and choosing the most stable one. But this is a vicious circle; without a tie-breaker reference, how does the slave know if it itself is off and that's why it shows variation against this one GM, but doesn't against another one. You need a quorum - minimum 3 GMs. You end up re-inventing NTP. Notably, TimeKeeper does this, it's able to observe multiple sources, but I'm not sure if it's able to qualify them. Unfortunately I haven't tested TimeKeeper.
ClockClass evaluation in the BMC order is between priority1 and priority2, so if you define priority1 to be the same value, when you're running a hardware GM and it's lost its reference and goes via holdover and freerun, it will degrade the clockClass accordingly (and other flags), causing it to drop its overal priority below other "healthy" GMs. This is what I do in production.
Even if PTPd master's NTP is off, it will still keep its slaves in sync with itself which, I agree, may be worse than other masters.
What you're suggesting is a worthy feature addition, but it will currently only work with NTPd - and not for long, because the NTPd integration I wrote for my production use, uses NTP mode 7 packets and with the latest stink around DDoS attacks resulting from people using NTPD in default configuration, it may not be possible to use mode 7 for much longer, we'll probably have to rewrite this for mode 6.
Anyhow, in general terms, what's ultimately needed is a way of talking to PTPD master's time source, whether NTP or PTP, and an ability to provide a clockClass degradation alternative value. In the short term, this could be talking to NTPd and / or monitoring the timex flags for degradation.
Most of the work is going into slave operation of ptpd, partially due to project members' interest and use cases. In my production environments I have hardware GMs with GPS timing and Rubidium oscillators so I don't have to deal with issues like this. A lot of progress has been made in GM operation of ptpd though.
Wojciech, your profound answers deserve to be included in the FAQ permanently! :-)
I didn't really suggest to include some new mechanism into PTPd, just wanted to make sure I didn't miss something already implemented. My planned configuration is 1+1 reserved masters that receive their reference time from a single custom source (thus implied quorum), with a custom driver for NTPd. Whenever a source line fails, the master using this line should step down. I think that a simple standalone supervisor will be sufficient, without modifying PTPd itself.
I am indeed treating this forum as a scrap space and some of this text will end up in the FAQ / book.
Ptpd aims to "minimise the damage" when reloading configuration, so it will restart the protocol only if absolutely necessary, so changing most clock quality description / priority settings is seamless.
I would advise that your supervisor add-on does what a typical hardware GM would do. The standard behaviour is that when the GM's clock source status changes, the clockClass property degrades (or raises) accordingly. The BMC algorithm is nothing but a series of comparisons of different values. Priority1 is compared first, clockClass effectively second, and Priority2 is the penultimate condition, just before the tie-breaking clock ID comparison. Therefore, if Priority1 is set to equal values on all of your multiple identical (parameter wise) GMs, clockClass effectively decides on the priority, and if all GMs are healthy, Priority2 decides on the order of selection. Priority1 can still be used to pre-empt the best master if needed (for this purpose, keep Priority1 at the same, non-zero value, so you can make something best master instantly by setting priority1 to zero).
You would need a copy of the standard (table 5) to know what values are typically used, so here's a breakdown:
clockClass ranges (in ptpd, the "preset" setting regulates the allowed ranges):
0-127: clock can only be: master when best, passive otherwise (master only)
128-254: clock can be master when best, otherwise slave (master or slave)
255: clock is slave-only
clockClass values:
For clocks with a primary reference such as GPS:
6 - master only, PTP timescale
7 - master only, clockClass 6 gone into holdover, PTP timescale
52 - master only, degradation alternative A for clock class 7: gone past holdover tolerance
187 - slave when not best, degradation alternative B for clock class 7: gone past holdover tolerance
For application-specific clock sources (like PTPd with NTPd):
13 - master only, ARB timescale (default in "masteronly" preset)
14 - master only, 13 gone into holdover, ARB timescale
58 - master only, degradation alt. for clock class 14: gone past holdover
193 - slave when not best, degradation alt. for clock class 14: gone past holdover
248 - used when no other definitions apply (default in "masterslave" preset)
So, you may choose to use the "masterslave" behaviour and your GM can become slave to the other PTP GMs when it's NTP reference is gone.
Last edit: Wojciech Owczarek 2014-02-18