Can you please explain the solution to the leap second bug posted on this
project (http://sourceforge.net/p/ptpd/bugs/31/)?
I am particularly interested to know how the fix addresses the concern in
the 5th paragraph of the bug? "In any case, computing offsets during a leap
second may yield large differences which could mess up the servo loop. So
may be the daemon's servo should simply skip computing offsets during an
ongoing leap second."
Best,
Nayeem
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The leap second flag is announced by the GM (following, say, a GPS time source leap second announcement) from up to 12 hours before the event. The flag denotes whether we're adding a second at midnight (inserting a second) or we're subtracting a second at midnight (skipping a second). The 1588 standard also defines when the change should happen - +/- 2 announce intervals around midnight as far as I remember. What the change means is that the event itself consists of two parts: a) the leap second flag in the announce messages is cleared and b) the announced UTC offset is changed accordingly. The native PTP time scale is the international atomic time (TAI) which is monotonic. UTC isn't, and is established using the combination of TAI base and the value of UTC offset.
If we correctly set the kernel leap second flag based on the leap second information from PTP, the Linux kernel (NTP API) clock subsystem implementation will skip from 23:59:58.999999999 straight to 00:00:00, skipping 23:59:59 when the flag is to skip a leap second. When the flag is set to add a second, the clock will run 23:59:59 twice (that's the raw time - the time formatting function will display this as 23:59:60 - this is a valid UTC timestamp).
During the same time, the PTP GM will announce the change. The thing is that we cannot predict exactly when this will be announced. So chances are that either the kernel does its work before we get the PTP announcement with new offset and new time, or we get the announcement before kernel does the insertion / deletion, so when this happens, you will suddenly get a one-second difference between the slave and the GM, a massive clock slew will be applied and your clock will be out of tune after the event, and it will take it a while to get back to normal. What makes things worse is that you're also sending delay request messages at the same time, so the chances are that even if you somehow noticed the UTC offset change and corrected yourself just in time, you will receive a reply for your "old" delay response and suddenly you'll get a 1-second one-way delay.
So what was implemented in ptpd is that it suspends clock offset calculation and clock offset correction for the period suggested by the IEEE spec which is from -2 announce intervals before midnight to +2 announce intervals after midnight. Thanks to this, when you resume the calculations and updates a) the OS clock has already completed the leap second operation and b) PTP is already announcing the new UTC offset, so things normally go smooth. Yes, you will possibly see a few microsecond offset spike after the event, but it's an inevitable consequence of clock not being controlled for those few seconds.
The kernel behaviour of skipping or adding a second may not be desired for certain applications, especially if you're using clock_gettime or gettimeofday with raw timespec or timeval - for time-critical applications, seeing the same timestamp twice may be quite, well, confusing. In addition to that, there was a bug last year on older Linux kernels whereby you would get a kernel panic during the event at heavy CPU loads. But this only shows that people have not tested this event well enough. And it makes me wonder why, because the IRS advertise the event six months ahead. I think the answer is that people take timing services for granted and often do not have the necessary knowledge.
And so, the other solution to this problem that I know of, and is being used by some software, is what I would call "consuming the leap second". This approach will correct the PTP-derived offset from master by a small fraction every time it's calculated, so that the second is made longer or shorter in the client OS, so that the clock will converge to the correct time after the event. Basically on top of normal clock drift correction, you will be applying an additional offset of (+/-)1s / seconds_to_midnight. This is tracking a moving target to the implementation may not be trivial - you need to ensure that different clocks across your timing domain that started at different times, will be synchronised with each other while this artificial offset is being injected. You are fine as long as all your clocks and all clocks in any external time-critical systems you're interfacing with, use the same methodology - which is not always the case. This is why personally I would prefer the full leap approach, however you really need to safeguard your applications against it so that it doesn't lead to unpredictable behaviour.
Thanks
Woj
Last edit: Wojciech Owczarek 2013-02-11
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
This solution seems reasonable to me, especially to avoid the problems that you mentioned with regard to clock_gettime/gettimeofday. What do you think? Do you have any opinion on this solution?
A related question, the use of CLOCK_ABSTIME is discouraged to avoid any problem related to leap second. Is there any common design patter/solution to still use CLOCK_ABSTIME based timers while not facing any problem of the current leap seconds handling in the kernel?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
Can you please explain the solution to the leap second bug posted on this
project (http://sourceforge.net/p/ptpd/bugs/31/)?
I am particularly interested to know how the fix addresses the concern in
the 5th paragraph of the bug? "In any case, computing offsets during a leap
second may yield large differences which could mess up the servo loop. So
may be the daemon's servo should simply skip computing offsets during an
ongoing leap second."
Best,
Nayeem
Nayeem,
The leap second flag is announced by the GM (following, say, a GPS time source leap second announcement) from up to 12 hours before the event. The flag denotes whether we're adding a second at midnight (inserting a second) or we're subtracting a second at midnight (skipping a second). The 1588 standard also defines when the change should happen - +/- 2 announce intervals around midnight as far as I remember. What the change means is that the event itself consists of two parts: a) the leap second flag in the announce messages is cleared and b) the announced UTC offset is changed accordingly. The native PTP time scale is the international atomic time (TAI) which is monotonic. UTC isn't, and is established using the combination of TAI base and the value of UTC offset.
If we correctly set the kernel leap second flag based on the leap second information from PTP, the Linux kernel (NTP API) clock subsystem implementation will skip from 23:59:58.999999999 straight to 00:00:00, skipping 23:59:59 when the flag is to skip a leap second. When the flag is set to add a second, the clock will run 23:59:59 twice (that's the raw time - the time formatting function will display this as 23:59:60 - this is a valid UTC timestamp).
During the same time, the PTP GM will announce the change. The thing is that we cannot predict exactly when this will be announced. So chances are that either the kernel does its work before we get the PTP announcement with new offset and new time, or we get the announcement before kernel does the insertion / deletion, so when this happens, you will suddenly get a one-second difference between the slave and the GM, a massive clock slew will be applied and your clock will be out of tune after the event, and it will take it a while to get back to normal. What makes things worse is that you're also sending delay request messages at the same time, so the chances are that even if you somehow noticed the UTC offset change and corrected yourself just in time, you will receive a reply for your "old" delay response and suddenly you'll get a 1-second one-way delay.
So what was implemented in ptpd is that it suspends clock offset calculation and clock offset correction for the period suggested by the IEEE spec which is from -2 announce intervals before midnight to +2 announce intervals after midnight. Thanks to this, when you resume the calculations and updates a) the OS clock has already completed the leap second operation and b) PTP is already announcing the new UTC offset, so things normally go smooth. Yes, you will possibly see a few microsecond offset spike after the event, but it's an inevitable consequence of clock not being controlled for those few seconds.
The kernel behaviour of skipping or adding a second may not be desired for certain applications, especially if you're using clock_gettime or gettimeofday with raw timespec or timeval - for time-critical applications, seeing the same timestamp twice may be quite, well, confusing. In addition to that, there was a bug last year on older Linux kernels whereby you would get a kernel panic during the event at heavy CPU loads. But this only shows that people have not tested this event well enough. And it makes me wonder why, because the IRS advertise the event six months ahead. I think the answer is that people take timing services for granted and often do not have the necessary knowledge.
And so, the other solution to this problem that I know of, and is being used by some software, is what I would call "consuming the leap second". This approach will correct the PTP-derived offset from master by a small fraction every time it's calculated, so that the second is made longer or shorter in the client OS, so that the clock will converge to the correct time after the event. Basically on top of normal clock drift correction, you will be applying an additional offset of (+/-)1s / seconds_to_midnight. This is tracking a moving target to the implementation may not be trivial - you need to ensure that different clocks across your timing domain that started at different times, will be synchronised with each other while this artificial offset is being injected. You are fine as long as all your clocks and all clocks in any external time-critical systems you're interfacing with, use the same methodology - which is not always the case. This is why personally I would prefer the full leap approach, however you really need to safeguard your applications against it so that it doesn't lead to unpredictable behaviour.
Thanks
Woj
Last edit: Wojciech Owczarek 2013-02-11
Hi Woj,
[I do not have automatic notifications of the posts in sourceforge. Sorry for late thank s.]
Thank you very much for the detail response. It is indeed very informative.
I remember a similar approach to your second suggestion which is implemented by Google in their internal systems to handling the leap second. http://googleblog.blogspot.com/2011/09/time-technology-and-leaping-seconds.html
This solution seems reasonable to me, especially to avoid the problems that you mentioned with regard to clock_gettime/gettimeofday. What do you think? Do you have any opinion on this solution?
A related question, the use of CLOCK_ABSTIME is discouraged to avoid any problem related to leap second. Is there any common design patter/solution to still use CLOCK_ABSTIME based timers while not facing any problem of the current leap seconds handling in the kernel?