tinker step 0 (always slew) and kernel time discipline - NTP

This is a discussion on tinker step 0 (always slew) and kernel time discipline - NTP ; I have an application that is sensitive to step corrections and am considering using 'tinker step 0' to disable them altogether. However, I noticed a thread on this topic in February 2005 ( http://lists.ntp.isc.org/pipermail/q...ry/004468.html ) that suggested setting 'tinker step ...

+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 20 of 40

Thread: tinker step 0 (always slew) and kernel time discipline

  1. tinker step 0 (always slew) and kernel time discipline

    I have an application that is sensitive to step corrections and am considering using 'tinker step 0' to disable them altogether. However, I noticed a thread on this topic in February 2005 (http://lists.ntp.isc.org/pipermail/q...ry/004468.html) that suggested setting 'tinker step 0' without explicitly using 'disable kernel' will essentially yield unpredictable behavior.

    So what does disable kernel do? Does it disable the NHPFL algorithm? Is this algorithm synonomous with the "kernel time discipline?" So when "disable kernel" has been used, how is the clock frequency adjusted? Also, why is the kernel time discipline disabled when a correction of > 0.5 seconds is required?

    ---
    Joe Harvell

  2. Re: tinker step 0 (always slew) and kernel time discipline

    In article ,
    Joe Harvell wrote:

    I have an application that is sensitive to step corrections and am considering using 'tinker step 0' to disable them altogether. However, I noticed a thread on this topic in February 2005 (http://lists.ntp.isc.org/pipermail/q...ry/004468.html) that suggested setting 'tinker step 0' without explicitly using 'disable kernel' will essentially yield unpredictable behavior.

    > So when "disable kernel" has been used, how is the clock frequency
    > adjusted? Also, why is the kernel time discipline disabled when a


    By doing all the calculations in user space and periodically (every 4 seconds
    on ntpd v3) calling adjtime to apply the slew correction for that second.
    The result is a sawtoothing of the phase.

    The kernel mode does the calculations every tick.

    > correction of > 0.5 seconds is required?


    I suspect the limit is imposed because something in the kernel overflows.

    PS. Please don't keep starting new threads for what is clearly part of
    a single thread.

  3. Re: tinker step 0 (always slew) and kernel time discipline

    Okay, I'll stick with this thread.

    I plan to use "tinker step 0" to prevent step corrections on an NTP client only host. I am only expecting large offsets in certain failure scenarios. In these scenarios, operator intervention is required and the clock will be stepped manually as part of a much larger procedure.

    What I want to understand is how using "tinker step 0" interacts with the kernel time discipline. Consider DM's (David Mills) 2005 post below:

    > In truth, the kernel is disabled only if the actual correction is
    > greater than 0.5 s no matter what the tinker. However, there could be
    > grief if the discipline switches from one to the other. There are a
    > rather large number of untested and probably dangerous things that
    > might happen with uncharted combinations of tinkers and I'm not
    > thrilled when spending lots of time fixing the broken combinations.
    > You are on your own, but a "diable kernel" command might be appropriate.


    When I first read this, I thought the kernel time discipline had some sort of dependency on step corrections being used for large errors. However, after reading David's "NTP Clock Discipline Principles" slides describing the NHPFL, it seems like this algorithm would be unaffected by whether or not step corrections are made. Is the concern here that if step corrections are not made (i.e. tinker step 0), then you have untested behavior resulting from a swtich from the daemon to the kernel discipline once the daemon discipline brings the offset to <0.5s?

    Also, DM made the following post later in the same thread:

    > As I said in my response to your message in my inbox, all claims on
    > stability are cancelled in the configuration you selected. I am not
    > surprised that you see large overshoots, instability and incorrect
    > behavior. There are all kinds of reasons for this, including violations
    > of the expected Unix behavior, unexpected transients between the kernel
    > and daemon feedback loops and who knows what else. The engineered
    > parameters are not designed for your application. The best choice, as I
    > said in my reply to your message, is to forget ntpd altogether and use
    > your wristwatch. You will have to analyze each and every case on your
    > own.


    Unfortunately, I can't tell from the context here whether the "selected configuration" is the one with or without "disable kernel" as part of it. It seems like DM is condemning the use of "tinker step 0" altogether. Is this in fact the case? Also, I get the "unexpected transients..." comment, but not the "violations of the expected Unix behavior." What expected Unix behavior? What violations? How would using "tinker step 0" in either configuration contribute to this?

    I would also like to point out a more recent thread (http://groups.google.com/group/comp....b7a8e20feff49f),
    in which it seems like a bug causing the behavior originally described by Nikolaev in the Feb 2005 thread was discovered and fixed. But there is no indication that the fix was ever sourced. Does anyone know whether this was in fact a bug and if it was fixed?

    Despite DM's assertion that "all claims on stability are cancelled...", I do not understand why there is a fundamental problem with using the kernel time discipline and "tinker step 0" together. What I'd really like is a detailed description of why this is the case.

    For starters, why is the kernel time discipline disabled when the offset is greater than 0.5 seconds? I do appreciate your speculative response to that question. Could someone verify whether this is in fact true?

    Now on to the "unexpected transients." If a large offset is being corrected via the daemon discipline, and then the offset gets to within 0.5 s, the kernel loop would be started, right? This should keep moving the offset towards zero, but with the NHPFL instead, right? Is there a concern that the offset could jump back and forth between the 0.5 s threshold *as a result* of switching between the daemon and kernel disciplines? Is that what DM is suggesting?

    David Woolley wrote:
    > In article ,
    > Joe Harvell wrote:
    >
    > I have an application that is sensitive to step corrections and am considering using 'tinker step 0' to disable them altogether. However, I noticed a thread on this topic in February 2005 (http://lists.ntp.isc.org/pipermail/q...ry/004468.html) that suggested setting 'tinker step 0' without explicitly using 'disable kernel' will essentially yield unpredictable behavior.
    >
    >> So when "disable kernel" has been used, how is the clock frequency
    >> adjusted? Also, why is the kernel time discipline disabled when a

    >
    > By doing all the calculations in user space and periodically (every 4 seconds
    > on ntpd v3) calling adjtime to apply the slew correction for that second.
    > The result is a sawtoothing of the phase.
    >
    > The kernel mode does the calculations every tick.
    >
    >> correction of > 0.5 seconds is required?

    >
    > I suspect the limit is imposed because something in the kernel overflows.
    >
    > PS. Please don't keep starting new threads for what is clearly part of
    > a single thread.


  4. Re: tinker step 0 (always slew) and kernel time discipline

    Joe Harvell wrote:

    > Okay, I'll stick with this thread.
    >
    > I plan to use "tinker step 0" to prevent step corrections on an NTP
    > client only host. I am only expecting large offsets in certain failure
    > scenarios. In these scenarios, operator intervention is required and
    > the clock will be stepped manually as part of a much larger procedure.
    >



    How about designing your NTP subnet in such a way as to prevent these
    failures in the first place?

    Since you say, elsewhere, that you are more concerned that time be
    strictly monotonically increasing than that it be accurate perhaps you
    don't need NTP at all; set your local clock from your wrist watch once a
    week while the application is not running

    Your original problem, IIRC, resulted from an extremely poor design of
    your NTP subnet; two servers each serving its unsynchronized local clock
    and drifting apart.

    If you really do need NTP the easiest configuration is for your client
    to use from four to seven servers. Those servers should be stratum 2
    internet servers (rules of engagement prohibit use of public stratum 1
    servers unless you are serving 100 or more clients). This requires that
    you study the list of public stratum two servers at
    http://ntp.isc.org/bin/view/Servers/...TwoTimeServers
    to find four to seven servers within, say, 300 miles of your site and
    adding these servers to your ntp.conf file. It also requires a
    connection to the internet that allows port 123 in both directions. If
    you specify the numeric IP address of each server, you need not open any
    other port in the firewall. If you wish to use domain names, the you
    will have to open the port(s) necessary to allow DNS to work (don't know
    which ones offhand.

    The simplest configuration is to make the machine running the
    application a stratum 1 server by installing ntpd and a GPS timing
    receiver as a hardware reference clock. The weakness of this
    configuration is that the GPS receiver becomes a single point of
    failure; if it dies, you rapidly lose any claim to accuracy. Since you
    don't insist on accuracy perhaps this would not be a problem. Actually,
    ntpd would continue to discipline the clock using the last known
    frequency correction so you would have several hours of "hold over"
    before your clock drifted significantly (assuming a controlled
    temperature in your data center).

    You can increase the reliability by using four GPS timing receivers to
    synchronize four NTP servers and configuring your client to use those
    four servers.


  5. Re: tinker step 0 (always slew) and kernel time discipline

    Richard:



    Richard B. Gilbert wrote:

    > How about designing your NTP subnet in such a way as to prevent these failures in the first place?
    >
    > Since you say, elsewhere, that you are more concerned that time be strictly monotonically increasing than that it be accurate perhaps you don't need NTP at all; set your local clock from your wrist watch once a week while the application is not running
    >
    > Your original problem, IIRC, resulted from an extremely poor design of your NTP subnet; two servers each serving its unsynchronized local clock and drifting apart.
    >
    > If you really do need NTP the easiest configuration is for your client to use from four to seven servers. Those servers should be stratum 2 internet servers (rules of engagement prohibit use of public stratum 1 servers unless you are serving 100 or more clients). This requires that you study the list of public stratum two servers at http://ntp.isc.org/bin/view/Servers/...TwoTimeServers
    > to find four to seven servers within, say, 300 miles of your site and adding these servers to your ntp.conf file. It also requires a connection to the internet that allows port 123 in both directions. If you specify the numeric IP address of each server, you need not open any other port in the firewall. If you wish to use domain names, the you will have to open the port(s) necessary to allow DNS to work (don't know which ones offhand.
    >
    > The simplest configuration is to make the machine running the application a stratum 1 server by installing ntpd and a GPS timing receiver as a hardware reference clock. The weakness of this configuration is that the GPS receiver becomes a single point of failure; if it dies, you rapidly lose any claim to accuracy. Since you don't insist on accuracy perhaps this would not be a problem. Actually, ntpd would continue to discipline the clock using the last known frequency correction so you would have several hours of "hold over" before your clock drifted significantly (assuming a controlled temperature in your data center).
    >
    > You can increase the reliability by using four GPS timing receivers to synchronize four NTP servers and configuring your client to use those four servers.



    Richard:

    I really appreciate the advice. I think you are getting the wrong idea
    about my approach to handling the problem since I don't seem concerned
    about the glaring problems in my configuration. The reason for this is
    the original problem manifested in a testbed for one of our products. I
    am concurrently tracking this down internally to determine whether the
    two servers are actually synced to a stratum 1 clock (or whether they
    are part of the same synchronization subnet at all). And I plan to
    correct the problem.

    Also, I completely agree that we should configure 4+ peers for each NTP
    client to avoid this failure scenario altogether.

    But keep in mind that it may not be practical for our customers to have
    4+ NTP servers in their synchronization subnet. And arguably, they
    deserve what they get if they fail to follow our recommendation to have
    more servers.

    Nevertheless, I am still very interested in preventing step corrections
    in these scenarios. And I think this is a legitimate concern. So I
    would really appreciate it if you could also address the questions in my
    post.

    Thanks

    ---
    Joe Harvell

  6. Re: tinker step 0 (always slew) and kernel time discipline

    Joseph Harvell wrote:
    > Richard:
    >
    >
    >
    > Richard B. Gilbert wrote:
    >
    >
    >>How about designing your NTP subnet in such a way as to prevent these failures in the first place?
    >>
    >>Since you say, elsewhere, that you are more concerned that time be strictly monotonically increasing than that it be accurate perhaps you don't need NTP at all; set your local clock from your wrist watch once a week while the application is not running
    >>
    >>Your original problem, IIRC, resulted from an extremely poor design of your NTP subnet; two servers each serving its unsynchronized local clock and drifting apart.
    >>
    >>If you really do need NTP the easiest configuration is for your client to use from four to seven servers. Those servers should be stratum 2 internet servers (rules of engagement prohibit use of public stratum 1 servers unless you are serving 100 or more clients). This requires that you study the list of public stratum two servers at http://ntp.isc.org/bin/view/Servers/...TwoTimeServers
    >> to find four to seven servers within, say, 300 miles of your site and adding these servers to your ntp.conf file. It also requires a connection to the internet that allows port 123 in both directions. If you specify the numeric IP address of each server, you need not open any other port in the firewall. If you wish to use domain names, the you will have to open the port(s) necessary to allow DNS to work (don't know which ones offhand.
    >>
    >>The simplest configuration is to make the machine running the application a stratum 1 server by installing ntpd and a GPS timing receiver as a hardware reference clock. The weakness of this configuration is that the GPS receiver becomes a single point of failure; if it dies, you rapidly lose any claim to accuracy. Since you don't insist on accuracy perhaps this would not be a problem. Actually, ntpd would continue to discipline the clock using the last known frequency correction so you would have several hours of "hold over" before your clock drifted significantly (assuming a controlled temperature in your data center).
    >>
    >>You can increase the reliability by using four GPS timing receivers to synchronize four NTP servers and configuring your client to use those four servers.

    >
    >
    >
    > Richard:
    >
    > I really appreciate the advice. I think you are getting the wrong idea
    > about my approach to handling the problem since I don't seem concerned
    > about the glaring problems in my configuration. The reason for this is
    > the original problem manifested in a testbed for one of our products. I
    > am concurrently tracking this down internally to determine whether the
    > two servers are actually synced to a stratum 1 clock (or whether they
    > are part of the same synchronization subnet at all). And I plan to
    > correct the problem.
    >
    > Also, I completely agree that we should configure 4+ peers for each NTP
    > client to avoid this failure scenario altogether.
    >
    > But keep in mind that it may not be practical for our customers to have
    > 4+ NTP servers in their synchronization subnet. And arguably, they
    > deserve what they get if they fail to follow our recommendation to have
    > more servers.
    >
    > Nevertheless, I am still very interested in preventing step corrections
    > in these scenarios. And I think this is a legitimate concern. So I
    > would really appreciate it if you could also address the questions in my
    > post.
    >
    > Thanks
    >
    > ---
    > Joe Harvell


    I lack the expertise to answer your question as now put! I've never
    done such a thing or needed to.

    The tinker keyword is, IMHO, well named! My understanding is that it is
    intended for "tinkering" rather than for production use. It lets you
    experiment without having to modify the code and rebuild each time.
    It's, AFAIK, unsupported; if NTP malfunctions while you are using tinker
    and you report it, the reply is likely be "then don't do that!"

    IF your customers use NTP, it's THEIR responsibility to design and
    operate their synchronization subnet properly. It's YOUR responsibility
    to warn your customers that horrible things may happen if time is
    stepped while your application is running.

    Also note that ntpd is not the only method of managing computer clocks;
    there is SNTP, "Open" NTP, the "daytime" protocol, rdate (Unix/Linux),
    etc, etc. Some people use ntpdate in a cron job and that WILL, by
    default, step the time. NTP is probably the best if you need/want
    accurate time but the other means hang on for various reasons.

  7. Re: tinker step 0 (always slew) and kernel time discipline

    Joe,

    "disable kernel" does exactly and precisely that. In that case
    corrections are handled as if the kernel code does not exist.

    Ordinarily, to disable the step correction also disables the kernel.
    Your "tinker step 0" exposed a bug, now fixed, in which the kernel was
    not automatically disabled in that case.

    Dave

    Joe Harvell wrote:
    > I have an application that is sensitive to step corrections and am
    > considering using 'tinker step 0' to disable them altogether. However,
    > I noticed a thread on this topic in February 2005
    > (http://lists.ntp.isc.org/pipermail/q...ry/004468.html)
    > that suggested setting 'tinker step 0' without explicitly using 'disable
    > kernel' will essentially yield unpredictable behavior.
    >
    > So what does disable kernel do? Does it disable the NHPFL algorithm?
    > Is this algorithm synonomous with the "kernel time discipline?" So when
    > "disable kernel" has been used, how is the clock frequency adjusted?
    > Also, why is the kernel time discipline disabled when a correction of >
    > 0.5 seconds is required?
    >
    > ---
    > Joe Harvell


  8. Re: tinker step 0 (always slew) and kernel time discipline

    David L. Mills wrote:
    > Joe,
    >
    > "disable kernel" does exactly and precisely that. In that case
    > corrections are handled as if the kernel code does not exist.
    >
    > Ordinarily, to disable the step correction also disables the kernel.
    > Your "tinker step 0" exposed a bug, now fixed, in which the kernel was
    > not automatically disabled in that case.


    In the Feb, 2005 thread, you stated that the kernel was disabled exactly if the amount of the correction was >0.5 seconds and hand nothing to do with the step threshold.

    Do you happen to know in which release of ntp the behavior was changed to what you are describing now?

    >
    > Dave
    >
    > Joe Harvell wrote:
    >> I have an application that is sensitive to step corrections and am
    >> considering using 'tinker step 0' to disable them altogether.
    >> However, I noticed a thread on this topic in February 2005
    >> (http://lists.ntp.isc.org/pipermail/q...ry/004468.html)
    >> that suggested setting 'tinker step 0' without explicitly using
    >> 'disable kernel' will essentially yield unpredictable behavior.
    >>
    >> So what does disable kernel do? Does it disable the NHPFL algorithm?
    >> Is this algorithm synonomous with the "kernel time discipline?" So
    >> when "disable kernel" has been used, how is the clock frequency
    >> adjusted? Also, why is the kernel time discipline disabled when a
    >> correction of > 0.5 seconds is required?
    >>
    >> ---
    >> Joe Harvell


  9. Re: tinker step 0 (always slew) and kernel time discipline

    Guys,

    Every few months I have to remind the group about the laws of physics
    and the evils of implementation. First, the ntpd discipline algorithm
    described in various documents and the book is basically a linear,
    type-II phase-lock loop with characteristics carefully tailored for
    optimum response. In principle, there is no limitation on the step
    threshold or even if it is disabled. However, there are gotchas:

    1. The original Unix adjtime() function uses a constant slew rate of 500
    PPM. This results in very long convergence times for a large initial
    offset of several seconds or more.

    2. In an apparently misguided attempt to "fix" this problem, Linux (and
    Solaris) added variable slew feature to adjtime(). This creates an extra
    poll in the discipline loop resulting in unstable behavior, especially
    with initial offsets of a second or more.

    3. The reason for the 0.5 second limitation in the kernel code is that
    the ntp_adjtime() function is limited to a one-second interval due to
    overflow in the 32-bit word.

    4. In the design of the nanokernel code that leaves here, the time as
    seen by the application is not stepped backward unless the kernel clock
    is stepped backward more than two seconds. So, if the kernel clock is
    stepped backward more than the step threshold and less than two seconds,
    the time as observed by an application would effectively stand still or
    advance at a low rate for the interval required. There are other
    features designed to avoid inconsistent reading of the kernel clock via
    various means in and out of the kernel. These features have been
    implemented in the stock Tru64 kernel for the Alpha, but so far as I
    know have not been implemented in other kernels.

    5. If for some reason the server(s) are not reachable at startup and the
    applications must start, then I would assume the applications would
    fail, since the time is not synchronized. If the applications use the
    NTP system primatives, the synchronization condition is readily apparent
    in the return code. Since they can't run anyway, there is no harm in
    stepping the clock, no matter what the initial offset. Forcing a slew in
    this case would seem highly undesirable, unless the application can
    tolerate large differences between clocks and, in that case, using ntpd
    is probably a poor choice in the first place.

    Dave

    David Woolley wrote:

    > In article ,
    > Joe Harvell wrote:
    >
    > I have an application that is sensitive to step corrections and am considering using 'tinker step 0' to disable them altogether. However, I noticed a thread on this topic in February 2005 (http://lists.ntp.isc.org/pipermail/q...ry/004468.html) that suggested setting 'tinker step 0' without explicitly using 'disable kernel' will essentially yield unpredictable behavior.
    >
    >
    >>So when "disable kernel" has been used, how is the clock frequency
    >>adjusted? Also, why is the kernel time discipline disabled when a

    >
    >
    > By doing all the calculations in user space and periodically (every 4 seconds
    > on ntpd v3) calling adjtime to apply the slew correction for that second.
    > The result is a sawtoothing of the phase.
    >
    > The kernel mode does the calculations every tick.
    >
    >
    >>correction of > 0.5 seconds is required?

    >
    >
    > I suspect the limit is imposed because something in the kernel overflows.
    >
    > PS. Please don't keep starting new threads for what is clearly part of
    > a single thread.


  10. Re: tinker step 0 (always slew) and kernel time discipline

    Joe,

    I expect by now most folks will look at the subject line and punch the
    delete key.

    For all practical purposes the step threshold, adjtime() slew rate,
    kernel offset limit and daemon/kernel parameters have nothing to do with
    each other. Even at large offsets the two loops can converge the time
    within one hour; however, the daemon is constrained by the adjtime()
    slew rate of 500 PPM and the kernel is constrained by the offset limit
    of +-0.5 s. There is of course the adjtime() mods in Linux and Solaris
    that make large slew adjustments unstable. This has been confirmed with
    Solaris and by your observations in Linux.

    Switching between the daemon and kernel disciplines is tricky because
    such things as the frequency, time constant, PPS discipline, etc., have
    to be matched at each switch. Avoiding steps while accepting presumably
    large initial, offsets makes the extra precision available with the
    kernel effectively useless anyway.

    I don't handle the release process, so I don't track which little thing
    that changes in the development version shows up in any particular
    release. I think your other questions have been addressed in earlier posts.

    Dave

    Joe Harvell wrote:

    > David L. Mills wrote:
    >
    >> Joe,
    >>
    >> "disable kernel" does exactly and precisely that. In that case
    >> corrections are handled as if the kernel code does not exist.
    >>
    >> Ordinarily, to disable the step correction also disables the kernel.
    >> Your "tinker step 0" exposed a bug, now fixed, in which the kernel was
    >> not automatically disabled in that case.

    >
    >
    > In the Feb, 2005 thread, you stated that the kernel was disabled exactly
    > if the amount of the correction was >0.5 seconds and hand nothing to do
    > with the step threshold.
    >
    > Do you happen to know in which release of ntp the behavior was changed
    > to what you are describing now?
    >
    >>
    >> Dave
    >>
    >> Joe Harvell wrote:
    >>
    >>> I have an application that is sensitive to step corrections and am
    >>> considering using 'tinker step 0' to disable them altogether.
    >>> However, I noticed a thread on this topic in February 2005
    >>> (http://lists.ntp.isc.org/pipermail/q...ry/004468.html)
    >>> that suggested setting 'tinker step 0' without explicitly using
    >>> 'disable kernel' will essentially yield unpredictable behavior.
    >>>
    >>> So what does disable kernel do? Does it disable the NHPFL
    >>> algorithm? Is this algorithm synonomous with the "kernel time
    >>> discipline?" So when "disable kernel" has been used, how is the
    >>> clock frequency adjusted? Also, why is the kernel time discipline
    >>> disabled when a correction of > 0.5 seconds is required?
    >>>
    >>> ---
    >>> Joe Harvell


  11. Re: tinker step 0 (always slew) and kernel time discipline

    David L. Mills wrote:

    >
    > 5. If for some reason the server(s) are not reachable at startup and the
    > applications must start, then I would assume the applications would
    > fail, since the time is not synchronized. If the applications use the
    > NTP system primatives, the synchronization condition is readily apparent
    > in the return code. Since they can't run anyway, there is no harm in
    > stepping the clock, no matter what the initial offset. Forcing a slew in
    > this case would seem highly undesirable, unless the application can
    > tolerate large differences between clocks and, in that case, using ntpd
    > is probably a poor choice in the first place.
    >


    I agree that the condition of no time servers reachable on startup is the most common case where a large offset will eventually be observed. I agree that the application should detect this and fail before starting up. I am concerned about clock and network failure scenarios that cause an NTP client to see two different NTP servers with very different times.

    This actually happened in a testbed for our application. NTP stats show that over the course of 22 days, the offsets of two configured NTP servers (both ours) serving one of our NTP clients started diverging up to a maximum distance of 800 seconds. During this time, our NTP client stepped its clock forward 940 times and backwards 803 times, with increasing magnitudes up to ~400 seconds. The problem went away when someone "added an IP address to the configuration of one of the NTP servers." (I am still trying to determine exactly what happened). The ntp.conf files of the NTP client, the stats, and a nice graph of the offsets is found at http://dingo.dogpad.net/ntpProblem/.

    I concede that only having 2 NTP servers for our host made this problem more likely to occur. But considering the mayhem caused by jerking the clock back and forth every 15 minues for 22 days, I think it is worth investigating whether to eliminate stepping altogether.

    I still don't understand why the clock was being stepped back and forth. One of the NTP servers showed up with 80f4 (unreachable) status every 15 minutes for the entire 22 days, but with 90f4 (reject) and 96f4 (sys.peer) in between. Oddly, this server was one of two servers, but the *other* server was the preferred peer. I wonder why this peer would ever be selected as the sys.peer since the prefer peer was only reported unreachable 10 times over this 22 day period. Would this be because the selection algorithm finds no intersection?

    Maybe the behavior I saw was a bug, and not the expected consequence of a failure scenario in which 2 NTP servers have diverging clocks.

  12. Re: tinker step 0 (always slew) and kernel time discipline

    Joe,

    Dave's patch will be in the next ntp-dev tarball snapshot roll, which will
    happen automatically in about 14 hours' time, and could happen sooner if I
    "push".

    If you want to see the patch and apply it yourself, in a little while it
    will be in both the bk-ntp-dev-send@ mailing list archives and in (what I
    trust will be) the suitably-commented changeset visible in the ntp-dev tree
    at http://ntp.bkbits.com .

    H

  13. Re: tinker step 0 (always slew) and kernel time discipline

    Richard B. Gilbert wrote:
    > Joe Harvell wrote:
    >
    >> David L. Mills wrote:
    >>


    >> I concede that only having 2 NTP servers for our host made this
    >> problem more likely to occur. But considering the mayhem caused by
    >> jerking the clock back and forth every 15 minues for 22 days, I think
    >> it is worth investigating whether to eliminate stepping altogether.
    >>

    >
    > Why didn't anyone notice the problem for 22 days? If, indeed, it caused
    > mayhem, why was it allowed to continue for so long?


    I see your point. I don't know for sure if it really caused problems. I suspect I will begin to see a large number of bug reports coming out of this test lab once they start filtering back to the design team. But it is quite possible there weren't any big problems or they went unnoticed. It really depends on the type of testing they were performing. This application is a call processing application, implementing call signaling protocols, and a host of other proprietary protocols for OAM (Operations, Administration, Maintenance) of the software itself. The big problems I would expect to have occurred fall into two categories: 1) problems stemming from protocol timers expiring both early and late; and 2) accounting records for the calls themselves showing inaccurate (including negative) duration. The software that did notice the problem was the software responsible for journaling application state from one process to another, as part of a 1+1 fault tolerance system. Th
    is software was measuring round-trip latencies between it and its mate by bouncing a measurement from its own clock off of its mate and then re-sampling its own clock to see the RTT. These RTT measurements only takes place during failure recovery scenarios, which is what was being tested at the time.

    Since our customers are telecommunications service providers, I expect they would notice negative durations for their billing records. I am trying to prevent this from ever occurring. However, based on the response I've received from Dr. Mills in this thread, it seems like the daemon feedback loop is unstable as a result of OS developers implementing variable slew rates into adjtime. So it looks like if we continue with NTP, the better choice is to use the kernel time discipline for stable time. We will have to engineer the network so that multiple failures would be required to necessitate the stepping in the first place.

    I wonder if it would be good to add a description to the NTP FAQ about this? The key points to include I think should be why the kernel time discipline is disabled when the step threshold is changed, and also some indication that the daemon feedback loop is broken to begin with. I am not the first person to go down this path.

    Thanks again for your responses.

    ---
    Joe Harvell

  14. Re: tinker step 0 (always slew) and kernel time discipline

    Joe Harvell wrote:

    > Richard B. Gilbert wrote:
    >
    >> Joe Harvell wrote:
    >>
    >>> David L. Mills wrote:
    >>>

    >
    >
    >
    >>> I concede that only having 2 NTP servers for our host made this
    >>> problem more likely to occur. But considering the mayhem caused by
    >>> jerking the clock back and forth every 15 minues for 22 days, I think
    >>> it is worth investigating whether to eliminate stepping altogether.
    >>>

    >>
    >> Why didn't anyone notice the problem for 22 days? If, indeed, it
    >> caused mayhem, why was it allowed to continue for so long?

    >
    >
    > I see your point. I don't know for sure if it really caused problems.
    > I suspect I will begin to see a large number of bug reports coming out
    > of this test lab once they start filtering back to the design team. But
    > it is quite possible there weren't any big problems or they went
    > unnoticed. It really depends on the type of testing they were
    > performing. This application is a call processing application,
    > implementing call signaling protocols, and a host of other proprietary
    > protocols for OAM (Operations, Administration, Maintenance) of the
    > software itself. The big problems I would expect to have occurred fall
    > into two categories: 1) problems stemming from protocol timers expiring
    > both early and late; and 2) accounting records for the calls themselves
    > showing inaccurate (including negative) duration. The software that did
    > notice the problem was the software responsible for journaling
    > application state from one process to another, as part of a 1+1 fault
    > tolerance system. Th
    > is software was measuring round-trip latencies between it and its mate
    > by bouncing a measurement from its own clock off of its mate and then
    > re-sampling its own clock to see the RTT. These RTT measurements only
    > takes place during failure recovery scenarios, which is what was being
    > tested at the time.
    >
    > Since our customers are telecommunications service providers, I expect
    > they would notice negative durations for their billing records. I am
    > trying to prevent this from ever occurring. However, based on the
    > response I've received from Dr. Mills in this thread, it seems like the
    > daemon feedback loop is unstable as a result of OS developers
    > implementing variable slew rates into adjtime. So it looks like if we
    > continue with NTP, the better choice is to use the kernel time
    > discipline for stable time. We will have to engineer the network so
    > that multiple failures would be required to necessitate the stepping in
    > the first place.
    >
    > I wonder if it would be good to add a description to the NTP FAQ about
    > this? The key points to include I think should be why the kernel time
    > discipline is disabled when the step threshold is changed, and also some
    > indication that the daemon feedback loop is broken to begin with. I am
    > not the first person to go down this path.
    >
    > Thanks again for your responses.
    >
    > ---
    > Joe Harvell


    The telephone companies tend to be very aware of time and timing. The
    time division multiplexing of T1 and T3 lines requires splitting the
    second very precisely. Cellular phones also require very precise
    timing. I would expect them to be able to provide your application with
    an ultra accurate and ultra stable time time signal of some sort.

    I think you are okay as long as the customer knows that the time must
    NOT step.

    Agree about the documentation; there are many things that are not as
    well documented as they might be. OTOH, as long as you configure and
    operate ntpd as designed, it tends to work very well.

  15. Re: tinker step 0 (always slew) and kernel time discipline

    In article ,
    Joe Harvell wrote:

    > This actually happened in a testbed for our application. NTP stats show

    * that over the course of 22 days, the offsets of two configured NTP
    * servers (both ours) serving one of our NTP clients started diverging
    * up to a maximum distance of 800 seconds. During this time, our NTP

    This could only happen if either the implementation was broken, or
    they were mis-using the local clock pseudo reference clock. If the
    servers were using a proper reference clock as their primary source,
    root dispersion would have exceeded it's maximum value when the
    error was probably a lot less than a second and the servers would have
    been rejected completely.

    Configuring a local clock breaks this process, so should never be done
    as default (even though distributors like doing this). In many cases,
    it is best not to have a local reference clock configured at all. If you
    do have more than one configured, you should arrange make each server have
    a different stratum, with steps of two between them, so that there is a
    a well defined priority amongst the different machines.

    If you don't have any real reference clocks in the overall network, it
    is even more important that there is normally only one possible choice
    of local clock reference. Having two local clock references that are
    diverging violates the fundamental principle that all NTP times are
    traceable to a single (and preferably UTC) time.

    * client stepped its clock forward 940 times and backwards 803 times,
    * with increasing magnitudes up to ~400 seconds. The problem went away
    * when someone "added an IP address to the configuration of one of the
    * NTP servers." (I am still trying to determine exactly what happened).

    That sounds like that server had a local reference clock as
    fallback.

    * The ntp.conf files of the NTP client, the stats, and a nice graph of
    * the offsets is found at http://dingo.dogpad.net/ntpProblem/.

    > I concede that only having 2 NTP servers for our host made this problem

    * more likely to occur. But considering the mayhem caused by jerking the
    * clock back and forth every 15 minues for 22 days, I think it is worth
    * investigating whether to eliminate stepping altogether.

    15 minutes sounds like the verification before ntpd becomes convinced
    that its time really is seriously wrong.

    > I still don't understand why the clock was being stepped back and forth.

    * One of the NTP servers showed up with 80f4 (unreachable) status every
    * 15 minutes for the entire 22 days, but with 90f4 (reject) and 96f4
    * (sys.peer) in between. Oddly, this server was one of two servers,
    * but the *other* server was the preferred peer. I wonder why this peer

    Normally, I believe, if you have just two servers and they have non-
    intersecting error bounds, they will both be rejected and the
    system will free run. However, I think that prefer confuses the issue,
    by not allowing the preferred one to be discarded. I have a feeling
    this is actually done by saying that the system stops discarding when
    it would discard that one. I suppose that the other one could still
    be in contention at that point.

    * would ever be selected as the sys.peer since the prefer peer was only
    * reported unreachable 10 times over this 22 day period. Would this be
    * because the selection algorithm finds no intersection?

    > Maybe the behavior I saw was a bug, and not the expected consequence of

    * a failure scenario in which 2 NTP servers have diverging clocks.

    The expected behaviour is that this has happened because one is giving
    a false time and the other is giving UTC time. The remaining servers
    will also give UTC time, so the bad one will get voted out.

    I don't think prefer is intended to deal with broken clocks, only with
    more accurate ones.

  16. Re: tinker step 0 (always slew) and kernel time discipline

    David Woolley wrote:

    > In article ,
    > Joe Harvell wrote:
    >
    >
    >>This actually happened in a testbed for our application. NTP stats show

    >
    > * that over the course of 22 days, the offsets of two configured NTP
    > * servers (both ours) serving one of our NTP clients started diverging
    > * up to a maximum distance of 800 seconds. During this time, our NTP
    >
    > This could only happen if either the implementation was broken, or
    > they were mis-using the local clock pseudo reference clock. If the
    > servers were using a proper reference clock as their primary source,
    > root dispersion would have exceeded it's maximum value when the
    > error was probably a lot less than a second and the servers would have
    > been rejected completely.
    >



    This just what happened. Both servers seem to have been serving their
    unsynchronized local clocks. The servers diverged and the client had no
    way to judge which one was more nearly correct. If the configuration
    was not designed to fail, it certainly was left prone to failure of just
    the sort that actually happened.

    He would have been better off not using NTP at all! The clock might not
    have been accurate but it would never have stepped.




  17. Re: tinker step 0 (always slew) and kernel time discipline

    Thanks for the information about fallback local reference clocks. This will help me when I talk to whomever is responsible for the NTP configuration in this lab.

    > Normally, I believe, if you have just two servers and they have non-
    > intersecting error bounds, they will both be rejected and the
    > system will free run. However, I think that prefer confuses the issue,
    > by not allowing the preferred one to be discarded. I have a feeling
    > this is actually done by saying that the system stops discarding when
    > it would discard that one. I suppose that the other one could still
    > be in contention at that point.
    >


    I agree that the "prefer" keyword doesn't make sense in the case of an NTP client without a physical clock.

    Documentation about the "prefer" keyword states that if the prefer peer survives the selection algorithm, then it will always survive the clustering algorithm. Additionally, if the prefer peer does survive, then the combining algorithm is not used, and the "prefer" peer's clock is supposed to be the only clock used when correcting the local clock.

    So if the two servers have non-intersecting error bounds, then I would expect both peers (including the "prefer" peer) to not survive the selection algorithm, and the local clock should free run.

    I wonder if whoever decided to use the "prefer" keyword thought it would serve the same purpose as putting the fallback reference clocks at different strata.

    I'm not sure I can readily reproduce the scenario without the "prefer" keyword being used. But I can look into doing this if it would help.


    > The expected behaviour is that this has happened because one is giving
    > a false time and the other is giving UTC time. The remaining servers
    > will also give UTC time, so the bad one will get voted out.
    >
    > I don't think prefer is intended to deal with broken clocks, only with
    > more accurate ones.


  18. Re: tinker step 0 (always slew) and kernel time discipline

    Richard B. Gilbert wrote:
    > David Woolley wrote:
    >
    >> In article ,
    >> Joe Harvell wrote:


    >
    > He would have been better off not using NTP at all! The clock might not
    > have been accurate but it would never have stepped.
    >
    >
    >


    I couldn't agree more I hope you guys realize this configuration is someone else's doing. I'm just trying to get it fixed.



  19. Re: tinker step 0 (always slew) and kernel time discipline

    Joe Harvell wrote:

    > Richard B. Gilbert wrote:
    >
    >> David Woolley wrote:
    >>
    >>> In article ,
    >>> Joe Harvell wrote:

    >
    >
    >>
    >> He would have been better off not using NTP at all! The clock might
    >> not have been accurate but it would never have stepped.
    >>
    >>
    >>

    >
    > I couldn't agree more I hope you guys realize this configuration is
    > someone else's doing. I'm just trying to get it fixed.
    >
    >

    The first pointed question to ask: "Was anyone at all minding the store
    for those 22 days?" I mean it looks as if no one was paying any
    attention to the NTP servers and no one was paying any attention to the
    test system. If someone was using it to test something, he couldn't
    have been paying close attention to the results.

    The second question to ask is "Couldn't the application be doing some
    checking for 'impossible' results such as a call ending before it began?"

  20. Re: tinker step 0 (always slew) and kernel time discipline

    'prefer' on a local refclock means something different from 'prefer' on a
    real refclock.

    H