Server offset included in served time? - NTP

This is a discussion on Server offset included in served time? - NTP ; Hi, Does an NTP servers take into account it's estimated offset in serving time to others? If I am a server and think I am 1.5 milliseconds off from true time, will I include this in the timestamps of my ...

+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 20 of 27

Thread: Server offset included in served time?

  1. Server offset included in served time?

    Hi,

    Does an NTP servers take into account it's estimated offset in serving time
    to others? If I am a server and think I am 1.5 milliseconds off from true
    time, will I include this in the timestamps of my ntp replies to others?

    Thanks

    Howard Barina

  2. Re: Server offset included in served time?

    Howard Barina wrote:
    >
    > Does an NTP servers take into account it's estimated offset in serving time


    There seem to have been a lot of questions asked in the last month that
    are based on the false assumption that "offset" measures the difference
    between the local clock and true time. Has someone published a
    misleading document, somewhere?

    > to others? If I am a server and think I am 1.5 milliseconds off from true
    > time, will I include this in the timestamps of my ntp replies to others?


    An NTP client never thinks that its time is wrong. If it did, it would
    be admitting that the NTP algorithms are wrong. Therefore the server
    always serves its client's idea of the time.

    The "offset" should always be within the statistical error from the
    current measurement history (if ntpd suspects otherwise, it steps,
    and/or reduces the poll interval, to try to rapidly re-acquire that
    condition). Immediately after startup, there will be little history, so
    quite large offsets will still be consistent with that history.

    There are people who who argue that the NTP algorithms are fundamentally
    flawed and don't give the statistically best time in real world
    situations. I think they have some credibility, but NTP's inventor,
    does not.

    However, what one can certainly say is that combining "offset" with the
    local clock is very unlikely to give you a better time than just using
    the local time at all times except immediately after startup or after
    significant upsets.

    "offset" is the difference between the clients best estimate of the true
    time and a weighted average of the most recent "best" measurements from
    its upstream servers. Those measurements have a significant measurement
    error.

  3. Re: Server offset included in served time?

    David Woolley wrote:
    > Howard Barina wrote:
    >>
    >> Does an NTP servers take into account it's estimated offset in serving
    >> time

    >
    > There seem to have been a lot of questions asked in the last month that
    > are based on the false assumption that "offset" measures the difference
    > between the local clock and true time. Has someone published a
    > misleading document, somewhere?
    >
    >> to others? If I am a server and think I am 1.5 milliseconds off from
    >> true
    >> time, will I include this in the timestamps of my ntp replies to others?

    >
    > An NTP client never thinks that its time is wrong. If it did, it would
    > be admitting that the NTP algorithms are wrong. Therefore the server
    > always serves its client's idea of the time.
    >
    > The "offset" should always be within the statistical error from the
    > current measurement history (if ntpd suspects otherwise, it steps,
    > and/or reduces the poll interval, to try to rapidly re-acquire that
    > condition). Immediately after startup, there will be little history, so
    > quite large offsets will still be consistent with that history.
    >
    > There are people who who argue that the NTP algorithms are fundamentally
    > flawed and don't give the statistically best time in real world
    > situations. I think they have some credibility, but NTP's inventor,
    > does not.
    >


    I hope you did not mean to say that David Mills has no credibility but
    what you wrote sure reads that way!!

  4. Re: Server offset included in served time?

    Richard B. Gilbert wrote:
    > David Woolley wrote:


    >> There are people who who argue that the NTP algorithms are
    >> fundamentally flawed and don't give the statistically best time in
    >> real world situations. I think they have some credibility, but NTP's
    >> inventor, does not.
    >>

    >
    > I hope you did not mean to say that David Mills has no credibility but
    > what you wrote sure reads that way!!


    ".....think that way" was supposed to be elided.

  5. Re: Server offset included in served time?

    "Richard B. Gilbert" writes:

    >David Woolley wrote:
    >> Howard Barina wrote:
    >>>
    >>> Does an NTP servers take into account it's estimated offset in serving
    >>> time

    >>
    >> There seem to have been a lot of questions asked in the last month that
    >> are based on the false assumption that "offset" measures the difference
    >> between the local clock and true time. Has someone published a
    >> misleading document, somewhere?
    >>
    >>> to others? If I am a server and think I am 1.5 milliseconds off from
    >>> true
    >>> time, will I include this in the timestamps of my ntp replies to others?

    >>
    >> An NTP client never thinks that its time is wrong. If it did, it would
    >> be admitting that the NTP algorithms are wrong. Therefore the server
    >> always serves its client's idea of the time.
    >>
    >> The "offset" should always be within the statistical error from the
    >> current measurement history (if ntpd suspects otherwise, it steps,
    >> and/or reduces the poll interval, to try to rapidly re-acquire that
    >> condition). Immediately after startup, there will be little history, so
    >> quite large offsets will still be consistent with that history.
    >>
    >> There are people who who argue that the NTP algorithms are fundamentally
    >> flawed and don't give the statistically best time in real world
    >> situations. I think they have some credibility, but NTP's inventor,
    >> does not.
    >>


    >I hope you did not mean to say that David Mills has no credibility but
    >what you wrote sure reads that way!!


    Just because A has credibility does not mean that B does not. credibility
    is not an exclusive property. There can be disagreements between honest men
    without one having "no credibility".

    Note that the NTP algorithms are NOT fundamentally flawed. They are great.
    They are just, in my opinion and measurements, not the best.

  6. Re: Server offset included in served time?

    On 2008-09-12, David Woolley wrote:
    > Howard Barina wrote:
    >>
    >> Does an NTP servers take into account it's estimated offset in serving time
    >> to others? If I am a server and think I am 1.5 milliseconds off from true
    >> time, will I include this in the timestamps of my ntp replies to others?

    >
    > An NTP client never thinks that its time is wrong. If it did, it would
    > be admitting that the NTP algorithms are wrong. Therefore the server
    > always serves its client's idea of the time.
    >
    > The "offset" should always be within the statistical error from the
    > current measurement history (if ntpd suspects otherwise, it steps,
    > and/or reduces the poll interval, to try to rapidly re-acquire that
    > condition). Immediately after startup, there will be little history, so
    > quite large offsets will still be consistent with that history.


    A single snapshot value is of limited value because values such as the
    offset are contantly changing. The long term stability of the clock is
    more important than a single snapshot.

    peer.awk from the ./scripts directory in the Distribution is good tool
    for summarizing peerstats files.

    --
    Steve Kostecke
    NTP Public Services Project - http://support.ntp.org/

  7. Re: Server offset included in served time?

    Steve Kostecke writes:

    >On 2008-09-12, David Woolley wrote:
    >> Howard Barina wrote:
    >>>
    >>> Does an NTP servers take into account it's estimated offset in serving time
    >>> to others? If I am a server and think I am 1.5 milliseconds off from true
    >>> time, will I include this in the timestamps of my ntp replies to others?

    >>
    >> An NTP client never thinks that its time is wrong. If it did, it would
    >> be admitting that the NTP algorithms are wrong. Therefore the server
    >> always serves its client's idea of the time.
    >>
    >> The "offset" should always be within the statistical error from the
    >> current measurement history (if ntpd suspects otherwise, it steps,
    >> and/or reduces the poll interval, to try to rapidly re-acquire that
    >> condition). Immediately after startup, there will be little history, so
    >> quite large offsets will still be consistent with that history.


    >A single snapshot value is of limited value because values such as the
    >offset are contantly changing. The long term stability of the clock is
    >more important than a single snapshot.


    Of course any ntp query is a single snapshot.


    >peer.awk from the ./scripts directory in the Distribution is good tool
    >for summarizing peerstats files.




  8. Re: Server offset included in served time?

    David Woolley wrote:
    > Howard Barina wrote:
    >>
    >> Does an NTP servers take into account it's estimated offset in serving
    >> time

    [...]
    > An NTP client never thinks that its time is wrong. If it did, it would
    > be admitting that the NTP algorithms are wrong. Therefore the server
    > always serves its client's idea of the time.
    >
    > The "offset" should always be within the statistical error from the
    > current measurement history (if ntpd suspects otherwise, it steps,
    > and/or reduces the poll interval, to try to rapidly re-acquire that
    > condition). Immediately after startup, there will be little history, so
    > quite large offsets will still be consistent with that history.


    But what about the behaviour shortly after startup? The NTP daemon tries to
    determine the initial time offset from its upstream sources. Unless that
    initial offset exceeds the 128 ms limit it starts to slew its system time
    *very* slowly until the frequency drift has been compensated and the
    estimated time offset has been minimized.

    While the system time is being slewed it may be e.g. 120 ms off, and when
    the daemon sends the system time to its clients then it will serve a time
    which is 120 ms off.

    Martin
    --
    Martin Burnicki

    Meinberg Funkuhren
    Bad Pyrmont
    Germany

  9. Re: Server offset included in served time?

    Unruh wrote:

    > Just because A has credibility does not mean that B does not. credibility
    > is not an exclusive property. There can be disagreements between honest men
    > without one having "no credibility".
    >
    > Note that the NTP algorithms are NOT fundamentally flawed. They are great.
    > They are just, in my opinion and measurements, not the best.


    Of course, a lot of the issue resides with the definition of *best*.
    Bill has found that in his environment, the algorithms used by chrony
    preform better than those used by NTP. He has empirical data to back
    this up. He also believes that his environment is fairly typical,
    which is his opinion and more speculative. So for him, NTP's
    algorithms are not the *best*.

    My recent experiments show that the same algorithms that are great in
    NTP for the long haul, actually hinder its ability to get the offset
    and frequency right under many typical startup conditions. Dr. Mills
    isn't really interested in fixing this, not because he denies it, but
    because he doesn't think it is the best area for his labors. He is
    more interested in the long-haul.

    Brian Utterback

  10. Re: Server offset included in served time?

    Brian Utterback wrote:
    > Unruh wrote:
    >
    >> Just because A has credibility does not mean that B does not. credibility
    >> is not an exclusive property. There can be disagreements between
    >> honest men
    >> without one having "no credibility".
    >>
    >> Note that the NTP algorithms are NOT fundamentally flawed. They are
    >> great.
    >> They are just, in my opinion and measurements, not the best.

    >
    > Of course, a lot of the issue resides with the definition of *best*.
    > Bill has found that in his environment, the algorithms used by chrony
    > preform better than those used by NTP. He has empirical data to back
    > this up. He also believes that his environment is fairly typical, which
    > is his opinion and more speculative. So for him, NTP's algorithms are
    > not the *best*.
    >
    > My recent experiments show that the same algorithms that are great in
    > NTP for the long haul, actually hinder its ability to get the offset and
    > frequency right under many typical startup conditions. Dr. Mills isn't
    > really interested in fixing this, not because he denies it, but because
    > he doesn't think it is the best area for his labors. He is more
    > interested in the long-haul.


    This may be a problem for those who reboot their systems daily (or more
    frequently). My NTP server has an uptime of 121 days at the moment. I
    expect that it will remain up until the next time the power goes out for
    longer than the run time of my UPS.

  11. Re: Server offset included in served time?

    On 12 Sep, 22:14, David Woolley
    wrote:
    > Howard Barina wrote:
    >
    > > Does an NTP servers take into account it's estimated offset in serving time

    >
    > There seem to have been a lot of questions asked in the last month that
    > are based on the false assumption that "offset" measures the difference
    > between the local clock and true time. *Has someone published a
    > misleading document, somewhere?

    I think I am one of the people who has made this mistake.

    > "offset" is the difference between the clients best estimate of the true
    > time and a weighted average of the most recent "best" measurements from
    > its upstream servers. *Those measurements have a significant measurement
    > error.

    So I can't plot the offsets in the loopstats file to show how accurate
    the time was on an NTP client over a period of time?

    How can I obtain a historic record of the difference between true time
    and local clock time on an NTP client?

    Thanks

    Mike

  12. Re: Server offset included in served time?

    Mike K Smith wrote:

    > How can I obtain a historic record of the difference between true time
    > and local clock time on an NTP client?


    You need a source of true time that is, say, an order magnitude better
    than your expectation of the local clock. The problem, of course, is
    that if you had access to such a source of time, the obvious thing to do
    with it would be to discipline the local clock! That would leave you
    needing a source of time that is yet another order of magnitude better;
    loop.

  13. Re: Server offset included in served time?

    Martin Burnicki wrote:

    >
    > But what about the behaviour shortly after startup? The NTP daemon tries to
    > determine the initial time offset from its upstream sources. Unless that
    > initial offset exceeds the 128 ms limit it starts to slew its system time
    > *very* slowly until the frequency drift has been compensated and the
    > estimated time offset has been minimized.


    I've had some thoughts about this. As I see it the problems are:

    - ntpd doesn't have any persistent history of jitter, so has to start by
    assuming that the jitter is of the same order of magnitude as the offset
    (what people looking at offset often forget is that they have the
    benefit of hindsight).

    - ntpd is already at the shortest permitted time constant, and going
    lower would require faster polling, or compromising the level of
    oversampling, or length of the initial best measurement filter. It is
    this lower bound on the time constant that means that ntpd can get into
    a position where it should know that the time is wrong, but cannot fix
    it quickly.

    - the step limit is fixed at configuration time.

    One could deal with the first by making the smoothed jitter be
    persistent. That way ntpd can detect whether its offsets exceed
    reasonable jitter for the system, before it has enough history for the
    session to know the jitter from measurements just in the current session.

    Once one knows that offsets are high compared with jitter, one can
    address the time constant issue. Normally jitter << offset would tend
    to force the time constant down, but is has nowhere to go down. Maybe
    what is needed is to allow the degree of oversampling to compromised
    until one first begins to get offsets of the same order as the jitter.
    Maybe also use less then 8 filter slots.

    This may compromise the stability of downstream systems, so it may be
    necessary to stay in an alarm state until this stage of the pocess is
    complete. This may be a problem for people who want a whole network to
    power up at the same time, and quickly.

    If there were also a persistent record of a high percentile figure for
    the offset, one could also use that to set the step threshold during the
    startup phase, maybe reverting to the standard value, later, to give
    better tolerance of major network problems.
    >
    > While the system time is being slewed it may be e.g. 120 ms off, and when
    > the daemon sends the system time to its clients then it will serve a time
    > which is 120 ms off.


    To some extent the fact that systems are already experiencing this
    suggests, to me, that one might not need to alarm the time during a
    temporary short loop time constant phase at startup.

  14. Re: Server offset included in served time?

    Mike K Smith writes:

    >On 12 Sep, 22:14, David Woolley
    >wrote:
    >> Howard Barina wrote:
    >>
    >> > Does an NTP servers take into account it's estimated offset in serving =

    >time
    >>
    >> There seem to have been a lot of questions asked in the last month that
    >> are based on the false assumption that "offset" measures the difference
    >> between the local clock and true time. =A0Has someone published a
    >> misleading document, somewhere?

    >I think I am one of the people who has made this mistake.


    >> "offset" is the difference between the clients best estimate of the true
    >> time and a weighted average of the most recent "best" measurements from
    >> its upstream servers. =A0Those measurements have a significant measuremen=

    >t
    >> error.

    >So I can't plot the offsets in the loopstats file to show how accurate
    >the time was on an NTP client over a period of time?


    That is the best estimate that you have. Alternatively buy a GPS hook it up
    and measure the offset against that.



    >How can I obtain a historic record of the difference between true time
    >and local clock time on an NTP client?


    You need a source for "true time". A gps receiver will give you one with an
    accuracy of something like microseconds.



    >Thanks


    >Mike


  15. Re: Server offset included in served time?

    Unruh wrote:
    > Mike K Smith writes:


    >> So I can't plot the offsets in the loopstats file to show how accurate
    >> the time was on an NTP client over a period of time?

    >
    > That is the best estimate that you have. Alternatively buy a GPS hook it up
    > and measure the offset against that.


    But, if you use individual measurements, you will get a figure that,
    most of the time, is several times the true error and not necessarily in
    the right direction.

    What you can do, is to use some hindsight, and make a slightly better
    estimate of the true time by combining offsets from both before and
    after the time the local clock was read. That gives you an advantage
    during startup and after transients, but, in the steady state, the local
    clock time is going to be the same as such an enhanced statistic.

    In the steady state, all you can really deduce from the offsets is the
    amount of noise in the measurements. You can then expect the amount of
    noise in the local clock time to be several times less.

  16. Re: Server offset included in served time?

    David Woolley wrote:
    > Martin Burnicki wrote:


    >> While the system time is being slewed it may be e.g. 120 ms off, and when
    >> the daemon sends the system time to its clients then it will serve a time
    >> which is 120 ms off.

    >
    >
    > To some extent the fact that systems are already experiencing this
    > suggests, to me, that one might not need to alarm the time during a
    > temporary short loop time constant phase at startup.


    Does ntp serve "system" time or "correct" time to clients?

    uwe

  17. Re: Server offset included in served time?

    Uwe Klein wrote:

    >
    > Does ntp serve "system" time or "correct" time to clients?


    NTP serves system time, which its algorithms believe to be the best
    practical estimate of correct time within statistical error. What we
    are talking about here is making that assumption more valid during
    startup transients.

    Basically, as far as NTP is concerned, there is no difference between
    system time and correct time. (Earlier versions used to serve a
    corrected time during slew recoveries, but not during normal operation.
    That would not make a difference for the 120ms case discussed here, as
    that uses the normal control loop.)

  18. Re: Server offset included in served time?

    David Woolley wrote:
    > Martin Burnicki wrote:
    >> But what about the behaviour shortly after startup? The NTP daemon tries
    >> to determine the initial time offset from its upstream sources. Unless
    >> that initial offset exceeds the 128 ms limit it starts to slew its system
    >> time *very* slowly until the frequency drift has been compensated and the
    >> estimated time offset has been minimized.

    >
    > I've had some thoughts about this. As I see it the problems are:
    >
    > - ntpd doesn't have any persistent history of jitter, so has to start by
    > assuming that the jitter is of the same order of magnitude as the offset
    > (what people looking at offset often forget is that they have the
    > benefit of hindsight).


    Basically I agree. However, if the packet turnaround time is low, e.g below
    1 millisecond on a local net, and evaluation of the timestamps yields an
    offset of about 120 ms (according to my example shortly after startup) then
    the result clearly indicates the own system time is off by 120 ms.

    > - ntpd is already at the shortest permitted time constant, and going
    > lower would require faster polling, or compromising the level of
    > oversampling, or length of the initial best measurement filter. It is
    > this lower bound on the time constant that means that ntpd can get into
    > a position where it should know that the time is wrong, but cannot fix
    > it quickly.


    Yes, that's what I consider a real limitation of ntpd.

    Of course if you rely to some internet servers then there may be rather
    large turnaround times/delays. However, given the upstream server sends
    correct time stamps then then the error due to packet delays on the network
    can not exceed the packet turnaround interval. So if the turnaround
    interval is low (e.g. < 1 ms) and the time offset is high (e.g. ~120 ms)
    then ntpd should indeed start compensate the time offset much quicklier
    than it currently does.

    I often hear from NTP users (here in the NG, and from our customers) that
    this is the expected behaviour.

    > - the step limit is fixed at configuration time.
    >
    > One could deal with the first by making the smoothed jitter be
    > persistent. That way ntpd can detect whether its offsets exceed
    > reasonable jitter for the system, before it has enough history for the
    > session to know the jitter from measurements just in the current session.
    >
    > Once one knows that offsets are high compared with jitter, one can
    > address the time constant issue. Normally jitter << offset would tend
    > to force the time constant down, but is has nowhere to go down. Maybe
    > what is needed is to allow the degree of oversampling to compromised
    > until one first begins to get offsets of the same order as the jitter.
    > Maybe also use less then 8 filter slots.


    That's what I mean. If the measurement results undoubtly indicte the time
    offset is there then ntpd could immediately start to slew it away.

    > This may compromise the stability of downstream systems, so it may be
    > necessary to stay in an alarm state until this stage of the pocess is
    > complete. This may be a problem for people who want a whole network to
    > power up at the same time, and quickly.


    If a complete network is powered up, inclusive of the time server, and the
    initial time offset of the time server is 120 ms and is only slewed down
    after hours, do you really thing it's better to slew the time that slowly
    on the time server and providing a time to his clients which is 120 ms off?

    So the clients would start to synchronize to a reference time which is off
    by 120 ms and then follow the slow corrections of the time server?

    Alternatively, if the server would quickly slew its initial time offset then
    the clients would initially see a more accurate reference time and thus
    would earlier have an accurate time themselves.

    Martin
    --
    Martin Burnicki

    Meinberg Funkuhren
    Bad Pyrmont
    Germany

  19. Re: Server offset included in served time?

    David Woolley wrote:
    > Uwe Klein wrote:
    >
    >>
    >> Does ntp serve "system" time or "correct" time to clients?

    >
    >
    > NTP serves system time, which its algorithms believe to be the best
    > practical estimate of correct time within statistical error. What we
    > are talking about here is making that assumption more valid during
    > startup transients.
    >
    > Basically, as far as NTP is concerned, there is no difference between
    > system time and correct time. (Earlier versions used to serve a

    _________________________________^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^
    > corrected time during slew recoveries, but not during normal operation.

    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    > That would not make a difference for the 120ms case discussed here, as
    > that uses the normal control loop.)


    The second paragraph seems to answer my question. ( afaiu )

    This would implicate that clients syncing to a ( both freshly started )
    slewing server will be presented with an offset that is
    ( to make it interesting ) slewing too.

    Doesn't this lead clients to first slew towards the servers offset and
    than in a chained way slew back to the "right" time together?
    ( assumed the client came up at the same time and with a better/other match )

    uwe


  20. Re: Server offset included in served time?

    David Woolley writes:

    >Unruh wrote:
    >> Mike K Smith writes:


    >>> So I can't plot the offsets in the loopstats file to show how accurate
    >>> the time was on an NTP client over a period of time?

    >>
    >> That is the best estimate that you have. Alternatively buy a GPS hook it up
    >> and measure the offset against that.


    >But, if you use individual measurements, you will get a figure that,
    >most of the time, is several times the true error and not necessarily in
    >the right direction.


    No idea what that sentence means. Are you refering to the gps readings
    which will be at worst 2 orders of magnitude better than that offsets from
    a generic network?

    >What you can do, is to use some hindsight, and make a slightly better
    >estimate of the true time by combining offsets from both before and
    >after the time the local clock was read. That gives you an advantage


    Unfortunately those offsets are not very useful, because the clock that
    read them has had its offsets and rates changed (by ntp) since then. Ie the
    measured offsets are not a good estimate of the offsets from "true time".


    >during startup and after transients, but, in the steady state, the local
    >clock time is going to be the same as such an enhanced statistic.


    >In the steady state, all you can really deduce from the offsets is the
    >amount of noise in the measurements. You can then expect the amount of
    >noise in the local clock time to be several times less.


    No, the noise in the local clock may well dominate those offsets. This is
    what happens for example to a system which is controlled by a hardware
    refclock. The noise is mainly the temperature dependence of the local clock,
    not the measurement noise. See www.theory.physics.ubc.ca/chrony/chrony.html
    and the measurements for string, a GPS/ntp controlled machine. Much of the
    noise is rate noise from the daily variations in temperature.



+ Reply to Thread
Page 1 of 2 1 2 LastLast