Speed of ntp convergence

"Richard B. Gilbert" <rgilbert88@comcast.net> writes:
[color=blue]
>Unruh wrote:[color=green]
>> "Richard B. Gilbert" <rgilbert88@comcast.net> writes:
>>[color=darkred]
>>> Unruh wrote:
>>>> "David J Taylor" <david-taylor@blueyonder.neither-this-part.nor-this-bit.co.uk> writes:
>>>>
>>>>> Hal Murray wrote:
>>>>>>>> Try switching it off, changing the value int he drift file by say
>>>>>>>> 50PPM and
>>>>>>>> then switching it on again, and see how long it takes to recover
>>>>>>>> from that.
>>>>>>> Why would I do that? The drift values rarely change by more than
>>>>>>> five, certainly not by 50. If you are seeing a change of 50, then
>>>>>>> perhaps that it part of your problem?
>>>>>> A big step like that makes it easy to see how the system responds.
>>>>>> At least if it's a linear system. :)
>>>>> Yes, I appreciate that, Hal, but it doesn't emulate the situation here
>>>>> very well, which I understood to be slow convergence after a routine
>>>>> start. It sounds as if the OP may have an incorrect drift file - it's
>>>>> worth checking that it /is/ being updated.
>>>>
>>>>
>>>> The drift file read 10. The actual drift was 250 (determined after the
>>>> system had settled down). The drift file never changed even after a day of
>>>> running. ntp does not seem to be rewriting the drift file. Now that is a
>>>> problem (although with the apparent Linux bug in the timing routines where is
>>>> miscalibrates the clock on bootup, the drift is NOT stable over reboots
>>>> anyway, so the existence of a drift file is irrelevant. ) However, the question is
>>>> about the bahaviour of ntp. ntp should NOT be taking 10 hours to get over a
>>>> wrong value in the drift file.[/color]
>>[color=darkred]
>>> That's easy to fix! If the drift file is not correct, remove it before
>>> starting ntpd.[/color]
>>
>> Of course. However, I have no idea it is incorrect until after ntp has
>> started up and shown me it was incorrect.
>>[color=darkred]
>>> How do you tell if it's incorrect? Since ntpd is supposed to
>>> update/rewrite the drift file every sixty minutes, a drift file more
>>> than sixty minutes old is suspect![/color]
>>
>> I think my problem was that the permissions on /etc/ntp/drift were
>> incorrect ( owned by root rather than by ntp). But that makes no
>> difference to how ntp behaves. ntp should do the "right thing" even if the
>> drift file is wrong. It should take a bit longer, but not 10 hours longer.
>> And with the current apparent bug in Linux wehre the system time is
>> miscalibrated, it would seem that the drift file on Linux is ALWAYS wrong.
>>[/color][/color]
[color=blue]
>Do not blame ntpd for the consequences of your errors! If ntpd is
>configured correctly and operated correctly, it behaves quite well.[/color]

???? I was not blaming ntp for my error. I was blaming ntp for its reaction
to "my error" . And note a bad drift file is now the standard for
Linux. For example between two reboots, my drift rate went from 180PPM to
250PPM. No drift file would have fixed that.

NTP handles drift errors badly. But that is not the question I asked.
So far NOONE has answered the
question-- why if my poll internal is 16 sec is the time scale for the
error correction 1 hour?


[color=blue]
>And if you can write an ntpd equivalent that works better, I'm sure that
>most of us would be interested and, perhaps, even grateful![/color]

already written, IF you do not want refclocks and you are running Linux. It
is called chrony.

David Woolley wrote:[color=blue]
> Richard B. Gilbert wrote:
>[color=green]
>>
>> The internet of today is similar. It may be a little better but I
>> wouldn't count on it!
>>[/color]
> For one thing, the components from which it is made are 1,000 times
> faster and have 1,000 times the memory.
>
> (There are other differences, like the de-skilling of sytem management.)[/color]

And there is at least 1,000 times the traffic load that there used to be!

I'd be more impressed if you could spell/type "system" correctly!

Unruh wrote:[color=blue]
> "Richard B. Gilbert" <rgilbert88@comcast.net> writes:
>[color=green]
>> Unruh wrote:[color=darkred]
>>> "Richard B. Gilbert" <rgilbert88@comcast.net> writes:
>>>
>>>> Unruh wrote:
>>>>> "David J Taylor" <david-taylor@blueyonder.neither-this-part.nor-this-bit.co.uk> writes:
>>>>>
>>>>>> Hal Murray wrote:
>>>>>>>>> Try switching it off, changing the value int he drift file by say
>>>>>>>>> 50PPM and
>>>>>>>>> then switching it on again, and see how long it takes to recover
>>>>>>>>> from that.
>>>>>>>> Why would I do that? The drift values rarely change by more than
>>>>>>>> five, certainly not by 50. If you are seeing a change of 50, then
>>>>>>>> perhaps that it part of your problem?
>>>>>>> A big step like that makes it easy to see how the system responds.
>>>>>>> At least if it's a linear system. :)
>>>>>> Yes, I appreciate that, Hal, but it doesn't emulate the situation here
>>>>>> very well, which I understood to be slow convergence after a routine
>>>>>> start. It sounds as if the OP may have an incorrect drift file - it's
>>>>>> worth checking that it /is/ being updated.
>>>>>
>>>>> The drift file read 10. The actual drift was 250 (determined after the
>>>>> system had settled down). The drift file never changed even after a day of
>>>>> running. ntp does not seem to be rewriting the drift file. Now that is a
>>>>> problem (although with the apparent Linux bug in the timing routines where is
>>>>> miscalibrates the clock on bootup, the drift is NOT stable over reboots
>>>>> anyway, so the existence of a drift file is irrelevant. ) However, the question is
>>>>> about the bahaviour of ntp. ntp should NOT be taking 10 hours to get over a
>>>>> wrong value in the drift file.
>>>> That's easy to fix! If the drift file is not correct, remove it before
>>>> starting ntpd.
>>> Of course. However, I have no idea it is incorrect until after ntp has
>>> started up and shown me it was incorrect.
>>>
>>>> How do you tell if it's incorrect? Since ntpd is supposed to
>>>> update/rewrite the drift file every sixty minutes, a drift file more
>>>> than sixty minutes old is suspect!
>>> I think my problem was that the permissions on /etc/ntp/drift were
>>> incorrect ( owned by root rather than by ntp). But that makes no
>>> difference to how ntp behaves. ntp should do the "right thing" even if the
>>> drift file is wrong. It should take a bit longer, but not 10 hours longer.
>>> And with the current apparent bug in Linux wehre the system time is
>>> miscalibrated, it would seem that the drift file on Linux is ALWAYS wrong.
>>>[/color][/color]
>[color=green]
>> Do not blame ntpd for the consequences of your errors! If ntpd is
>> configured correctly and operated correctly, it behaves quite well.[/color]
>
> ???? I was not blaming ntp for my error. I was blaming ntp for its reaction
> to "my error" . And note a bad drift file is now the standard for
> Linux. For example between two reboots, my drift rate went from 180PPM to
> 250PPM. No drift file would have fixed that.
>
> NTP handles drift errors badly. But that is not the question I asked.
> So far NOONE has answered the
> question-- why if my poll internal is 16 sec is the time scale for the
> error correction 1 hour?
>[/color]

How big is the error? Why is your poll interval 16 seconds? (If you
are using a refclock, I withdraw the question!) Are you setting the
correct time at startup with ntpd -g or ntpdate or sntp?

If the drift file is known to be incorrect it's best practice to remove
or delete it (depending on which operation your O/S supports) in which
case nptd makes no assumptions about the drift and attempts to measure
it. A drift file that was last modified more than 60 minutes ago should
be assumed to be incorrect.

<snip>

One very good way to avoid startup problems is leave it running, which I do!

On Sat, 01 Nov 2008 03:17:15 +0000, Unruh wrote:
[color=blue]
> Just another data point on the behaviour of ntp. My ntp went down ( due
> to something removing the ntp user from the password file).[/color]

Something? FULL STOP! I'd consider the machine to be compromised and
in need of a rebuild from scratch.

Root Kits are known to diddle with the machine's clocks/timers in order
to steal machine cycles undetected. In order to make the root kit work,
the intruder most likely had to disable ntp, because it also diddles with
the machines clocks/timers and interferes with the functioning of the
root kit.

[snip]
[color=blue]
> But never mind my concern about the markovian system feedback system ntp
> uses. That argument I am sure everyone is tired of. What concerns me is
> the long (1hr) time constant of the feedback loop, about 200 times
> longer than the poll interval. Ie, it does not seem to me that ntp is
> fulfilling its design criteria.[/color]

If you've had a root kit installed on your machine before starting ntp,
ntp will no longer be seeing the time in your machine, it will most likely
be seeing whatever fake time the root kit decides to show it at any given
instant, so it really should not be a surprise that ntp might take a bit
longer to figure out what it is converging to. :) The fact that ntp
actually does converge under these conditions only attests to the
robustness of its design.

If you are serious about having accurate and reliable time, you ought
to be running on dedicated hardware with an embedded Operating System
that can not be compromised. There is a good How-To Build A Stratum One
Time Server published here: [url]http://www.febo.com/pages/soekris/[/url]

And, if you are going to go as far as to build a stratum one server
described above, then make sure you go all the way and plug it into a
good UPS to ensure that your time server never goes down. That way,
ntp will just do its thing and you won't have to deal with the short
comings, perceived or otherwise, of ntp at start up.
[color=blue]
>
>
> Here after 5.5 hours after startup is the ouput of ntpq -p
>
> string[root]>ntpq -p
> remote refid st t when poll reach delay offset
> jitter
> ==============================================================================
> +tick.usask.ca .GPS. 1 u 18 64 377 44.925 1.455
> 4.252 +ntp.ubc.ca 140.142.16.34 2 u 44 64 343 0.672
> 0.260 0.767 *SHM(0) .PPS. 0 l 1 16 377
> 0.000 1.136 0.023[/color]

"Richard B. Gilbert" <rgilbert88@comcast.net> writes:
[color=blue]
>Unruh wrote:[color=green]
>> "Richard B. Gilbert" <rgilbert88@comcast.net> writes:
>>[color=darkred]
>>> Unruh wrote:
>>>> "Richard B. Gilbert" <rgilbert88@comcast.net> writes:
>>>>
>>>>> Unruh wrote:
>>>>>> "David J Taylor" <david-taylor@blueyonder.neither-this-part.nor-this-bit.co.uk> writes:
>>>>>>
>>>>>>> Hal Murray wrote:
>>>>>>>>>> Try switching it off, changing the value int he drift file by say
>>>>>>>>>> 50PPM and
>>>>>>>>>> then switching it on again, and see how long it takes to recover
>>>>>>>>>> from that.
>>>>>>>>> Why would I do that? The drift values rarely change by more than
>>>>>>>>> five, certainly not by 50. If you are seeing a change of 50, then
>>>>>>>>> perhaps that it part of your problem?
>>>>>>>> A big step like that makes it easy to see how the system responds.
>>>>>>>> At least if it's a linear system. :)
>>>>>>> Yes, I appreciate that, Hal, but it doesn't emulate the situation here
>>>>>>> very well, which I understood to be slow convergence after a routine
>>>>>>> start. It sounds as if the OP may have an incorrect drift file - it's
>>>>>>> worth checking that it /is/ being updated.
>>>>>>
>>>>>> The drift file read 10. The actual drift was 250 (determined after the
>>>>>> system had settled down). The drift file never changed even after a day of
>>>>>> running. ntp does not seem to be rewriting the drift file. Now that is a
>>>>>> problem (although with the apparent Linux bug in the timing routines where is
>>>>>> miscalibrates the clock on bootup, the drift is NOT stable over reboots
>>>>>> anyway, so the existence of a drift file is irrelevant. ) However, the question is
>>>>>> about the bahaviour of ntp. ntp should NOT be taking 10 hours to get over a
>>>>>> wrong value in the drift file.
>>>>> That's easy to fix! If the drift file is not correct, remove it before
>>>>> starting ntpd.
>>>> Of course. However, I have no idea it is incorrect until after ntp has
>>>> started up and shown me it was incorrect.
>>>>
>>>>> How do you tell if it's incorrect? Since ntpd is supposed to
>>>>> update/rewrite the drift file every sixty minutes, a drift file more
>>>>> than sixty minutes old is suspect!
>>>> I think my problem was that the permissions on /etc/ntp/drift were
>>>> incorrect ( owned by root rather than by ntp). But that makes no
>>>> difference to how ntp behaves. ntp should do the "right thing" even if the
>>>> drift file is wrong. It should take a bit longer, but not 10 hours longer.
>>>> And with the current apparent bug in Linux wehre the system time is
>>>> miscalibrated, it would seem that the drift file on Linux is ALWAYS wrong.
>>>>[/color]
>>[color=darkred]
>>> Do not blame ntpd for the consequences of your errors! If ntpd is
>>> configured correctly and operated correctly, it behaves quite well.[/color]
>>
>> ???? I was not blaming ntp for my error. I was blaming ntp for its reaction
>> to "my error" . And note a bad drift file is now the standard for
>> Linux. For example between two reboots, my drift rate went from 180PPM to
>> 250PPM. No drift file would have fixed that.
>>
>> NTP handles drift errors badly. But that is not the question I asked.
>> So far NOONE has answered the
>> question-- why if my poll internal is 16 sec is the time scale for the
>> error correction 1 hour?
>>[/color][/color]
[color=blue]
>How big is the error? Why is your poll interval 16 seconds? (If you[/color]

It IS a refclock-- GPS PPS via refclock_shm.
[color=blue]
>are using a refclock, I withdraw the question!) Are you setting the
>correct time at startup with ntpd -g or ntpdate or sntp?[/color]

ntpd -g I am using the shm driver. ntp first uses other servers to set the
time, and then starts the shm pps. The drift is way off, it seems because
of the bug in the linux time driver ( the drift rate changes by 50PPM after
a reboot).

[color=blue]
>If the drift file is known to be incorrect it's best practice to remove
>or delete it (depending on which operation your O/S supports) in which
>case nptd makes no assumptions about the drift and attempts to measure
>it. A drift file that was last modified more than 60 minutes ago should
>be assumed to be incorrect.[/color]

I should NOT have to check up on the drift file to see if it is correct.
That is the computer's job. ntp should NOT take 10 hours to correct a wrong
drift file. With a 16 sec poll, ntp should NOT have a 1 hour time constant.



[color=blue]
><snip>[/color]
[color=blue]
>One very good way to avoid startup problems is leave it running, which I do![/color]

Speechless <noone@nowhere.com> writes:
[color=blue]
>On Sat, 01 Nov 2008 03:17:15 +0000, Unruh wrote:[/color]
[color=blue][color=green]
>> Just another data point on the behaviour of ntp. My ntp went down ( due
>> to something removing the ntp user from the password file).[/color][/color]
[color=blue]
>Something? FULL STOP! I'd consider the machine to be compromised and
>in need of a rebuild from scratch.[/color]

Nope. It was probably the fact that on Mandriva ntp is given the uid of
498, while my user uids start at 200. Thus when my script changes the
users, from a master file, it wiped out the ntp user from the one machine
on which ntp was running. It is stupid that ntp was assigned such a high
uid.


[color=blue]
>Root Kits are known to diddle with the machine's clocks/timers in order
>to steal machine cycles undetected. In order to make the root kit work,
>the intruder most likely had to disable ntp, because it also diddles with
>the machines clocks/timers and interferes with the functioning of the
>root kit.[/color]
[color=blue]
>[snip][/color]
[color=blue][color=green]
>> But never mind my concern about the markovian system feedback system ntp
>> uses. That argument I am sure everyone is tired of. What concerns me is
>> the long (1hr) time constant of the feedback loop, about 200 times
>> longer than the poll interval. Ie, it does not seem to me that ntp is
>> fulfilling its design criteria.[/color][/color]
[color=blue]
>If you've had a root kit installed on your machine before starting ntp,
>ntp will no longer be seeing the time in your machine, it will most likely
>be seeing whatever fake time the root kit decides to show it at any given
>instant, so it really should not be a surprise that ntp might take a bit
>longer to figure out what it is converging to. :) The fact that ntp
>actually does converge under these conditions only attests to the
>robustness of its design.[/color]

You are riding off into the desert on your false assumptions.
[color=blue]
>If you are serious about having accurate and reliable time, you ought
>to be running on dedicated hardware with an embedded Operating System
>that can not be compromised. There is a good How-To Build A Stratum One
>Time Server published here: [url]http://www.febo.com/pages/soekris/[/url][/color]

All systems can be comprimized. But that is irrelevant. An ordinary run of
the mill computer with $60 worth of parts can nowadays be a good stratum 1
server. And even your dedicated machine can go down-- disk error, hardware
wearout, long power failure, .....

[color=blue]
>And, if you are going to go as far as to build a stratum one server
>described above, then make sure you go all the way and plug it into a
>good UPS to ensure that your time server never goes down. That way,
>ntp will just do its thing and you won't have to deal with the short
>comings, perceived or otherwise, of ntp at start up.[/color]

"That's not a bug-- that is just something to avoid asking the program to
do."

But still no answer-- why is the time constant 1 hour when the poll
interval is 16sec?
Note that this ALSO affects the ability of the system to respond to temp
changes.


[color=blue][color=green]
>>
>>
>> Here after 5.5 hours after startup is the ouput of ntpq -p
>>
>> string[root]>ntpq -p
>> remote refid st t when poll reach delay offset
>> jitter
>> ==============================================================================
>> +tick.usask.ca .GPS. 1 u 18 64 377 44.925 1.455
>> 4.252 +ntp.ubc.ca 140.142.16.34 2 u 44 64 343 0.672
>> 0.260 0.767 *SHM(0) .PPS. 0 l 1 16 377
>> 0.000 1.136 0.023[/color][/color]

Unruh wrote:[color=blue]
> "Richard B. Gilbert" <rgilbert88@comcast.net> writes:
>[color=green]
>> Unruh wrote:[color=darkred]
>>> "Richard B. Gilbert" <rgilbert88@comcast.net> writes:
>>>
>>>> Unruh wrote:
>>>>> "Richard B. Gilbert" <rgilbert88@comcast.net> writes:
>>>>>
>>>>>> Unruh wrote:
>>>>>>> "David J Taylor" <david-taylor@blueyonder.neither-this-part.nor-this-bit.co.uk> writes:
>>>>>>>
>>>>>>>> Hal Murray wrote:
>>>>>>>>>>> Try switching it off, changing the value int he drift file by say
>>>>>>>>>>> 50PPM and
>>>>>>>>>>> then switching it on again, and see how long it takes to recover
>>>>>>>>>>> from that.
>>>>>>>>>> Why would I do that? The drift values rarely change by more than
>>>>>>>>>> five, certainly not by 50. If you are seeing a change of 50, then
>>>>>>>>>> perhaps that it part of your problem?
>>>>>>>>> A big step like that makes it easy to see how the system responds.
>>>>>>>>> At least if it's a linear system. :)
>>>>>>>> Yes, I appreciate that, Hal, but it doesn't emulate the situation here
>>>>>>>> very well, which I understood to be slow convergence after a routine
>>>>>>>> start. It sounds as if the OP may have an incorrect drift file - it's
>>>>>>>> worth checking that it /is/ being updated.
>>>>>>> The drift file read 10. The actual drift was 250 (determined after the
>>>>>>> system had settled down). The drift file never changed even after a day of
>>>>>>> running. ntp does not seem to be rewriting the drift file. Now that is a
>>>>>>> problem (although with the apparent Linux bug in the timing routines where is
>>>>>>> miscalibrates the clock on bootup, the drift is NOT stable over reboots
>>>>>>> anyway, so the existence of a drift file is irrelevant. ) However, the question is
>>>>>>> about the bahaviour of ntp. ntp should NOT be taking 10 hours to get over a
>>>>>>> wrong value in the drift file.
>>>>>> That's easy to fix! If the drift file is not correct, remove it before
>>>>>> starting ntpd.
>>>>> Of course. However, I have no idea it is incorrect until after ntp has
>>>>> started up and shown me it was incorrect.
>>>>>
>>>>>> How do you tell if it's incorrect? Since ntpd is supposed to
>>>>>> update/rewrite the drift file every sixty minutes, a drift file more
>>>>>> than sixty minutes old is suspect!
>>>>> I think my problem was that the permissions on /etc/ntp/drift were
>>>>> incorrect ( owned by root rather than by ntp). But that makes no
>>>>> difference to how ntp behaves. ntp should do the "right thing" even if the
>>>>> drift file is wrong. It should take a bit longer, but not 10 hours longer.
>>>>> And with the current apparent bug in Linux wehre the system time is
>>>>> miscalibrated, it would seem that the drift file on Linux is ALWAYS wrong.
>>>>>
>>>> Do not blame ntpd for the consequences of your errors! If ntpd is
>>>> configured correctly and operated correctly, it behaves quite well.
>>> ???? I was not blaming ntp for my error. I was blaming ntp for its reaction
>>> to "my error" . And note a bad drift file is now the standard for
>>> Linux. For example between two reboots, my drift rate went from 180PPM to
>>> 250PPM. No drift file would have fixed that.
>>>
>>> NTP handles drift errors badly. But that is not the question I asked.
>>> So far NOONE has answered the
>>> question-- why if my poll internal is 16 sec is the time scale for the
>>> error correction 1 hour?
>>>[/color][/color]
>[color=green]
>> How big is the error? Why is your poll interval 16 seconds? (If you[/color]
>
> It IS a refclock-- GPS PPS via refclock_shm.
>[color=green]
>> are using a refclock, I withdraw the question!) Are you setting the
>> correct time at startup with ntpd -g or ntpdate or sntp?[/color]
>
> ntpd -g I am using the shm driver. ntp first uses other servers to set the
> time, and then starts the shm pps. The drift is way off, it seems because
> of the bug in the linux time driver ( the drift rate changes by 50PPM after
> a reboot).
>
>[color=green]
>> If the drift file is known to be incorrect it's best practice to remove
>> or delete it (depending on which operation your O/S supports) in which
>> case nptd makes no assumptions about the drift and attempts to measure
>> it. A drift file that was last modified more than 60 minutes ago should
>> be assumed to be incorrect.[/color]
>
> I should NOT have to check up on the drift file to see if it is correct.
> That is the computer's job. ntp should NOT take 10 hours to correct a wrong
> drift file. With a 16 sec poll, ntp should NOT have a 1 hour time constant.[/color]

Your opinion! The designers/developers evidently disagree.

On Sun, Nov 2, 2008 at 3:27 PM, David Woolley
<david@ex.djwhome.demon.co.uk.invalid> wrote:[color=blue]
> Richard B. Gilbert wrote:[color=green]
>> The internet of today is similar. It may be a little better but I
>> wouldn't count on it![/color]
> For one thing, the components from which it is made are 1,000 times
> faster and have 1,000 times the memory.
> (There are other differences, like the de-skilling of sytem management.)[/color]

More significant is that the "backbone" of the Internet is no longer
centrally controlled by an orginzation like DARPA. So one of the major
design assumptions of NTP - symmetric routing of packets - is no
longer valid. Asymmetric routes are now the rule rather than the
exception (unless all traffic is local inside the same ISP).

--
RPM

Richard B. Gilbert wrote:
[color=blue]
>
> Your opinion! The designers/developers evidently disagree.[/color]

Designer, singular, as far as these issues are concerned.

At least two new people have disagreed with the designer, recently, on
the newsgroup, and decided that NTP is unsuitable for their application
because of its poor startup behaviour.

David Woolley wrote:[color=blue]
> Richard B. Gilbert wrote:
>[color=green]
>>
>> Your opinion! The designers/developers evidently disagree.[/color]
>
> Designer, singular, as far as these issues are concerned.
>
> At least two new people have disagreed with the designer, recently, on
> the newsgroup, and decided that NTP is unsuitable for their application
> because of its poor startup behaviour.[/color]

Ntpd was not intended to be bounced up and down like a basketball and I
doubt very much that startup speed was a design consideration. As my
systems tend to run for months between reboots, the startup behavior of
ntpd is not terribly important to me.

If, for some reason, you must reboot your systems daily and you need an
accurate clock within seconds of booting, you will just have to find a
tool better suited to the job than ntpd.

"Richard B. Gilbert" <rgilbert88@comcast.net> writes:
[color=blue]
>David Woolley wrote:[color=green]
>> Richard B. Gilbert wrote:
>>[color=darkred]
>>>
>>> Your opinion! The designers/developers evidently disagree.[/color]
>>
>> Designer, singular, as far as these issues are concerned.
>>
>> At least two new people have disagreed with the designer, recently, on
>> the newsgroup, and decided that NTP is unsuitable for their application
>> because of its poor startup behaviour.[/color][/color]
[color=blue]
>Ntpd was not intended to be bounced up and down like a basketball and I
>doubt very much that startup speed was a design consideration. As my
>systems tend to run for months between reboots, the startup behavior of
>ntpd is not terribly important to me.[/color]
[color=blue]
>If, for some reason, you must reboot your systems daily and you need an
>accurate clock within seconds of booting, you will just have to find a
>tool better suited to the job than ntpd.[/color]

Of course that is an option. However surely another option is to try to get
ntp to start up faster-- ntp is not a force of nature, which you either
accept or reject ( like gravity say) but is a piece of software written by
people which can be changed, and whose design goals can be influenced. IF
the startup behaviour of ntp were crucial to its successful operation, then
you are right, we would simply have to accept its behaviour. But I at least
do not think it is critical to its successful operation.
Many features have been added to ntp over the years, and they occured
because users requested certain features, or the designers decided they
would be a good idea.

On Mon, 03 Nov 2008 06:25:15 +0000, Unruh wrote:
[color=blue]
> Speechless <noone@nowhere.com> writes:
>[color=green]
>>On Sat, 01 Nov 2008 03:17:15 +0000, Unruh wrote:[/color]
>[color=green][color=darkred]
>>> Just another data point on the behaviour of ntp. My ntp went down (
>>> due to something removing the ntp user from the password file).[/color][/color]
>[color=green]
>>Something? FULL STOP! I'd consider the machine to be compromised and
>>in need of a rebuild from scratch.[/color]
>
> Nope. It was probably the fact that on Mandriva ntp is given the uid of
> 498, while my user uids start at 200. Thus when my script changes the
> users, from a master file, it wiped out the ntp user from the one
> machine on which ntp was running. It is stupid that ntp was assigned
> such a high uid.
>[/color]

Okay, that reduces but does not eliminate the possibility.
[color=blue]
>
>[color=green]
>>Root Kits are known to diddle with the machine's clocks/timers in order
>>to steal machine cycles undetected. In order to make the root kit work,
>>the intruder most likely had to disable ntp, because it also diddles
>>with the machines clocks/timers and interferes with the functioning of
>>the root kit.[/color]
>[color=green]
>>[snip][/color]
>[color=green][color=darkred]
>>> But never mind my concern about the markovian system feedback system
>>> ntp uses. That argument I am sure everyone is tired of. What concerns
>>> me is the long (1hr) time constant of the feedback loop, about 200
>>> times longer than the poll interval. Ie, it does not seem to me that
>>> ntp is fulfilling its design criteria.[/color][/color]
>[color=green]
>>If you've had a root kit installed on your machine before starting ntp,
>>ntp will no longer be seeing the time in your machine, it will most
>>likely be seeing whatever fake time the root kit decides to show it at
>>any given instant, so it really should not be a surprise that ntp might
>>take a bit longer to figure out what it is converging to. :) The fact
>>that ntp actually does converge under these conditions only attests to
>>the robustness of its design.[/color]
>
> You are riding off into the desert on your false assumptions.[/color]

The only assumptions I've made so far are that:
a) you possess a modicum of intelligence and
b) you are experiencing a genuine problem not encountered by anyone else

Thank you for setting me straight about my assumptions...you'll be added
to my kill file in a moment...
[color=blue]
>[color=green]
>>If you are serious about having accurate and reliable time, you ought to
>>be running on dedicated hardware with an embedded Operating System that
>>can not be compromised. There is a good How-To Build A Stratum One Time
>>Server published here: [url]http://www.febo.com/pages/soekris/[/url][/color]
>
> All systems can be comprimized. But that is irrelevant. An ordinary run
> of the mill computer with $60 worth of parts can nowadays be a good
> stratum 1 server. And even your dedicated machine can go down-- disk
> error, hardware wearout, long power failure, .....
>[/color]

You obviously have not looked at the web site above to study in depth
and learn what it really takes to build a stratum one time server that
meets the requirements of a discriminating time freak. If the $60 pile
of parts running your ntp was good enough to meet _your_ requirements,
you would not be in this news group moaning and groaning about ntp.

It should not be necessary to spell it out for you that if you are
dissatisfied with the results you are getting, then you need to DO
SOMETHING DIFFERENT to GET DIFFERENT RESULTS that might be more to your
liking.

For example, my assistant runs ntp on her budget priced general purpose
machine and she is absolutely thrilled that her machine is able to keep
time with an accuracy of "within one second" of her wrist watch. She is
not in this news group complaining about ntp and you are. One of the
things you might try to do DIFFERENT, is to use a better quality wrist
watch when checking the time on your machine.

Your other opportunity, to do something DIFFERENT, might be to put the
pile of crap running your ntp into the thrash bin and to acquire
timekeeping hardware with a proven track record for meeting the
requirements of a discriminating time freak and, to run a DIFFERENT
Operating System on said hardware that has a proven track record in
demanding timekeeping applications. For the discriminating time freak,
Linux is not the Operating System to use. Yes, some do use Linux,
but not the discriminating connoisseurs of time.

There are a number of resources on the web, in addition to the one I've
referenced, that provide a step-by-step how-to for building a stratum
one time server capable of tracking true UTC within a bandwidth
of less than 150 nanoseconds, all within the budget of a typical cash
strapped university researcher; however, building such a project entails
allocating and expending a bit more in effort and resources than just
$60 for a pile of parts slapped together by someone who is unable to
demonstrate that there is some congruence between academic credentials
and innate intelligence he might possess.
[color=blue]
>[color=green]
>>And, if you are going to go as far as to build a stratum one server
>>described above, then make sure you go all the way and plug it into a
>>good UPS to ensure that your time server never goes down. That way, ntp
>>will just do its thing and you won't have to deal with the short
>>comings, perceived or otherwise, of ntp at start up.[/color]
>
> "That's not a bug-- that is just something to avoid asking the program
> to do."
>
> But still no answer-- why is the time constant 1 hour when the poll
> interval is 16sec?[/color]

In your case, the answer most likely lies in the realm of the ephemeral...
perhaps better known to you as the Heisenberg uncertainty principle.

[color=blue]
> Note that this ALSO affects the ability of the system to respond to temp
> changes.
>[/color]

Time freaks keep their time servers at a stable temperature. An
inexpensive way to stabilize the temperature is to place the time server
into a cavity of a large thermal mass. What you use for a thermal mass
is left to your creative imagination. One example comes in the form of
a 250kg cast iron truck engine block, that can be sourced from a local
auto wrecking shop for little more than the cost of delivery. Add some
fiber glass bat insulation at strategic places to limit air circulation
and you are good to go.

Speechless wrote:
[color=blue]
> b) you are experiencing a genuine problem not encountered by anyone else[/color]

Lots of people report poor startup behaviour. Two have abandoned ntpd
on this newsgroup in the last few weeks because of this.
[color=blue]
>
> It should not be necessary to spell it out for you that if you are
> dissatisfied with the results you are getting, then you need to DO
> SOMETHING DIFFERENT to GET DIFFERENT RESULTS that might be more to your
> liking.[/color]

It is perfectly valid to try and get the product improved, rather than
simply encouraging everyone to vote with their feet.[color=blue]
>
> For example, my assistant runs ntp on her budget priced general purpose
> machine and she is absolutely thrilled that her machine is able to keep
> time with an accuracy of "within one second" of her wrist watch. She is[/color]

ntpd considers accuracies of worse than about 128ms to be broken, so
anyone who is only interested in 1 second accuracy is either getting a
lot better than they expect, or doesn't really need ntpd.

If you are only interested in 1 second accuracy, you don't get
convergence issues, because ntpd will go into error recovery long before
you reach a second. If you are out by that much at startup, you will be
rapidly brought into that range. If you go out during operation, there
will be an upwards of 15 minutes delay, but that will start at 128ms,
and the time will be abruptly corrected, normally long before it reaches
1 second. None of that really depends on the algorithms that make ntpd
what it is.
[color=blue]
> not in this news group complaining about ntp and you are. One of the
> things you might try to do DIFFERENT, is to use a better quality wrist
> watch when checking the time on your machine.
>[/color]
[color=blue]
>[/color]

David Woolley schrieb:[color=blue]
> Speechless wrote:
>
>[color=green]
>> b) you are experiencing a genuine problem not encountered by anyone else
>>[/color]
>
> Lots of people report poor startup behaviour. Two have abandoned ntpd
> on this newsgroup in the last few weeks because of this.
>
>[color=green]
>> It should not be necessary to spell it out for you that if you are
>> dissatisfied with the results you are getting, then you need to DO
>> SOMETHING DIFFERENT to GET DIFFERENT RESULTS that might be more to your
>> liking.
>>[/color]
>
> It is perfectly valid to try and get the product improved, rather than
> simply encouraging everyone to vote with their feet.
>[color=green]
>> For example, my assistant runs ntp on her budget priced general purpose
>> machine and she is absolutely thrilled that her machine is able to keep
>> time with an accuracy of "within one second" of her wrist watch. She is
>>[/color]
>
> ntpd considers accuracies of worse than about 128ms to be broken, so
> anyone who is only interested in 1 second accuracy is either getting a
> lot better than they expect, or doesn't really need ntpd.
>
> If you are only interested in 1 second accuracy, you don't get
> convergence issues, because ntpd will go into error recovery long before
> you reach a second. If you are out by that much at startup, you will be
> rapidly brought into that range. If you go out during operation, there
> will be an upwards of 15 minutes delay, but that will start at 128ms,
> and the time will be abruptly corrected, normally long before it reaches
> 1 second. None of that really depends on the algorithms that make ntpd
> what it is.
>
>[color=green]
>> not in this news group complaining about ntp and you are. One of the
>> things you might try to do DIFFERENT, is to use a better quality wrist
>> watch when checking the time on your machine.
>>
>>[/color][/color]
Ok, I am not currently following the list on a per mail basis (WORK, a
LOT, out of office, too..), but just caught this.. Still I want to
reply, since it became a rather big topic, as it looks..

I just recently tested my ntp-(pps-)machine in a large open hall and
found it to be unable to settle down even after a day! It looks like the
temperature changes in this open hall environment make the machine
jitter with values between -200 and 200 ms regularly and unpredictable.
Whenever I checked it, values were wildly varying inbetween those.. The
same machine was able to at least settle down after an hour or so in a
heated inside environment. I slowly come to believe that ntp simply
lacks switches for adjusting how drastically the slewing is done in case
the environmental temperature is varying drastically as well.

What really bothers me is that ntp is totally capable of telling me its
offset (and so how wrong the time /being served/ is) but not capable of
reacting to it, due to it's programmed slowness, wich as we prove in
some cases leads to the software being unusable.

That's too bad and not neccesary :( - I wish I was a programmer...

Best regards,
.../nico berndt

Nicola Berndt wrote:
[color=blue]
>
> What really bothers me is that ntp is totally capable of telling me its
> offset (and so how wrong the time /being served/ is) but not capable of[/color]

Careful. The theory behind ntpd is that, in normal operation, offset
consists only of measurement error, not of real clock error. The
problem with ntpd is that, after startup, or rapid temperature changes,
that assumption isn't valid, and offset contains a large component of
real error. Worse, it is obvious to a human observer, even without
hindsight, that that is the case.

In these situations, ntpd continues to believe that offset is noise and
not signal.

(One does have to beware of hindsight. In judging ntpd, one must only
look at what has happened before its decisions.)
[color=blue]
> reacting to it, due to it's programmed slowness, wich as we prove in
> some cases leads to the software being unusable.[/color]

I will top post this reply since there is too much garbled nonesense.

ntp is capable of disciplining the local clock on a run of the mill PC with
a $60 GPS receiver to a few microseconds. So lets take that as the
standard.

The question I have is why, given a poll interval of 16 seconds, the time
constant of the convergence of ntp is 1 hour. This is a technical
question, which you seem incapable of answering.

Could I, if I designed a thermally issolated system and a better clock
source get that long term accuracy of the system down to 1 ns? Yes. I could. It would
be stupid for the purpose ( keeping other machines on time) since the transfer of
time to any other computer via the net would degrade
that accuracy to tens of microseconds, so all of the work to get the machine down to nanosecond
accuracy would be wasted, but I could do so. But that would still be
irrelevant to the question.

ntp has a slow convergence, a slow reaction to errors. But as I thought I
understood the algorithm, that timescale was of
the order of 16 times the poll interval. Mine is 200 times
the poll interval. That indicates one of three things.
a) I do not properly understand the design of ntp and a time constant of
200 times the poll interval meets the design criteria.
b) There is some parameter in my ntp that has been misconfigured
c) There is a bug in ntp. I would think that even you should be
interested in bugs in ntp being fixed.

The only reason I post here is because of the third possiblity. No-one,
except perhaps me, is interested in the first two nor do I expect them to
be.


Speechless <noone@nowhere.com> writes:
[color=blue]
>On Mon, 03 Nov 2008 06:25:15 +0000, Unruh wrote:[/color]
[color=blue][color=green]
>> Speechless <noone@nowhere.com> writes:
>>[color=darkred]
>>>On Sat, 01 Nov 2008 03:17:15 +0000, Unruh wrote:[/color]
>>[color=darkred]
>>>> Just another data point on the behaviour of ntp. My ntp went down (
>>>> due to something removing the ntp user from the password file).[/color]
>>[color=darkred]
>>>Something? FULL STOP! I'd consider the machine to be compromised and
>>>in need of a rebuild from scratch.[/color]
>>
>> Nope. It was probably the fact that on Mandriva ntp is given the uid of
>> 498, while my user uids start at 200. Thus when my script changes the
>> users, from a master file, it wiped out the ntp user from the one
>> machine on which ntp was running. It is stupid that ntp was assigned
>> such a high uid.
>>[/color][/color]
[color=blue]
>Okay, that reduces but does not eliminate the possibility.[/color]
[color=blue][color=green]
>>
>>[color=darkred]
>>>Root Kits are known to diddle with the machine's clocks/timers in order
>>>to steal machine cycles undetected. In order to make the root kit work,
>>>the intruder most likely had to disable ntp, because it also diddles
>>>with the machines clocks/timers and interferes with the functioning of
>>>the root kit.[/color]
>>[color=darkred]
>>>[snip][/color]
>>[color=darkred]
>>>> But never mind my concern about the markovian system feedback system
>>>> ntp uses. That argument I am sure everyone is tired of. What concerns
>>>> me is the long (1hr) time constant of the feedback loop, about 200
>>>> times longer than the poll interval. Ie, it does not seem to me that
>>>> ntp is fulfilling its design criteria.[/color]
>>[color=darkred]
>>>If you've had a root kit installed on your machine before starting ntp,
>>>ntp will no longer be seeing the time in your machine, it will most
>>>likely be seeing whatever fake time the root kit decides to show it at
>>>any given instant, so it really should not be a surprise that ntp might
>>>take a bit longer to figure out what it is converging to. :) The fact
>>>that ntp actually does converge under these conditions only attests to
>>>the robustness of its design.[/color]
>>
>> You are riding off into the desert on your false assumptions.[/color][/color]
[color=blue]
>The only assumptions I've made so far are that:
>a) you possess a modicum of intelligence and
>b) you are experiencing a genuine problem not encountered by anyone else[/color]
[color=blue]
>Thank you for setting me straight about my assumptions...you'll be added
>to my kill file in a moment...[/color]
[color=blue][color=green]
>>[color=darkred]
>>>If you are serious about having accurate and reliable time, you ought to
>>>be running on dedicated hardware with an embedded Operating System that
>>>can not be compromised. There is a good How-To Build A Stratum One Time
>>>Server published here: [url]http://www.febo.com/pages/soekris/[/url][/color]
>>
>> All systems can be comprimized. But that is irrelevant. An ordinary run
>> of the mill computer with $60 worth of parts can nowadays be a good
>> stratum 1 server. And even your dedicated machine can go down-- disk
>> error, hardware wearout, long power failure, .....
>>[/color][/color]
[color=blue]
>You obviously have not looked at the web site above to study in depth
>and learn what it really takes to build a stratum one time server that
>meets the requirements of a discriminating time freak. If the $60 pile
>of parts running your ntp was good enough to meet _your_ requirements,
>you would not be in this news group moaning and groaning about ntp.[/color]
[color=blue]
>It should not be necessary to spell it out for you that if you are
>dissatisfied with the results you are getting, then you need to DO
>SOMETHING DIFFERENT to GET DIFFERENT RESULTS that might be more to your
>liking.[/color]
[color=blue]
>For example, my assistant runs ntp on her budget priced general purpose
>machine and she is absolutely thrilled that her machine is able to keep
>time with an accuracy of "within one second" of her wrist watch. She is
>not in this news group complaining about ntp and you are. One of the
>things you might try to do DIFFERENT, is to use a better quality wrist
>watch when checking the time on your machine.[/color]
[color=blue]
>Your other opportunity, to do something DIFFERENT, might be to put the
>pile of crap running your ntp into the thrash bin and to acquire
>timekeeping hardware with a proven track record for meeting the
>requirements of a discriminating time freak and, to run a DIFFERENT
>Operating System on said hardware that has a proven track record in
>demanding timekeeping applications. For the discriminating time freak,
>Linux is not the Operating System to use. Yes, some do use Linux,
>but not the discriminating connoisseurs of time.[/color]
[color=blue]
>There are a number of resources on the web, in addition to the one I've
>referenced, that provide a step-by-step how-to for building a stratum
>one time server capable of tracking true UTC within a bandwidth
>of less than 150 nanoseconds, all within the budget of a typical cash
>strapped university researcher; however, building such a project entails
>allocating and expending a bit more in effort and resources than just
>$60 for a pile of parts slapped together by someone who is unable to
>demonstrate that there is some congruence between academic credentials
>and innate intelligence he might possess.[/color]
[color=blue][color=green]
>>[color=darkred]
>>>And, if you are going to go as far as to build a stratum one server
>>>described above, then make sure you go all the way and plug it into a
>>>good UPS to ensure that your time server never goes down. That way, ntp
>>>will just do its thing and you won't have to deal with the short
>>>comings, perceived or otherwise, of ntp at start up.[/color]
>>
>> "That's not a bug-- that is just something to avoid asking the program
>> to do."
>>
>> But still no answer-- why is the time constant 1 hour when the poll
>> interval is 16sec?[/color][/color]
[color=blue]
>In your case, the answer most likely lies in the realm of the ephemeral...
>perhaps better known to you as the Heisenberg uncertainty principle.[/color]

[color=blue][color=green]
>> Note that this ALSO affects the ability of the system to respond to temp
>> changes.
>>[/color][/color]
[color=blue]
>Time freaks keep their time servers at a stable temperature. An
>inexpensive way to stabilize the temperature is to place the time server
>into a cavity of a large thermal mass. What you use for a thermal mass
>is left to your creative imagination. One example comes in the form of
>a 250kg cast iron truck engine block, that can be sourced from a local
>auto wrecking shop for little more than the cost of delivery. Add some
>fiber glass bat insulation at strategic places to limit air circulation
>and you are good to go.[/color]

Nicola Berndt wrote:
[][color=blue]
> I just recently tested my ntp-(pps-)machine in a large open hall and
> found it to be unable to settle down even after a day! It looks like
> the temperature changes in this open hall environment make the machine
> jitter with values between -200 and 200 ms regularly and
> unpredictable. Whenever I checked it, values were wildly varying
> inbetween those.. The same machine was able to at least settle down
> after an hour or so in a heated inside environment. I slowly come to
> believe that ntp simply lacks switches for adjusting how drastically
> the slewing is done in case the environmental temperature is varying
> drastically as well.[/color]
[][color=blue]
> Best regards,
> ../nico berndt[/color]

Nico,

From what you report, I would suggest that something is very broken. I
have a PC (FreeBSD, NTP, PPS GPS source) running in a domestic environment
where, at the moment the central heating comes on at 0600. Here is the
reported offset:

[url]http://www.satsignal.eu/mrtg/pixie_ntp.html[/url]

Switching the room heating on causes 10usec offset, not 200msec.

The other Windows PCs synced from that PC are generally within 10msec:

[url]http://www.satsignal.eu/mrtg/daily_ntp.html[/url]

Cheers,
David

Unruh wrote:[color=blue]
> Evandro Menezes <evandro@mailinator.com> writes:
> Sure. Mine is a stock Linux PC hardware, and running 4.2.4p4, not 4.2.0
>
> It is possible a bug has crept into the software,[/color]

AFAIR older versions of ntpd *did* converge quite a bit faster than the
current -stable and -dev versions. I'm afraid it's not a bug but a
"feature" the current versions do converge so incredibly slow.

Martin
--
Martin Burnicki

Meinberg Funkuhren
Bad Pyrmont
Germany

David Woolley wrote:[color=blue]
> Nicola Berndt wrote:
>[color=green]
>>
>> What really bothers me is that ntp is totally capable of telling me its
>> offset (and so how wrong the time /being served/ is) but not capable of[/color]
>
> Careful. The theory behind ntpd is that, in normal operation, offset
> consists only of measurement error, not of real clock error. The
> problem with ntpd is that, after startup, or rapid temperature changes,
> that assumption isn't valid, and offset contains a large component of
> real error. Worse, it is obvious to a human observer, even without
> hindsight, that that is the case.[/color]

Some time ago I have made some tests comparing the system time disciplined
by NTP against the time of a built-in GPS PCI card. The results showed the
offset reported by ntpd did very well match the offset measured against the
GPS card.

I think the current development of NTP goes into a direction which may cover
some special scenarions well, but matches less and less the requirements of
"normal" users.

Martin
--
Martin Burnicki

Meinberg Funkuhren
Bad Pyrmont
Germany

David J Taylor wrote:[color=blue]
> Unruh wrote:
> [][color=green]
>> An hour later, it was still 7ms off, another hour, 2.6ms and another
>> hour
>> later, still 1.2 ms off. Ie, only after about 6 hours was it within a
>> ms of
>> the correct time. Now, usually this PPS controls the time to within
>> about 2us (not ms, usec) but it is apparently going to take over 10
>> hours to get
>> there. That is of course completely rediculous.[/color]
>
> There sounds to be something wrong with your system.[/color]

As mentioned in another reply, AFAIR older versions of the NTP daemon did
converge quite a bit faster than recent versions.
[color=blue]
> As a comparison, I have a very old Pentium 133 system here running FreeBSD
> with local GPS PPS and some other Internet-based stratum 2/3 servers
> (probably NTP pool and a fixed name). I'm sure it's well within a few
> minutes for it to reach it's full accuracy (tens of microseconds). For
> interest, I've just (0645 UTC) switched it off and on, and we will be able
> to watch its recovery here (30 minute updates):
>
> [url]http://www.satsignal.eu/mrtg/pixie_ntp.html[/url]
>
> Here it is about a minute after startup:
>
> ntpq -p pixie
> remote refid st t when poll reach delay offset
> jitter
>[/color]
==============================================================================[color=blue]
> +calx.pulsewidth 193.120.10.3 2 u 62 64 1 22.272 1.700
> 0.743
> +admin.islay.bit 192.33.96.102 2 u 62 64 1 21.131 0.921
> 1.112
> +dnscache-london 128.250.33.242 2 u 62 64 1 22.845 3.299
> 0.666
> 88-96-233-89.ds .PPS. 1 u 14 128 7 63.431 0.044
> 2.789
> *utserv.mcc.ac.u 193.62.22.98 2 u 64 64 1 26.494 4.312
> 0.829
> GPS_NMEA(1) .PPS. 0 l 12 64 3 0.000 -0.137
> 1.654
>
> .. and a few minutes later ...
>
> ntpq -p pixie
> remote refid st t when poll reach delay offset
> jitter
>[/color]
==============================================================================[color=blue]
> +calx.pulsewidth 193.120.10.3 2 u 54 64 37 22.348 2.946
> 0.877
> +admin.islay.bit 192.33.96.102 2 u 53 64 37 20.496 1.862
> 1.057
> +dnscache-london 128.250.33.242 2 u 57 64 37 23.090 3.809
> 0.662
> 88-96-233-89.ds .PPS. 1 u 134 256 17 63.431 0.044
> 2.007
> +utserv.mcc.ac.u 193.62.22.98 2 u 53 64 37 25.564 5.371
> 0.868
> *GPS_NMEA(1) .PPS. 0 l 3 64 77 0.000 -0.001
> 0.803
>
> It's using the out-of-the-box NTP code, and probably a rather old version
> of NTP.
>
> version="ntpd 4.2.0-a Sun May 8 06:01:21 UTC 2005 (1)"[/color]

Now the question is how a recent version of ntpd would converge on this
machine.

Martin
--
Martin Burnicki

Meinberg Funkuhren
Bad Pyrmont
Germany