Reference clock all messed up? - NTP

This is a discussion on Reference clock all messed up? - NTP ; Howdy all, I've got a problem that has been driving me nuts. Hopefully, somebody can give me a clue. I've been requested to configure an NTP server (192.168.2.1) for the local subnets that I'm responsible for. Unfortunately, firewall rules prevent ...

+ Reply to Thread
Results 1 to 9 of 9

Thread: Reference clock all messed up?

  1. Reference clock all messed up?

    Howdy all,

    I've got a problem that has been driving me nuts. Hopefully, somebody can
    give me a clue.

    I've been requested to configure an NTP server (192.168.2.1) for the local
    subnets that I'm responsible for. Unfortunately, firewall rules prevent me
    from accessing 123/UDP on the Internet, however there is a machine that
    appears to be running NTP on 192.168.1.1 (outside subnets I administer but
    internal to the company and accessible) which I would like to sync my server to.

    On 192.168.2.1 I'm running FreeBSD 5.4-STABLE with ntpd "4.2.0-a Thu Jan 19
    11:02:17 EST 2006". According to nmap, 192.168.1.1 might be FreeBSD 6.1 (86%
    sure).

    Note that we don't want all client NTP machines to access 192.168.1.1 -
    otherwise, things would have been much easier.

    --- /etc/ntp.conf on 192.168.2.1 ---
    # /etc/ntp.conf

    server 192.168.1.1

    driftfile /var/db/ntpd.drift

    # by default ignore all ntp packets
    restrict default ignore

    # allow localhost
    restrict 127.0.0.1 mask 255.255.255.255

    # accept packets from...
    restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
    restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
    restrict 192.168.4.0 mask 255.255.255.0 nomodify notrap
    restrict 192.168.5.0 mask 255.255.255.0 nomodify notrap
    --- end /etc/ntp.conf on 192.168.2.1 ---

    So I run the daemon:
    $ /usr/sbin/ntpd -p /var/run/ntpd.pid -c /etc/ntp.conf

    Using a client on one of the subnets I control, I run:
    $ sudo ntpdate 192.168.2.1
    5 Nov 18:39:36 ntpdate[21310]: no server suitable for synchronization found

    Seems odd, so I try:

    $ sudo ntpdate -d 192.168.2.1
    5 Nov 18:41:24 ntpdate[21447]: ntpdate 4.2.4p0@1.1472-o Thu Oct 4 22:22:32
    UTC 2007 (1)
    transmit(192.168.2.1)
    receive(192.168.2.1)
    transmit(192.168.2.1)
    receive(192.168.2.1)
    transmit(192.168.2.1)
    receive(192.168.2.1)
    transmit(192.168.2.1)
    receive(192.168.2.1)
    transmit(192.168.2.1)
    192.168.2.1: Server dropped: strata too high
    server 192.168.2.1, port 123
    stratum 16, precision -19, leap 11, trust 000
    refid [192.168.2.1], delay 0.02597, dispersion 0.00000
    transmitted 4, in filter 4
    reference time: 00000000.00000000 Thu, Feb 7 2036 17:28:16.000
    originate timestamp: cad947a4.ebce9575 Mon, Nov 5 2007 18:41:24.921
    transmit timestamp: cad947a4.f189ce4a Mon, Nov 5 2007 18:41:24.943
    filter delay: 0.02609 0.02597 0.02600 0.02600
    0.00000 0.00000 0.00000 0.00000
    filter offset: -0.02255 -0.02258 -0.02260 -0.02260
    0.000000 0.000000 0.000000 0.000000
    delay 0.02597, dispersion 0.00000
    offset -0.022588

    5 Nov 18:41:24 ntpdate[21447]: no server suitable for synchronization found


    I believe "leap 11" is key, possibly indicating that there is a time
    difference too great between the server and client. That certainly looks to
    be the case. Waiting almost an hour for 192.168.2.1 to sync to 192.168.1.1
    made no difference - the year stayed at 2036.

    Next up, I decided to verify that 192.168.1.1 is accurate. I don't have the
    ability to administer this box as it is outside my control, but I have no
    choice but to use it as a reference clock for 192.168.2.1.

    Again on an NTP client box:
    $ sudo ntpdate -d 192.168.1.1
    5 Nov 18:28:39 ntpdate[20392]: ntpdate 4.2.4p0@1.1472-o Thu Oct 4 22:22:32
    UTC 2007 (1)
    transmit(192.168.1.1)
    receive(192.168.1.1)
    transmit(192.168.1.1)
    receive(192.168.1.1)
    transmit(192.168.1.1)
    receive(192.168.1.1)
    transmit(192.168.1.1)
    receive(192.168.1.1)
    transmit(192.168.1.1)
    server 192.168.1.1, port 123
    stratum 4, precision -19, leap 00, trust 000
    refid [192.168.1.1], delay 0.03151, dispersion 0.00143
    transmitted 4, in filter 4
    reference time: cad94300.b36a1529 Mon, Nov 5 2007 18:21:36.700
    originate timestamp: cad944a7.538d4049 Mon, Nov 5 2007 18:28:39.326
    transmit timestamp: cad944a7.60bbc2b9 Mon, Nov 5 2007 18:28:39.377
    filter delay: 0.03151 0.03615 0.03386 0.03632
    0.00000 0.00000 0.00000 0.00000
    filter offset: -0.05936 -0.05702 -0.05825 -0.05692
    0.000000 0.000000 0.000000 0.000000
    delay 0.03151, dispersion 0.00143
    offset -0.059360

    5 Nov 18:28:39 ntpdate[20392]: Debug mode --not changing the system date
    5 Nov 18:28:39 ntpdate[20392]: adjust time server 192.168.1.1 offset
    -0.059360 sec
    abolte@sphinx:~$

    It looks okay, however the time is still around 7 minutes out. I believe
    192.168.2.1 to be more accurate.

    On a client, I run:
    $ ntpdate 192.168.1.1
    5 Nov 18:46:47 ntpdate[21910]: adjust time server 192.168.1.1 offset
    -0.014893 sec

    It made such a small adjustment to the clock... if it really needed to catch
    up some 7 minutes, I would have expected a bigger adjustment.

    After running the "ntpdate -d" command a few times against 192.168.1.1, I
    noticed that the time didn't seem stable. I wrote a basic script to put it
    to the test, with the intention of running it from 192.168.2.1.

    --- timemonitor.sh ---
    #!/usr/local/bin/bash

    # On a remote machine, run:
    ####################################
    # while [ 1 ] ; do #
    # nc -l -p 5000 -e /bin/date #
    # done #
    ####################################
    # to prove that this machine is sane. Specify the machine's IP address as a
    parameter.

    if [ -n "$1" ] ; then
    remoteIP=$1
    remoteIPTime=$(nc ${remoteIP} 5000 | tr -s ' ' | cut -d ' ' -f 4 ||
    exit 1)
    fi

    ntp_out=$(ntpdate -d isis.ca.com 2>&1 | grep time | head -n 2 | tr -s ' ' |
    cut -d ' ' -f 8 || exit 1)
    remoteTime=$(echo "$ntp_out" | sed -n -e 's/\..*//' -e '1p')
    localTime=$(echo "$ntp_out" | sed -n -e 's/\..*//' -e '2p')

    if [ -n "${remoteIP}" ] ; then
    echo "Remote time (${remoteIP}): ${remoteIPTime}"
    fi

    echo "Remote time: ${remoteTime}"
    echo "Local time: ${localTime}"

    localHour=${localTime%%:*}
    remoteHour=${remoteTime%%:*}

    localMin=${localTime%:*}
    localMin=${localMin#*:}
    remoteMin=${remoteTime%:*}
    remoteMin=${remoteMin#*:}

    localSec=${localTime##*:}
    remoteSec=${remoteTime##*:}


    echo "Difference: $(expr ${localHour} - ${remoteHour}) hours, $(expr
    ${localMin} - ${remoteMin}) minutes and $(expr ${localSec} - ${remoteSec})
    seconds."

    --- end timemonitor.sh ---

    Note that the script will also compare against another machine using netcat
    to pipe through the date command - that way when I run the script at
    5-minute intervals, I can be sure which machine is the one that is unstable.

    On a client machine (192.168.2.60), I run:
    $ while [ 1 ] ; do nc -l -p 5000 -e /bin/date; done


    Finally, on 192.168.2.1 I run:
    $ while [ 1 ] ; do echo "---" ; ./timemonitor.sh 192.168.2.60; sleep 300;
    done
    ---
    Remote time (155.35.180.60): 17:04:39
    Remote time: 16:47:35
    Local time: 17:04:39
    Difference: 1 hours, -43 minutes and 4 seconds.
    ---
    Remote time (155.35.180.60): 17:09:40
    Remote time: 17:04:39
    Local time: 17:09:39
    Difference: 0 hours, 5 minutes and 0 seconds.
    ---
    Remote time (155.35.180.60): 17:14:40
    Remote time: 17:04:39
    Local time: 17:14:40
    Difference: 0 hours, 10 minutes and 1 seconds.
    ---
    Remote time (155.35.180.60): 17:19:40
    Remote time: 17:04:39
    Local time: 17:19:40
    Difference: 0 hours, 15 minutes and 1 seconds.
    ---
    Remote time (155.35.180.60): 17:24:41
    Remote time: 17:21:46
    Local time: 17:24:41
    Difference: 0 hours, 3 minutes and -5 seconds.
    ---
    Remote time (155.35.180.60): 17:29:41
    Remote time: 17:21:46
    Local time: 17:29:41
    Difference: 0 hours, 8 minutes and -5 seconds.
    ---
    Remote time (155.35.180.60): 17:34:41
    Remote time: 17:21:46
    Local time: 17:34:41
    Difference: 0 hours, 13 minutes and -5 seconds.
    ---
    Remote time (155.35.180.60): 17:39:42
    Remote time: 17:38:50
    Local time: 17:39:41
    Difference: 0 hours, 1 minutes and -9 seconds.
    ---
    Remote time (155.35.180.60): 17:44:42
    Remote time: 17:38:50
    Local time: 17:44:42
    Difference: 0 hours, 6 minutes and -8 seconds.
    ---
    Remote time (155.35.180.60): 17:49:42
    Remote time: 17:47:24
    Local time: 17:49:42
    Difference: 0 hours, 2 minutes and 18 seconds.
    ---
    Remote time (155.35.180.60): 17:54:42
    Remote time: 17:47:24
    Local time: 17:54:42
    Difference: 0 hours, 7 minutes and 18 seconds.
    ---
    Remote time (155.35.180.60): 17:59:42
    Remote time: 17:47:24
    Local time: 17:59:43
    Difference: 0 hours, 12 minutes and 19 seconds.
    ---
    Remote time (155.35.180.60): 18:04:43
    Remote time: 18:04:31
    Local time: 18:04:43
    Difference: 0 hours, 0 minutes and 12 seconds.
    ---
    Remote time (155.35.180.60): 18:09:43
    Remote time: 18:04:31
    Local time: 18:09:43
    Difference: 0 hours, 5 minutes and 12 seconds.
    ---
    Remote time (155.35.180.60): 18:14:43
    Remote time: 18:04:31
    Local time: 18:14:44
    Difference: 0 hours, 10 minutes and 13 seconds.
    ---
    Remote time (155.35.180.60): 18:19:44
    Remote time: 18:04:31
    Local time: 18:19:44
    Difference: 0 hours, 15 minutes and 13 seconds.
    ---
    Remote time (155.35.180.60): 18:24:44
    Remote time: 18:21:36
    Local time: 18:24:44
    Difference: 0 hours, 3 minutes and 8 seconds.
    ^C
    $

    Either my understanding of the reference clock is wrong, or something crazy
    is happening on this network! It would appear to explain why the 192.168.2.1
    NTP server couldn't sync to 192.168.1.1 properly.

    Any ideas - any at all would be greatly appreciated.

    Thanks,

    Adam

  2. Re: Reference clock all messed up?

    >>> In article <472ED290.9050208@ca.com>, bolad03@ca.com (Adam Bolte) writes:

    Adam> --- /etc/ntp.conf on 192.168.2.1 --- # /etc/ntp.conf

    Adam> server 192.168.1.1

    iburst is your friend. See http://support.ntp.org/Support .

    Adam> Using a client on one of the subnets I control, I run: $ sudo ntpdate
    Adam> 192.168.2.1 5 Nov 18:39:36 ntpdate[21310]: no server suitable for
    Adam> synchronization found

    From that (or any other machine), try:

    ntpq -p 192.168.2.1

    and see what that says.

    Adam> I believe "leap 11" is key, possibly indicating that there is a time
    Adam> difference too great between the server and client.

    11 means the remote clock is not sync'd.

    H

  3. Re: Reference clock all messed up?


    >server 192.168.1.1


    ># by default ignore all ntp packets
    >restrict default ignore
    >
    ># allow localhost
    >restrict 127.0.0.1 mask 255.255.255.255
    >
    ># accept packets from...
    >restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
    >restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
    >restrict 192.168.4.0 mask 255.255.255.0 nomodify notrap
    >restrict 192.168.5.0 mask 255.255.255.0 nomodify notrap


    I don't see a restrict line that lets the answers
    from 192.168.1.1 in so the "default ignore" line
    will drop them.


    --
    These are my opinions, not necessarily my employer's. I hate spam.


  4. Re: Reference clock all messed up?

    Thanks Hal - that's the one! Everything appears working now.

    I had expected that a server line wouldn't require a restrict rule. I
    figured it was relevant only to clients.

    Thanks Harlan for your feedback also.

    -Adam

    Hal Murray wrote:
    >> server 192.168.1.1

    >
    >> # by default ignore all ntp packets
    >> restrict default ignore
    >>
    >> # allow localhost
    >> restrict 127.0.0.1 mask 255.255.255.255
    >>
    >> # accept packets from...
    >> restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
    >> restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
    >> restrict 192.168.4.0 mask 255.255.255.0 nomodify notrap
    >> restrict 192.168.5.0 mask 255.255.255.0 nomodify notrap

    >
    > I don't see a restrict line that lets the answers
    > from 192.168.1.1 in so the "default ignore" line
    > will drop them.
    >
    >


  5. Re: Reference clock all messed up?

    Adam Bolte wrote:
    > Howdy all,
    >
    > I've got a problem that has been driving me nuts. Hopefully, somebody can
    > give me a clue.
    >
    > I've been requested to configure an NTP server (192.168.2.1) for the local
    > subnets that I'm responsible for. Unfortunately, firewall rules prevent me
    > from accessing 123/UDP on the Internet, however there is a machine that
    > appears to be running NTP on 192.168.1.1 (outside subnets I administer but
    > internal to the company and accessible) which I would like to sync my server to.
    >
    > On 192.168.2.1 I'm running FreeBSD 5.4-STABLE with ntpd "4.2.0-a Thu Jan 19
    > 11:02:17 EST 2006". According to nmap, 192.168.1.1 might be FreeBSD 6.1 (86%
    > sure).
    >
    > Note that we don't want all client NTP machines to access 192.168.1.1 -
    > otherwise, things would have been much easier.
    >
    > --- /etc/ntp.conf on 192.168.2.1 ---
    > # /etc/ntp.conf
    >
    > server 192.168.1.1
    >


    Add iburst to this line for faster synchronization

    > driftfile /var/db/ntpd.drift
    >
    > # by default ignore all ntp packets
    > restrict default ignore
    >


    Why would you want to ignore all packets?

    > # allow localhost
    > restrict 127.0.0.1 mask 255.255.255.255


    If you don't have the previous line you don't need this line and the
    netmask is redundant here.

    >
    > # accept packets from...
    > restrict 192.168.2.0 mask 255.255.255.0 nomodify notrap
    > restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap
    > restrict 192.168.4.0 mask 255.255.255.0 nomodify notrap
    > restrict 192.168.5.0 mask 255.255.255.0 nomodify notrap


    I assume all of these subnets are what you want to control. Where is the
    line to allow 192.168.1.1 to send packets and modify the clock. Your
    restrict statements are what's killing you.

    > --- end /etc/ntp.conf on 192.168.2.1 ---
    >
    > So I run the daemon:
    > $ /usr/sbin/ntpd -p /var/run/ntpd.pid -c /etc/ntp.conf
    >


    Add -g to the command line to get it to initially no panic and to set
    the clock.

    > 192.168.2.1: Server dropped: strata too high
    > server 192.168.2.1, port 123
    > stratum 16, precision -19, leap 11, trust 000


    stratum 16 means that it's not synchronized and so it not allowing any
    client to get synchronization from it.

    > refid [192.168.2.1], delay 0.02597, dispersion 0.00000
    > transmitted 4, in filter 4
    > reference time: 00000000.00000000 Thu, Feb 7 2036 17:28:16.000
    > originate timestamp: cad947a4.ebce9575 Mon, Nov 5 2007 18:41:24.921
    > transmit timestamp: cad947a4.f189ce4a Mon, Nov 5 2007 18:41:24.943
    > filter delay: 0.02609 0.02597 0.02600 0.02600
    > 0.00000 0.00000 0.00000 0.00000
    > filter offset: -0.02255 -0.02258 -0.02260 -0.02260
    > 0.000000 0.000000 0.000000 0.000000
    > delay 0.02597, dispersion 0.00000
    > offset -0.022588
    >
    > 5 Nov 18:41:24 ntpdate[21447]: no server suitable for synchronization found
    >
    >
    > I believe "leap 11" is key, possibly indicating that there is a time
    > difference too great between the server and client.


    leap 11 makes no difference since the clock is not synchronized to anything.

    > Again on an NTP client box:
    > $ sudo ntpdate -d 192.168.1.1
    > 5 Nov 18:28:39 ntpdate[20392]: ntpdate 4.2.4p0@1.1472-o Thu Oct 4 22:22:32
    > stratum 4, precision -19, leap 00, trust 000


    This system is synchronized as a stratum 4 box.

    You should use ntpq to look at your servers and not ntpdate.

    Danny

  6. Re: Reference clock all messed up?

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1

    Hi Danny,

    > Add iburst to this line for faster synchronization

    Thanks, but being an PDC I really didn't want the clock to change too
    quickly. This may seem strange not already having NTP, but the network setup
    has recently changed which is what broke NTP in the first place.

    >> driftfile /var/db/ntpd.drift
    >>
    >> # by default ignore all ntp packets
    >> restrict default ignore
    >>

    >
    > Why would you want to ignore all packets?


    All but the exceptions underneath. I don't want untrusted networks messing
    with my NTP server. I don't control the firewall, so I want to do what I can
    in the NTP config. Even if I did, I would rather this in case the firewall
    ever breaks.

    >> # allow localhost
    >> restrict 127.0.0.1 mask 255.255.255.255

    >
    > If you don't have the previous line you don't need this line and the
    > netmask is redundant here.


    I'm aware, however specifying everything is preferable to me should the
    defaults change.

    > I assume all of these subnets are what you want to control. Where is the
    > line to allow 192.168.1.1 to send packets and modify the clock. Your
    > restrict statements are what's killing you.


    Thanks again, but Hal beat you to it a few days ago.

    >
    > Add -g to the command line to get it to initially no panic and to set
    > the clock.

    Again, not sure if this is safe on a PDC.

    > stratum 16 means that it's not synchronized and so it not allowing any
    > client to get synchronization from it.
    >
    > leap 11 makes no difference since the clock is not synchronized to anything.
    >
    >> Again on an NTP client box:
    >> $ sudo ntpdate -d 192.168.1.1
    >> 5 Nov 18:28:39 ntpdate[20392]: ntpdate 4.2.4p0@1.1472-o Thu Oct 4 22:22:32
    >> stratum 4, precision -19, leap 00, trust 000

    >
    > This system is synchronized as a stratum 4 box.
    >
    > You should use ntpq to look at your servers and not ntpdate.


    Thanks for the pointers.

    - -Adam
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.6 (GNU/Linux)

    iD8DBQFHN32YcLWyK3dciBYRAlW+AJ9JMDdIRXUtdUSEA2wkPo YVOy9FaQCfRkPx
    myldyO32CazdaExyaL243pI=
    =zvET
    -----END PGP SIGNATURE-----

  7. Re: Reference clock all messed up?

    >>> In article <47377D98.1050608@ca.com>, bolad03@ca.com (Adam Bolte) writes:

    Adam> Hi Danny,
    >> Add iburst to this line for faster synchronization


    Adam> Thanks, but being an PDC I really didn't want the clock to change too
    Adam> quickly. This may seem strange not already having NTP, but the network
    Adam> setup has recently changed which is what broke NTP in the first place.

    Huh? The iburst will affect how quickly your machine decides who to listen
    to for the correct time. It has nothing to do with the speed with which
    your machine's clock then syncs to the correct time.

    >> If you don't have ... you don't need this line and the
    >> netmask is redundant here.


    Adam> I'm aware, however specifying everything is preferable to me should
    Adam> the defaults change.

    Yay.

    >> Add -g to the command line to get it to initially no panic and to set
    >> the clock.


    Adam> Again, not sure if this is safe on a PDC.

    It should be safe for the initial boot.

    It *may* not be safe on a restart, especially if you have database software
    that expects time to monotonically increase.

    I recommend you figure out (as best you can) and *document* your reasons,
    and then you can create a ntp.conf file that will implement those choices.

    If you discover an overlooked case, you update the documentation and the
    config file.

    If a case becomes obsolete/irrelevant, ditto.

    H

  8. Re: Reference clock all messed up?


    > Huh? The iburst will affect how quickly your machine decides who to listen
    > to for the correct time. It has nothing to do with the speed with which
    > your machine's clock then syncs to the correct time.

    I see. Thanks for the correction.

    > Adam> Again, not sure if this is safe on a PDC.
    > It should be safe for the initial boot.
    >
    > It *may* not be safe on a restart, especially if you have database software
    > that expects time to monotonically increase.
    >
    > I recommend you figure out (as best you can) and *document* your reasons,
    > and then you can create a ntp.conf file that will implement those choices.
    >
    > If you discover an overlooked case, you update the documentation and the
    > config file.
    >
    > If a case becomes obsolete/irrelevant, ditto.

    Essentially a design document for the config files. Makes sense. Thanks Harlan.

    -Adam

  9. Re: Reference clock all messed up?

    Adam Bolte wrote:
    > Hi Danny,
    >
    >> Add iburst to this line for faster synchronization

    > Thanks, but being an PDC I really didn't want the clock to change too
    > quickly. This may seem strange not already having NTP, but the network setup
    > has recently changed which is what broke NTP in the first place.
    >


    The iburst option has nothing to do with changing quickly. It has to do
    with initial synchronization with the remote NTP server. If you are that
    worried about changes happening too quickly add the -x option to slew
    always. I don't recommend it even on a PDC.

    >>> driftfile /var/db/ntpd.drift
    >>>
    >>> # by default ignore all ntp packets
    >>> restrict default ignore
    >>>

    >> Why would you want to ignore all packets?

    >
    > All but the exceptions underneath. I don't want untrusted networks messing
    > with my NTP server. I don't control the firewall, so I want to do what I can
    > in the NTP config. Even if I did, I would rather this in case the firewall
    > ever breaks.
    >


    If the firewall breaks NTP is the least of your problems. I'd hardly be
    worrying about that.

    >> Add -g to the command line to get it to initially no panic and to set
    >> the clock.

    > Again, not sure if this is safe on a PDC.
    >


    It's perfectly safe.

    Danny

+ Reply to Thread