No libntp.so - NTP

This is a discussion on No libntp.so - NTP ; Hello NTP-World :-), we are implementing a NTP supervision for our ATC middleware. Initially we are doing it by repeated "ntpq" executions and textual evaluations of the results. We have had to notice that pipes are very unsuited for the ...

+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 20 of 31

Thread: No libntp.so

  1. No libntp.so

    Hello NTP-World :-),

    we are implementing a NTP supervision for our ATC middleware. Initially we are
    doing it by repeated "ntpq" executions and textual evaluations of the
    results. We have had to notice that pipes are very unsuited for the task, so
    we really fork it and close its stdin to make it flush. It works, but it is
    unconvincing in performance (latency).

    It appears that the whole ntpq call is relatively slow when, when we do it on
    e.g. RHEL 5.1 on modern Dual Core hardware, we get up to 25ms execution time,
    which is very long for us.

    As we would like do other important stuff in the same process, we would like
    to do that faster. So I had a look at the source and found that ntpq is using
    a libntp.a that nobody seems to package though. There is no libntp.so either,
    we had searched for those initially, but no luck.

    Our aims can be realized (and must be short term it seems) with a fork of your
    work that builds a shared library for use in our software. While that is
    workable, it is also bad thing to do in the first place.

    What we really would prefer instead, is to see is the addition of a target to
    your makefile to make a libntp.so buildable. Then, we could already use the
    original releases as from you. From there we would work with downstreams
    (Debian and RHEL) to convince them to include libntp.so in the NTP packages
    and (libntp.a and headers obviously) possible through some ntp-devel package,
    at which point we could drop building from source again.

    The question of course would be: Will you merge such Makefile patches from us
    in the first place. And do you see any reason why this should not be done
    like this. After all, it has not been done for a long time already, so
    possibly this is on purpose?

    Obviously we hope you will allow us to be good Free Software citizens and let
    us drop the fork down the road. Will you?

    Thanks in advance,
    Kay Hayen

  2. Re: No libntp.so

    kayhayen@gmx.de (Kay Hayen) writes:

    >Hello NTP-World :-),


    >we are implementing a NTP supervision for our ATC middleware. Initially we are
    >doing it by repeated "ntpq" executions and textual evaluations of the
    >results. We have had to notice that pipes are very unsuited for the task, so
    >we really fork it and close its stdin to make it flush. It works, but it is
    >unconvincing in performance (latency).


    Why not fflush it? And why would you need lack of latency. It is a report.
    It is not time critical or should not be. What you are doing sounds bizzare
    to me.


    >It appears that the whole ntpq call is relatively slow when, when we do it on
    >e.g. RHEL 5.1 on modern Dual Core hardware, we get up to 25ms execution time,
    >which is very long for us.


    What in the world are you trying to do? ntpq is NOT the way to get the
    current time from the system. Use gettimeofday (linux) to get the current
    time with sub microsecond latency. Do not use ntpq.



    >As we would like do other important stuff in the same process, we would like
    >to do that faster. So I had a look at the source and found that ntpq is using
    >a libntp.a that nobody seems to package though. There is no libntp.so either,
    >we had searched for those initially, but no luck.


    >Our aims can be realized (and must be short term it seems) with a fork of your
    >work that builds a shared library for use in our software. While that is
    >workable, it is also bad thing to do in the first place.


    >What we really would prefer instead, is to see is the addition of a target to
    >your makefile to make a libntp.so buildable. Then, we could already use the
    >original releases as from you. From there we would work with downstreams
    >(Debian and RHEL) to convince them to include libntp.so in the NTP packages
    >and (libntp.a and headers obviously) possible through some ntp-devel package,
    >at which point we could drop building from source again.


    Please tell us what you are trying to do. You are telling us your solution
    ( which sounds really bad) instead of the problem you are trying to solve,
    and then trying to convince the whole communtity to impliment your
    solution.



    >The question of course would be: Will you merge such Makefile patches from us
    >in the first place. And do you see any reason why this should not be done
    >like this. After all, it has not been done for a long time already, so
    >possibly this is on purpose?


    Does anyone see any reason why it should be done.


    >Obviously we hope you will allow us to be good Free Software citizens and let
    >us drop the fork down the road. Will you?


    You are of course free to drop it. Whether anyone will pick it up is
    another question. Certainly you have not convinced me, but that is
    irrelevant since I have absolutely no control over what goes into ntp. But
    that may or may be indicative of how well you have convinced more
    influencial people.


    >Thanks in advance,
    >Kay Hayen


  3. Re: No libntp.so

    Kay Hayen wrote:
    > Hello NTP-World :-),
    >
    > we are implementing a NTP supervision for our ATC middleware. Initially we are

    ATC as in air traffic control?

    > doing it by repeated "ntpq" executions and textual evaluations of the
    > results. We have had to notice that pipes are very unsuited for the task, so
    > we really fork it and close its stdin to make it flush. It works, but it is
    > unconvincing in performance (latency).
    >
    > It appears that the whole ntpq call is relatively slow when, when we do it on
    > e.g. RHEL 5.1 on modern Dual Core hardware, we get up to 25ms execution time,
    > which is very long for us.

    First think about what ntpq has to do during startup. Then think again ;-)

    On the piping issue:
    Thats what ptys are for. You would have ntpq as a running process that can
    be queried at any time.
    You run into nonlinebuffering issues on stdin and stdout due to both
    not being ttys.

    And you seem to work through your jobs in a serial way? Use select().

    > Obviously we hope you will allow us to be good Free Software citizens and let
    > us drop the fork down the road. Will you?


    you may want to look around to how these people solved their probs:
    http://wiki.tcl.tk/15535

    uwe



  4. Re: No libntp.so

    Kay,

    Kay Hayen wrote:
    > Hello NTP-World :-),
    >
    > we are implementing a NTP supervision for our ATC middleware. Initially we
    > are doing it by repeated "ntpq" executions and textual evaluations of the
    > results. We have had to notice that pipes are very unsuited for the task,
    > so we really fork it and close its stdin to make it flush. It works, but
    > it is unconvincing in performance (latency).
    >
    > It appears that the whole ntpq call is relatively slow when, when we do it
    > on e.g. RHEL 5.1 on modern Dual Core hardware, we get up to 25ms execution
    > time, which is very long for us.


    As mentioned in another reply it depends on what ntpq has to do during those
    25 milliseconds. Please be aware that this may include DNS lookups which
    might fail, so the the call would be take until the DNS lookup times out.

    A proper approch would be to start a thread or task which would do the ntpq
    stuff asynchrounously and report the results to your main application when
    the reply has become available.

    > As we would like do other important stuff in the same process, we would
    > like to do that faster. So I had a look at the source and found that ntpq
    > is using a libntp.a that nobody seems to package though. There is no
    > libntp.so either, we had searched for those initially, but no luck.


    Libntp just contains a couple of functions which are shared between several
    programs of the NTP package, e.g. ntpd, ntpq and others. It is used during
    the build process of the package in order to save compiule time.

    Even if you would be made available als shared object file it would not
    provide the full ntpq functionality.

    > Our aims can be realized (and must be short term it seems) with a fork of
    > your work that builds a shared library for use in our software. While that
    > is workable, it is also bad thing to do in the first place.
    >
    > What we really would prefer instead, is to see is the addition of a target
    > to your makefile to make a libntp.so buildable. Then, we could already use
    > the original releases as from you. From there we would work with
    > downstreams (Debian and RHEL) to convince them to include libntp.so in the
    > NTP packages and (libntp.a and headers obviously) possible through some
    > ntp-devel package, at which point we could drop building from source
    > again.


    IMHO it would not make much sense to provide the existing libntp as shared
    object library. There is some ongoing work which will encapsulate the ntpq
    functionality in a shared object library, so it would be easier to use from
    an own application.

    However, those calls may also block, so the basic requirement to call those
    functions from an own tread/task would be unchanged.

    > The question of course would be: Will you merge such Makefile patches from
    > us in the first place. And do you see any reason why this should not be
    > done like this. After all, it has not been done for a long time already,
    > so possibly this is on purpose?
    >
    > Obviously we hope you will allow us to be good Free Software citizens and
    > let us drop the fork down the road. Will you?


    The question is not basically whether a patch is accepted. In the first
    place the question is whether a patch makes sense at all. As already
    mentioned above I think it does not much sense just to provide libntp
    as .so file.

    Martin
    --
    Martin Burnicki

    Meinberg Funkuhren
    Bad Pyrmont
    Germany

  5. Re: No libntp.so

    In article <200808260659.13498.kayhayen@gmx.de>,
    kayhayen@gmx.de (Kay Hayen) wrote:

    > Hello NTP-World :-),
    >
    > we are implementing a NTP supervision for our ATC middleware. Initially we are
    > doing it by repeated "ntpq" executions and textual evaluations of the
    > results. We have had to notice that pipes are very unsuited for the task, so
    > we really fork it and close its stdin to make it flush. It works, but it is
    > unconvincing in performance (latency).


    What exactly is "NTP supervision"? What problem does it solve?

    Joe Gwinn

  6. Re: No libntp.so

    Hello Uwe,

    thank you for your reply. :-)

    Am Dienstag, 26. August 2008 22:29:03 schrieb Uwe Klein:
    > Kay Hayen wrote:
    > > Hello NTP-World :-),
    > >
    > > we are implementing a NTP supervision for our ATC middleware. Initially
    > > we are

    >
    > ATC as in air traffic control?


    Yes that's right. I have this understanding that NTP was invented in the
    field, wasn't it?

    > > doing it by repeated "ntpq" executions and textual evaluations of the
    > > results. We have had to notice that pipes are very unsuited for the task,
    > > so we really fork it and close its stdin to make it flush. It works, but
    > > it is unconvincing in performance (latency).
    > >
    > > It appears that the whole ntpq call is relatively slow when, when we do
    > > it on e.g. RHEL 5.1 on modern Dual Core hardware, we get up to 25ms
    > > execution time, which is very long for us.

    >
    > First think about what ntpq has to do during startup. Then think again ;-)


    Starting ntpq does to seem to do nothing at all. When we do the peers command,
    I saw it can be slow without -n option (checking with tshark), but we are
    using that.

    I don't think we want to use ntpq at all though.

    > On the piping issue:
    > Thats what ptys are for. You would have ntpq as a running process that can
    > be queried at any time.
    > You run into nonlinebuffering issues on stdin and stdout due to both
    > not being ttys.


    Like I said, we know, pipes are not suited at all for the task. I must admit,
    I was not aware of the issue before, I was under impression, we could simply
    set stdin/stdout to no buffering, but it appears for pipes that is not an
    option, and thinking about it, pipes really need to buffer.

    We were considering to use an expect wrapper (python-expect) to channel the
    data to us, and it would certainly improve the results. But I see no reason
    to use that workaround at all. Why should we make ntpq draw ASCII art only to
    parse it and endure the changes between your releases?

    What we really want to do is to use libntp to send out some queries to the
    ntpd servers we monitor (10 of them) and async wait for replies to process
    them immediately and with minimal latency.

    > And you seem to work through your jobs in a serial way? Use select().


    Well, yes of course. Short of using expect, we ran ntpq each time anew and
    fetched its response immediately. That was good enough for a prototype that
    re-used parsing code that we already have for ntpq output, but now we would
    like to do better.

    I don't think we would have done ntpq parsing had it not been that this code
    existed for historic reasons.

    > > Obviously we hope you will allow us to be good Free Software citizens and
    > > let us drop the fork down the road. Will you?

    >
    > you may want to look around to how these people solved their probs:
    > http://wiki.tcl.tk/15535


    That part of EUROCONTROL (and them in general) is well known to me. As
    libntp.a seems to offer all it takes, this question is more about how to get
    access to it in sane way.

    Best regards,
    Kay Hayen

  7. Re: No libntp.so

    Kay Hayen wrote:
    > Hello Uwe,
    >
    > thank you for your reply. :-)


    > Yes that's right. I have this understanding that NTP was invented in the
    > field, wasn't it?

    David Mills "invented" ntp in 1985 for authoritative time distribution
    and syncing of machines. Notice "time" and "authoritative".

    >>On the piping issue:
    >>Thats what ptys are for. You would have ntpq as a running process that can
    >>be queried at any time.
    >>You run into nonlinebuffering issues on stdin and stdout due to both
    >>not being ttys.

    >
    >
    > Like I said, we know, pipes are not suited at all for the task. I must admit,
    > I was not aware of the issue before, I was under impression, we could simply
    > set stdin/stdout to no buffering, but it appears for pipes that is not an
    > option, and thinking about it, pipes really need to buffer.

    MY guess is you don't know enough about unix(like) systems to build any
    meaningfull software with the stringent requirements linked to your
    applications environment.
    You may want to read up on pipes, ptys and associated buffering.

    >
    > We were considering to use an expect wrapper (python-expect) to channel the
    > data to us, and it would certainly improve the results. But I see no reason
    > to use that workaround at all. Why should we make ntpq draw ASCII art only to
    > parse it and endure the changes between your releases?

    I personally don't release anything.

    >
    > What we really want to do is to use libntp to send out some queries to the
    > ntpd servers we monitor (10 of them) and async wait for replies to process
    > them immediately and with minimal latency.
    >
    >
    >>And you seem to work through your jobs in a serial way? Use select().

    >
    >
    > Well, yes of course. Short of using expect, we ran ntpq each time anew and
    > fetched its response immediately. That was good enough for a prototype that
    > re-used parsing code that we already have for ntpq output, but now we would
    > like to do better.

    well parsing any output with expect is trivial.
    I wonder why Eurocontrol is well served with tcl.

    G!
    uwe

    --
    Uwe Klein [mailto:uklein@klein-messgeraete.de]
    KLEIN MESSGERAETE Habertwedt 1
    D-24376 Groedersby b. Kappeln, GERMANY
    phone: +49 4642 920 123 FAX: +49 4642 920 125

  8. Re: No libntp.so

    Hello Mr. Unruh,

    Am Dienstag, 26. August 2008 21:31:00 schrieb Unruh:
    > kayhayen@gmx.de (Kay Hayen) writes:
    > >Hello NTP-World :-),
    > >
    > >we are implementing a NTP supervision for our ATC middleware. Initially we
    > > are doing it by repeated "ntpq" executions and textual evaluations of the
    > > results. We have had to notice that pipes are very unsuited for the task,
    > > so we really fork it and close its stdin to make it flush. It works, but
    > > it is unconvincing in performance (latency).

    >
    > Why not fflush it? And why would you need lack of latency. It is a report.
    > It is not time critical or should not be. What you are doing sounds bizzare
    > to me.


    I don't think it's a lack of our sides to fflush. Without a tty, ntpq itself
    will not flush I think. So when using pipes, we have to close stdin, which
    will make it flush its output due to exiting.

    And to not have a visible zombie (for too long), we need to wait for its exit
    code too.

    > >It appears that the whole ntpq call is relatively slow when, when we do it
    > > on e.g. RHEL 5.1 on modern Dual Core hardware, we get up to 25ms
    > > execution time, which is very long for us.

    >
    > What in the world are you trying to do? ntpq is NOT the way to get the
    > current time from the system. Use gettimeofday (linux) to get the current
    > time with sub microsecond latency. Do not use ntpq.


    Well, we are trying to get ntpd status information of a single node. I think
    we are using "peers", "assoc" and "rv" commands to learn the current offsets,
    sync states with peers, etc.

    The execution of this seems to be 25ms in cases, that's much I believe for
    nothing at all. Which is why we want to use libntp.a and avoid to use ntpq at
    all for querying the information we seek.

    > >What we really would prefer instead, is to see is the addition of a target
    > > to your makefile to make a libntp.so buildable. Then, we could already
    > > use the original releases as from you. From there we would work with
    > > downstreams (Debian and RHEL) to convince them to include libntp.so in
    > > the NTP packages and (libntp.a and headers obviously) possible through
    > > some ntp-devel package, at which point we could drop building from source
    > > again.

    >
    > Please tell us what you are trying to do. You are telling us your solution
    > ( which sounds really bad) instead of the problem you are trying to solve,
    > and then trying to convince the whole communtity to impliment your
    > solution.


    Uhm, well you are right.

    We need to assess NTP status information. We consider the details of every
    peer of the system, reachability, jitter, etc.

    Ideally as it occurs with no delays. Ideally we would be able to poll() some
    file descriptor whenever something has changed for an ntpd. I don't think
    such an interface exists. Short of that we would check it with reasonable
    frequency.

    We need to do it for many hosts to calculate a "CLOCK" status for the host,
    and the clock status would e.g. be bad if an ntpd is synchronized to an
    internal host, but not one the allowed external hosts. There is a set of
    criterias subject to user configuration.

    We need to be able to actively restrict NTP servers that are out bounds with
    offsets. Servers that e.g. mishandle a leap second and are 1 second off from
    that time on, or have other malfunctions, will be disabled that way.

    We need to publish the information for all hosts via SNMP.

    That's about it.

    I understand that ntpq could sort of be convinced to give out the information
    in reasonable time frames if we have it continously running as a child and
    get it to flush by using ptys.

    I also understand that ntpq simply uses libntp.a and decorates the results
    with some ASCII art that we would decode back. That's merely a source of bugs
    and uglyness. The ptys and extra process only introduces useless delays.

    > >The question of course would be: Will you merge such Makefile patches from
    > > us in the first place. And do you see any reason why this should not be
    > > done like this. After all, it has not been done for a long time already,
    > > so possibly this is on purpose?

    >
    > Does anyone see any reason why it should be done.


    Well, if there was an libntp.so, I would e.g. expect Python bindings to exist
    and the need for using ntpq in python-expect and parsing its output would
    simply go away entirely. Performance and ease of integration can only
    increase that way.

    I was asking for a reason why it is not so, because about everybody else in
    Free Software has an lib available. It's hard to find a server on the system
    without a lib package these days, isn't it?

    Reasons to not do it would only be things like "but we don't want a stable
    interface", etc. that is why I am asking, because it could well be that it's
    on purpose.

    I am not trying to make your decisions.

    But certainly the first thing we looked for when we started was libntp.[a|so]
    in our Linux distribution, but it didn't exist. A check with search engines
    didn't reveal why this is so. It could simply be because nobody else cared
    (deeply enough) about it so far.

    > >Obviously we hope you will allow us to be good Free Software citizens and
    > > let us drop the fork down the road. Will you?

    >
    > You are of course free to drop it. Whether anyone will pick it up is
    > another question. Certainly you have not convinced me, but that is
    > irrelevant since I have absolutely no control over what goes into ntp. But
    > that may or may be indicative of how well you have convinced more
    > influencial people.


    Let me be clear: I did not mean to fork the ntp code base in a public way. I
    can have libntp.a as it is now and can actually use it, in our own project
    (which will be Free Software later on), but that feels wrong.

    It's waste in two ways:

    First if we patch only for us in that way and compile ntp ourselves, others
    won't benefit from it. Second we track ntp upstream over and over and build
    code that is already on the target system again, just not accessible, because
    there are only binaries, no libs.

    What we are not asking is for NTP community to give up or change its goals. I
    don't want that, I couldn't achieve it. I just don't know them!

    We are offering our willingness to:

    a) Create patches that make the libntp.a and libntp.so (we are not yet ready
    with that, as we have other important milestones before we start this).
    b) Work to integrate them into your work.
    c) Work with distributors to change their packaging of ntp to take benefit of
    it.

    If NTP community says no to step b) we can choose a much easier way of solving
    a) because we don't need build the library in a tremendously portable way, as
    an example.

    Lets say, I have narrowed this (in my mind) down to availablity of libntp.a
    and/or libntp.so in common ntp installations for ready access by our
    software. And my suspect is that due to the BSD-alike license normally
    everybody just silently takes the code.

    I believe older middleware from another vendor that I am familiar with did
    that probably, unless of course they implemented the RFC themselves. But both
    would be waste, and we want to be good Free Software citizens.

    Best regards,
    Kay Hayen

  9. Re: No libntp.so

    Hello Martin,

    thanks for the reply, you wrote:

    > As mentioned in another reply it depends on what ntpq has to do during
    > those 25 milliseconds. Please be aware that this may include DNS lookups
    > which might fail, so the the call would be take until the DNS lookup times
    > out.


    We are using "ntpq -n" which makes a huge difference for the peers command and
    make all DNS lookups obsolete. We are also not using DNS on these machines,
    but only a hosts file, so normally this couldn't play a role.

    > > As we would like do other important stuff in the same process, we would
    > > like to do that faster. So I had a look at the source and found that ntpq
    > > is using a libntp.a that nobody seems to package though. There is no
    > > libntp.so either, we had searched for those initially, but no luck.

    >
    > Libntp just contains a couple of functions which are shared between several
    > programs of the NTP package, e.g. ntpd, ntpq and others. It is used during
    > the build process of the package in order to save compiule time.
    >
    > Even if you would be made available als shared object file it would not
    > provide the full ntpq functionality.


    I have not come to fully understand where the code is that is executed when I
    enter "peers", but it seemed that the encoding and decoding of packets could
    be found there at least.

    But checking now, the RFC 2553 is not the NTP one. Would you happen to know if
    the decoding/encoding of the packets are part of the library? I could not
    identify it right now.

    > IMHO it would not make much sense to provide the existing libntp as shared
    > object library. There is some ongoing work which will encapsulate the ntpq
    > functionality in a shared object library, so it would be easier to use from
    > an own application.


    Can you point me to that work? It seems like we would want to contribute to
    that effort then.

    > However, those calls may also block, so the basic requirement to call those
    > functions from an own tread/task would be unchanged.


    One would hope that it could provide both kinds of APIs by allowing us to poll
    on the reply receiving socket. That's not normally very hard to refactor code
    that waits for a reply to give back control and continue when the reply comes
    in. Provided of course, the protocol allows questions and replies to be
    matched by another criterion than sequence, can they?

    > > The question of course would be: Will you merge such Makefile patches
    > > from us in the first place. And do you see any reason why this should not
    > > be done like this. After all, it has not been done for a long time
    > > already, so possibly this is on purpose?
    > >
    > > Obviously we hope you will allow us to be good Free Software citizens and
    > > let us drop the fork down the road. Will you?

    >
    > The question is not basically whether a patch is accepted. In the first
    > place the question is whether a patch makes sense at all. As already
    > mentioned above I think it does not much sense just to provide libntp
    > as .so file.


    If it's only based on the fact that the NTP code isn't ready yet, we will be
    happy as it seems that we can work with you to get it there.

    It seems I was overly optimistic when I saw the listing of libntp directory,
    because it included an RFC in file name. Too bad that's not the NTP RFC. :-/

    Best regards,
    Kay Hayen

  10. Re: No libntp.so

    Hello Uwe,

    > > Like I said, we know, pipes are not suited at all for the task. I must
    > > admit, I was not aware of the issue before, I was under impression, we
    > > could simply set stdin/stdout to no buffering, but it appears for pipes
    > > that is not an option, and thinking about it, pipes really need to
    > > buffer.

    >
    > MY guess is you don't know enough about unix(like) systems to build any
    > meaningfull software with the stringent requirements linked to your
    > applications environment.
    > You may want to read up on pipes, ptys and associated buffering.


    I have no idea what triggered this.

    > > We were considering to use an expect wrapper (python-expect) to channel
    > > the data to us, and it would certainly improve the results. But I see no
    > > reason to use that workaround at all. Why should we make ntpq draw ASCII
    > > art only to parse it and endure the changes between your releases?

    >
    > I personally don't release anything.


    I didn't say so. In German I would have said "Dnderungen zwischen Euren
    Releases" and it would have been clear that I didn't mean you personally, of
    I did course not. In English it happens to be the same word, "your" for one
    person or a group, sorry if I was not clear enough.

    > > What we really want to do is to use libntp to send out some queries to
    > > the ntpd servers we monitor (10 of them) and async wait for replies to
    > > process them immediately and with minimal latency.
    > >
    > >>And you seem to work through your jobs in a serial way? Use select().

    > >
    > > Well, yes of course. Short of using expect, we ran ntpq each time anew
    > > and fetched its response immediately. That was good enough for a
    > > prototype that re-used parsing code that we already have for ntpq output,
    > > but now we would like to do better.

    >
    > well parsing any output with expect is trivial.
    > I wonder why Eurocontrol is well served with tcl.


    And the point of it is what? I didn't say anything against tcl, nor would I,
    nor could I. It's a fine tool.

    But given the choice, we would prefer to use a libntp. What's wrong with
    that,
    except that you don't find it necessary? On the pro-side there is:

    a) A significant reduction in lines of code used which can only lead to less
    bugs.
    b) The CPU used to make one set of queries will be lower potentially enabling
    a higher frequency of checks achieving lower latency in detecting NTP
    problems.

    Best regards,
    Kay Hayen

  11. Re: No libntp.so

    Hello Joseph,

    Am Mittwoch, 27. August 2008 15:00:25 schrieb Joseph Gwinn:
    > In article <200808260659.13498.kayhayen@gmx.de>,
    >
    > kayhayen@gmx.de (Kay Hayen) wrote:
    > > Hello NTP-World :-),
    > >
    > > we are implementing a NTP supervision for our ATC middleware. Initially
    > > we are doing it by repeated "ntpq" executions and textual evaluations of
    > > the results. We have had to notice that pipes are very unsuited for the
    > > task, so we really fork it and close its stdin to make it flush. It
    > > works, but it is unconvincing in performance (latency).

    >
    > What exactly is "NTP supervision"? What problem does it solve?


    I will gladly give examples of NTP functioning perfect and it still being
    needed, bare with me if they lack in one way or the other:

    a) Suppose you have a system of 10 hosts that must have near identical time,
    even if not correct. You may be using PC hardware with non-optimal clocks
    because DEC is no more. You want to know if one node has an offset higher
    than say 200ms. Can happen, and then you want to know, you don't even want to
    start the software if e.g. right after boot these conditions are not already
    met.

    b) Suppose you have 2 hosts of these 10, and they are the only ones with
    connection to an "NTP LAN". They peer the 5 or even 7 external NTP servers
    that you have. For the case that the "external" NTP servers with low stratum
    values failing, you make those 2 external hosts peer another, but at high
    stratum value. But if that has happenen, you want to know, because isolation
    from external time input will lead to trouble sooner or later.

    c) Suppose you have bought that fine NTP clock from a fine vendor, but it
    doesn't always work reliable. Some of them behave strange. I have seen my
    share of them. I think I know of a clock that doesn't work correctly when
    there is too much traffic on the LAN, it seems to be able to decode only so
    much in a given time, then drop things. The NTP algorithms correctly flag
    that clock (I don't know as what exactly they did, I didn't analyse that
    myself), but it's certainly nice to see that reported on your SNMP system
    where something becomes red instead of some log file only or false results
    from the software.

    I think "NTP supervision" mainly has to solve the problem of not allowing the
    software to continue on system with currently bad clock input and interfacing
    in a nice way with SNMP based supervision systems where a technical watch
    operator can easily visualize the health status of a system and take
    necessary actions.

    Best regards,
    Kay Hayen

  12. Re: No libntp.so

    Hello Kay,

    Kay Hayen wrote:
    > Hello Martin,
    >
    > thanks for the reply, you wrote:
    >
    >>As mentioned in another reply it depends on what ntpq has to do during
    >>those 25 milliseconds. Please be aware that this may include DNS lookups
    >>which might fail, so the the call would be take until the DNS lookup times
    >>out.

    >
    > We are using "ntpq -n" which makes a huge difference for the peers command and
    > make all DNS lookups obsolete. We are also not using DNS on these machines,
    > but only a hosts file, so normally this couldn't play a role.


    OK. However, if 25 milliseconds is still too long for other task you are
    running you should do this asynchronously anyway.

    >>>As we would like do other important stuff in the same process, we would
    >>>like to do that faster. So I had a look at the source and found that ntpq
    >>>is using a libntp.a that nobody seems to package though. There is no
    >>>libntp.so either, we had searched for those initially, but no luck.

    >>Libntp just contains a couple of functions which are shared between several
    >>programs of the NTP package, e.g. ntpd, ntpq and others. It is used during
    >>the build process of the package in order to save compiule time.
    >>
    >>Even if you would be made available als shared object file it would not
    >>provide the full ntpq functionality.

    >
    > I have not come to fully understand where the code is that is executed when I
    > enter "peers", but it seemed that the encoding and decoding of packets could
    > be found there at least.


    Search the code for dopeers(), dogetpeers(), doprintpeers(). The packets
    returned from ntpd already contain some ASCII text with the desired data.

    > But checking now, the RFC 2553 is not the NTP one. Would you happen to know if
    > the decoding/encoding of the packets are part of the library? I could not
    > identify it right now.


    This is mainly done in ntpq_subs.c which is not in the library. This is
    why I think it would not help you just to distribute libntp as a .so file.

    >>IMHO it would not make much sense to provide the existing libntp as shared
    >>object library. There is some ongoing work which will encapsulate the ntpq
    >>functionality in a shared object library, so it would be easier to use from
    >>an own application.

    >
    > Can you point me to that work? It seems like we would want to contribute to
    > that effort then.
    >
    >>However, those calls may also block, so the basic requirement to call those
    >>functions from an own tread/task would be unchanged.

    >
    > One would hope that it could provide both kinds of APIs by allowing us to poll
    > on the reply receiving socket. That's not normally very hard to refactor code
    > that waits for a reply to give back control and continue when the reply comes
    > in. Provided of course, the protocol allows questions and replies to be
    > matched by another criterion than sequence, can they?


    You should discuss this with the guy who's working on this. I'll ask him
    to contact you. BTW, I wonder why your post to which I'm just replying
    is not yet shown by the public news servers?

    >>>The question of course would be: Will you merge such Makefile patches
    >>>from us in the first place. And do you see any reason why this should not
    >>>be done like this. After all, it has not been done for a long time
    >>>already, so possibly this is on purpose?
    >>>
    >>>Obviously we hope you will allow us to be good Free Software citizens and
    >>>let us drop the fork down the road. Will you?

    >>The question is not basically whether a patch is accepted. In the first
    >>place the question is whether a patch makes sense at all. As already
    >>mentioned above I think it does not much sense just to provide libntp
    >>as .so file.

    >
    > If it's only based on the fact that the NTP code isn't ready yet, we will be
    > happy as it seems that we can work with you to get it there.
    >
    > It seems I was overly optimistic when I saw the listing of libntp directory,
    > because it included an RFC in file name. Too bad that's not the NTP RFC. :-/


    Right. The RFC for NTPv3 is 1305. See at the bottom of:
    http://www.meinberg.de/english/info/ntp.htm

    For NTPv4 there's no RFC but an IEEE draft. See:
    http://www.ietf.org/proceedings/05nov/ntp.html


    Best regards,

    Martin
    --
    Martin Burnicki

    Meinberg Funkuhren
    Bad Pyrmont
    Germany

  13. Re: No libntp.so

    On 2008-08-28, Martin Burnicki wrote:

    > Kay Hayen wrote:
    >
    >> One would hope that it could provide both kinds of APIs by allowing
    >> us to poll on the reply receiving socket. That's not normally very
    >> hard to refactor code that waits for a reply to give back control and
    >> continue when the reply comes in. Provided of course, the protocol
    >> allows questions and replies to be matched by another criterion than
    >> sequence, can they?

    >
    > You should discuss this with the guy who's working on this. I'll ask
    > him to contact you. BTW, I wonder why your post to which I'm just
    > replying is not yet shown by the public news servers?


    Kay is communicating via the questions@ list.

    All of his recent replies to this thread were being held for
    moderation. I've released this set of messages but did not clear his
    moderation bit.

    --
    Steve Kostecke
    NTP Public Services Project - http://support.ntp.org/

  14. Re: No libntp.so


    Steve Kostecke wrote:
    > On 2008-08-28, Martin Burnicki wrote:
    >
    >> Kay Hayen wrote:
    >>
    >>> One would hope that it could provide both kinds of APIs by allowing
    >>> us to poll on the reply receiving socket. That's not normally very
    >>> hard to refactor code that waits for a reply to give back control and
    >>> continue when the reply comes in. Provided of course, the protocol
    >>> allows questions and replies to be matched by another criterion than
    >>> sequence, can they?

    >>
    >> You should discuss this with the guy who's working on this. I'll ask
    >> him to contact you. BTW, I wonder why your post to which I'm just
    >> replying is not yet shown by the public news servers?

    >
    > Kay is communicating via the questions@ list.


    Yes I saw his email via question@ some hours before the messages showed up
    on the news server I'm using.

    > All of his recent replies to this thread were being held for
    > moderation. I've released this set of messages but did not clear his
    > moderation bit.


    Ah, that's that's the reason why.

    Thanks, Steve.

    Martin
    --
    Martin Burnicki

    Meinberg Funkuhren
    Bad Pyrmont
    Germany

  15. Re: No libntp.so

    In article <200808280152.48311.kayhayen@gmx.de>,
    kayhayen@gmx.de (Kay Hayen) wrote:

    > Hello Joseph,
    >
    > Am Mittwoch, 27. August 2008 15:00:25 schrieb Joseph Gwinn:
    > > In article <200808260659.13498.kayhayen@gmx.de>,
    > >
    > > kayhayen@gmx.de (Kay Hayen) wrote:
    > > > Hello NTP-World :-),
    > > >
    > > > we are implementing a NTP supervision for our ATC middleware. Initially
    > > > we are doing it by repeated "ntpq" executions and textual evaluations of
    > > > the results. We have had to notice that pipes are very unsuited for the
    > > > task, so we really fork it and close its stdin to make it flush. It
    > > > works, but it is unconvincing in performance (latency).

    > >
    > > What exactly is "NTP supervision"? What problem does it solve?

    >
    > I will gladly give examples of NTP functioning perfect and it still being
    > needed, bear with me if they lack in one way or the other:
    >
    > a) Suppose you have a system of 10 hosts that must have near identical time,
    > even if not correct. You may be using PC hardware with non-optimal clocks
    > because DEC is no more. You want to know if one node has an offset higher
    > than say 200ms. Can happen, and then you want to know, you don't even want to
    > start the software if e.g. right after boot these conditions are not already
    > met.
    >
    > b) Suppose you have 2 hosts of these 10, and they are the only ones with
    > connection to an "NTP LAN". They peer the 5 or even 7 external NTP servers
    > that you have. For the case that the "external" NTP servers with low stratum
    > values failing, you make those 2 external hosts peer another, but at high
    > stratum value. But if that has happenen, you want to know, because isolation
    > from external time input will lead to trouble sooner or later.
    >
    > c) Suppose you have bought that fine NTP clock from a fine vendor, but it
    > doesn't always work reliable. Some of them behave strange. I have seen my
    > share of them. I think I know of a clock that doesn't work correctly when
    > there is too much traffic on the LAN, it seems to be able to decode only so
    > much in a given time, then drop things. The NTP algorithms correctly flag
    > that clock (I don't know as what exactly they did, I didn't analyse that
    > myself), but it's certainly nice to see that reported on your SNMP system
    > where something becomes red instead of some log file only or false results
    > from the software.
    >
    > I think "NTP supervision" mainly has to solve the problem of not allowing the
    > software to continue on system with currently bad clock input and interfacing
    > in a nice way with SNMP based supervision systems where a technical watch
    > operator can easily visualize the health status of a system and take
    > necessary actions.


    OK. I see the problem.

    One simple and very fast approach that occurs to me is to write a small
    C program that sends a NTP packet to each of the NTP servers in
    question, and compares the resulting reply packets. By NTP rules, a
    given server should be queried no more than once per minute, and even
    that is probably faster than your application requires. Sounds like
    once per startup is more like it.

    NTP packets are small and simple, and the NTPv3 packet format is
    documented in Appendix A of RFC-1305. I would use the NTPv3 packet
    format for compatibility with both NTPv3 and NTPv4 servers.

    Joe Gwinn

  16. Re: No libntp.so

    Kay,

    Kay Hayen wrote:
    > Hello Uwe,
    > Use Klein wrote:
    >> Kay Hayen wrote:
    >> > We were considering to use an expect wrapper (python-expect) to channel
    >> > the data to us, and it would certainly improve the results. But I see
    >> > no reason to use that workaround at all. Why should we make ntpq draw
    >> > ASCII art only to parse it and endure the changes between your
    >> > releases?

    >>
    >> I personally don't release anything.


    Kay, I think your mentioning of "your releases" made Uwe thik you are
    assuming he's involved in the NTP release cycles, but he's not.

    > I didn't say so. In German I would have said "Dnderungen zwischen Euren
    > Releases"


    Auch mit "Euren Releases" ist Uwe angesprochen. See above.

    [...]
    > But given the choice, we would prefer to use a libntp. What's wrong with
    > that,
    > except that you don't find it necessary? On the pro-side there is:
    >
    > a) A significant reduction in lines of code used which can only lead to
    > less bugs.
    > b) The CPU used to make one set of queries will be lower potentially
    > enabling a higher frequency of checks achieving lower latency in detecting
    > NTP problems.


    The problem is that libntp is only used internally to build other
    components. It does not provide all the functionality provided by "ntpq
    -np" since lots of ntpq stuff is in the ntpq-specific source modules.

    Ntpq sends queries to ntpd, and ntpd replies with a long string of
    "variable=value" pairs. Ntpq then parses that string and displays the
    relevant information in a user-friendly way.

    A shared object library which supports ntpq-like queries must contain the
    basic functions of ntpq, i.e. send queries to ntpd and receive and assemble
    the replies which may arrive in multiple UDP packets.

    I think for external applications like the one you're going to write it
    would not be required first to print the results to a billboard and then
    parse that billboard. It would be more convenient to have a function which
    returns a string buffer with the var=val pairs, or another function which
    retrieves and only returns the value of a specific variable from that
    buffer.

    Which functionality exactly would be provided by such a library, and how
    those functions are called to avoid naming conflicts should be subject to
    discussions.

    BTW, do you know the NTP time server monitor?
    http://www.meinberg.de/english/sw/ti...er-monitor.htm

    That Windows program can monitor several NTP servers, and it uses an a DLL
    which encapsulates the ntpq functionality as described above. However, this
    contains some inofficial modifications of the NTP source code, i.e. those
    modifications are not in the public code base.

    The goal of the ongoing work on such an official library (which I've
    mentioned earlier) is to get those modifications to a proper state which
    can be submitted to the NTP code base.

    Martin
    --
    Martin Burnicki

    Meinberg Funkuhren
    Bad Pyrmont
    Germany

  17. Re: No libntp.so

    Hi Kay,

    I am working on a libntpq which I developed for a Google Summer of Code project.
    The main goal of this GSoC project was to build an SNMP sub agent daemon which
    would be used with any AgentX compliant SNMP agent (like net-snmp.org). The
    beast called ntpsnmpd will be responsible for the standard NTPv4 MIB that is
    currently in the standardization process within the IETF.

    Martin (my colleague here at Meinberg) asked me to contact you about this issue.
    You will find some information about this project here:
    http://support.ntp.org/bin/view/Dev/GSoC2008snmp

    At the moment I am working on releasing the next version of the NTP Installer
    for Windows (which will include 4.2.4p5) and as soon as I am done with this, I
    will continue to work on the libntpq stuff. It should appear pretty soon in the
    ntp-dev sourcetree (~2 weeks I guess). If you want to test it before that date,
    please contact me via email or telephone. See http://www.meinberg.de for contact
    details.

    In its current state libntpq will be a static library (libntpq.a), but it should
    be a snap to build it as a shared object version.

    The library itself contains the ntpq sourcecode and a number of wrapper
    functions that are exported and can be accessed from applications that are
    linked against the library.

    As Martin already pointed out, this approach results in getting rid of the ASCII
    output generation stuff (that is in your application thrown away by a parser)
    but otherwise resembles what ntpq would do. I think you will probably be able to
    speed things up a little bit because calling a library function will be faster
    than firing up a new process, open a pipe and then read and parse the output.

    Best Regards,
    Heiko

  18. Re: No libntp.so

    On 2008-08-27, Kay Hayen wrote:

    > "Joseph Gwinn" wrote:
    >
    >> kayhayen@gmx.de (Kay Hayen) wrote:
    >>
    >>> we are implementing a NTP supervision for our ATC middleware.
    >>> Initially we are doing it by repeated "ntpq" executions and textual
    >>> evaluations of the results. We have had to notice that pipes are
    >>> very unsuited for the task, so we really fork it and close its stdin
    >>> to make it flush. It works, but it is unconvincing in performance
    >>> (latency).


    Why is 25ms latency for statistics collection unacceptable for a
    protocol which uses polling intervals between 64 and 1024 seconds?

    >> What exactly is "NTP supervision"? What problem does it solve?

    >
    > I will gladly give examples of NTP functioning perfect and it still
    > being needed, bare with me if they lack in one way or the other:
    >
    > a) Suppose you have a system of 10 hosts that must have near identical
    > time, even if not correct. You may be using PC hardware with
    > non-optimal clocks because DEC is no more. You want to know if one
    > node has an offset higher than say 200ms. Can happen, and then you
    > want to know, you don't even want to start the software if e.g. right
    > after boot these conditions are not already met.


    By default ntpd steps the clock if the offset is greater than 128 ms.

    The only time that you should see an offset of this magnitude, and the
    associated clock step, is at boot time. If you are seeing repeated steps
    then you have issues outside of NTP that need to be dealt with.

    As far as monitoring, you can trivially use another ntpd to monitor
    those 10 hosts and compare their offset to your official time sources.
    This is accomplished by adding the client hosts to your monitoring ntpd
    with the following ntp.conf line:

    server client_N noselect

    Repeat this line for each client host. You will then be able to view an
    aggregated peers billboard which shows the offset/delay/jitter to the
    reference sources _and_ all of those 10 clients. If you enable
    statistics collection on the monitoring ntpd you can tail the peerstats
    file and monitor the client offset, etc., from there.

    > b) Suppose you have 2 hosts of these 10, and they are the only ones
    > with connection to an "NTP LAN". They peer the 5 or even 7 external
    > NTP servers that you have. For the case that the "external" NTP
    > servers with low stratum values failing, you make those 2 external
    > hosts peer another, but at high stratum value. But if that has
    > happenen, you want to know, because isolation from external time input
    > will lead to trouble sooner or later.


    You probably want to configure those 10 clients in Orphan Mode so that
    they will autonomously select a leader in the event that your external
    time sources are unreachable.

    > c) Suppose you have bought that fine NTP clock from a fine vendor,
    > but it doesn't always work reliable. Some of them behave strange. I
    > have seen my share of them. I think I know of a clock that doesn't
    > work correctly when there is too much traffic on the LAN, it seems
    > to be able to decode only so much in a given time, then drop things.
    > The NTP algorithms correctly flag that clock (I don't know as what
    > exactly they did, I didn't analyse that myself), but it's certainly
    > nice to see that reported on your SNMP system where something becomes
    > red instead of some log file only or false results from the software.


    If time is _that_ critical to your application then you need to provide
    an appropriate level of redundancy to withstand failures along with a
    reasonable diversity of time sources to avoid homogeneous malfunctions.
    And don't overlook the SPOF.

    > I think "NTP supervision" mainly has to solve the problem of not
    > allowing the software to continue on system with currently bad clock
    > input and interfacing in a nice way with SNMP based supervision
    > systems where a technical watch operator can easily visualize the
    > health status of a system and take necessary actions.


    As stated before, libntp only provides a few common functions used in
    a number of places in the NTP distribution. The place to look for the
    monitoring implementation is the ./ntpq directory in the Distribution.

    The NTP Copyright allows liberal use of the source code. See ./COPYRIGHT
    or ./html/copyright.html in the Distribution for the version specific
    copyright notice. The current NTP Copyright Notice is available on-line
    at http://www.eecis.udel.edu/~mills/ntp...copyright.html.

    --
    Steve Kostecke
    NTP Public Services Project - http://support.ntp.org/

  19. Re: No libntp.so

    Kay Hayen wrote:

    > I don't think it's a lack of our sides to fflush. Without a tty, ntpq itself
    > will not flush I think. So when using pipes, we have to close stdin, which
    > will make it flush its output due to exiting.
    >
    > And to not have a visible zombie (for too long), we need to wait for its exit
    > code too.

    As I wrote before that is the reason why ptys exist.
    ( and the mechanism used by expect to talk to spawned processes ).

    man openpty, login_tty, forkpty

    forkpty() followed by exec*() in the child is probably what you want.
    you will then have a child that reacts in a linebuffered way like
    you are used from interactively run apps.

    uwe

  20. Re: No libntp.so

    kayhayen@gmx.de (Kay Hayen) writes:

    >Hello Uwe,


    >thank you for your reply. :-)


    >Am Dienstag, 26. August 2008 22:29:03 schrieb Uwe Klein:
    >> Kay Hayen wrote:
    >> > Hello NTP-World :-),
    >> >
    >> > we are implementing a NTP supervision for our ATC middleware. Initially
    >> > we are

    >>
    >> ATC as in air traffic control?


    >Yes that's right. I have this understanding that NTP was invented in the
    >field, wasn't it?


    >> > doing it by repeated "ntpq" executions and textual evaluations of the
    >> > results. We have had to notice that pipes are very unsuited for the task,
    >> > so we really fork it and close its stdin to make it flush. It works, but
    >> > it is unconvincing in performance (latency).
    >> >
    >> > It appears that the whole ntpq call is relatively slow when, when we do
    >> > it on e.g. RHEL 5.1 on modern Dual Core hardware, we get up to 25ms
    >> > execution time, which is very long for us.

    >>
    >> First think about what ntpq has to do during startup. Then think again ;-)


    >Starting ntpq does to seem to do nothing at all. When we do the peers command,
    >I saw it can be slow without -n option (checking with tshark), but we are
    >using that.


    >I don't think we want to use ntpq at all though.


    >> On the piping issue:
    >> Thats what ptys are for. You would have ntpq as a running process that can
    >> be queried at any time.
    >> You run into nonlinebuffering issues on stdin and stdout due to both
    >> not being ttys.


    >Like I said, we know, pipes are not suited at all for the task. I must admit,
    >I was not aware of the issue before, I was under impression, we could simply
    >set stdin/stdout to no buffering, but it appears for pipes that is not an
    >option, and thinking about it, pipes really need to buffer.


    >We were considering to use an expect wrapper (python-expect) to channel the
    >data to us, and it would certainly improve the results. But I see no reason
    >to use that workaround at all. Why should we make ntpq draw ASCII art only to
    >parse it and endure the changes between your releases?


    >What we really want to do is to use libntp to send out some queries to the
    >ntpd servers we monitor (10 of them) and async wait for replies to process
    >them immediately and with minimal latency.


    >> And you seem to work through your jobs in a serial way? Use select().


    >Well, yes of course. Short of using expect, we ran ntpq each time anew and
    >fetched its response immediately. That was good enough for a prototype that
    >re-used parsing code that we already have for ntpq output, but now we would
    >like to do better.


    >I don't think we would have done ntpq parsing had it not been that this code
    >existed for historic reasons.


    >> > Obviously we hope you will allow us to be good Free Software citizens and
    >> > let us drop the fork down the road. Will you?

    >>
    >> you may want to look around to how these people solved their probs:
    >> http://wiki.tcl.tk/15535


    >That part of EUROCONTROL (and them in general) is well known to me. As
    >libntp.a seems to offer all it takes, this question is more about how to get
    >access to it in sane way.


    You still have not told us what problem you are trying to solve. If you
    want to know what time those various ntp devices have and whether they are
    on time, just send them an ntp request packet and see what time they
    deliver back to you. But please, tell us the problem you are trying to
    solve. You are completely obsessed with your own solution, which sounds to
    me at least to be highly suboptimal. And you want the whole community to
    change to make your solution easy to impliment. Now that may be a valid
    need and a valid solution, but at least to me it does not sound like it.


+ Reply to Thread
Page 1 of 2 1 2 LastLast