No libntp.so - NTP

This is a discussion on No libntp.so - NTP ; kayhayen@gmx.de (Kay Hayen) writes: >Hello Mr. Unruh, >> >> What in the world are you trying to do? ntpq is NOT the way to get the >> current time from the system. Use gettimeofday (linux) to get the current >> ...

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2
Results 21 to 31 of 31

Thread: No libntp.so

  1. Re: No libntp.so

    kayhayen@gmx.de (Kay Hayen) writes:

    >Hello Mr. Unruh,


    >>
    >> What in the world are you trying to do? ntpq is NOT the way to get the
    >> current time from the system. Use gettimeofday (linux) to get the current
    >> time with sub microsecond latency. Do not use ntpq.


    >Well, we are trying to get ntpd status information of a single node. I think
    >we are using "peers", "assoc" and "rv" commands to learn the current offsets,
    >sync states with peers, etc.


    The ntpd status does not change for minutes to hours. Why in the world is
    25ms of concern. If you want to know what time the far machine has you are
    going about it in a very bad way. But I have no idea yet what you want.


    >The execution of this seems to be 25ms in cases, that's much I believe for
    >nothing at all. Which is why we want to use libntp.a and avoid to use ntpq at
    >all for querying the information we seek.


    >> >What we really would prefer instead, is to see is the addition of a target
    >> > to your makefile to make a libntp.so buildable. Then, we could already
    >> > use the original releases as from you. From there we would work with
    >> > downstreams (Debian and RHEL) to convince them to include libntp.so in
    >> > the NTP packages and (libntp.a and headers obviously) possible through
    >> > some ntp-devel package, at which point we could drop building from source
    >> > again.

    >>
    >> Please tell us what you are trying to do. You are telling us your solution
    >> ( which sounds really bad) instead of the problem you are trying to solve,
    >> and then trying to convince the whole communtity to impliment your
    >> solution.


    >Uhm, well you are right.


    >We need to assess NTP status information. We consider the details of every
    >peer of the system, reachability, jitter, etc.


    And 25ms matters why?


    >Ideally as it occurs with no delays. Ideally we would be able to poll() some
    >file descriptor whenever something has changed for an ntpd. I don't think
    >such an interface exists. Short of that we would check it with reasonable
    >frequency.


    >We need to do it for many hosts to calculate a "CLOCK" status for the host,
    >and the clock status would e.g. be bad if an ntpd is synchronized to an
    >internal host, but not one the allowed external hosts. There is a set of
    >criterias subject to user configuration.


    milliseconds or even seconds make no difference to this. No clock wanders
    off time that rapidly.


    >We need to be able to actively restrict NTP servers that are out bounds with
    >offsets. Servers that e.g. mishandle a leap second and are 1 second off from
    >that time on, or have other malfunctions, will be disabled that way.


    Again milliseconds is irrelevant. And why not set up your own gps server
    with external ones as backups.



    >We need to publish the information for all hosts via SNMP.


    >That's about it.


    >I understand that ntpq could sort of be convinced to give out the information
    >in reasonable time frames if we have it continously running as a child and
    >get it to flush by using ptys.


    >I also understand that ntpq simply uses libntp.a and decorates the results
    >with some ASCII art that we would decode back. That's merely a source of bugs
    >and uglyness. The ptys and extra process only introduces useless delays.


    >> >The question of course would be: Will you merge such Makefile patches from
    >> > us in the first place. And do you see any reason why this should not be
    >> > done like this. After all, it has not been done for a long time already,
    >> > so possibly this is on purpose?

    >>
    >> Does anyone see any reason why it should be done.


    >Well, if there was an libntp.so, I would e.g. expect Python bindings to exist
    >and the need for using ntpq in python-expect and parsing its output would
    >simply go away entirely. Performance and ease of integration can only
    >increase that way.


    >I was asking for a reason why it is not so, because about everybody else in
    >Free Software has an lib available. It's hard to find a server on the system
    >without a lib package these days, isn't it?


    >Reasons to not do it would only be things like "but we don't want a stable
    >interface", etc. that is why I am asking, because it could well be that it's
    >on purpose.


    >I am not trying to make your decisions.


    >But certainly the first thing we looked for when we started was libntp.[a|so]
    >in our Linux distribution, but it didn't exist. A check with search engines
    >didn't reveal why this is so. It could simply be because nobody else cared
    >(deeply enough) about it so far.


    >> >Obviously we hope you will allow us to be good Free Software citizens and
    >> > let us drop the fork down the road. Will you?

    >>
    >> You are of course free to drop it. Whether anyone will pick it up is
    >> another question. Certainly you have not convinced me, but that is
    >> irrelevant since I have absolutely no control over what goes into ntp. But
    >> that may or may be indicative of how well you have convinced more
    >> influencial people.


    >Let me be clear: I did not mean to fork the ntp code base in a public way. I
    >can have libntp.a as it is now and can actually use it, in our own project
    >(which will be Free Software later on), but that feels wrong.


    >It's waste in two ways:


    >First if we patch only for us in that way and compile ntp ourselves, others
    >won't benefit from it. Second we track ntp upstream over and over and build
    >code that is already on the target system again, just not accessible, because
    >there are only binaries, no libs.


    >What we are not asking is for NTP community to give up or change its goals. I
    >don't want that, I couldn't achieve it. I just don't know them!


    >We are offering our willingness to:


    >a) Create patches that make the libntp.a and libntp.so (we are not yet ready
    >with that, as we have other important milestones before we start this).
    >b) Work to integrate them into your work.
    >c) Work with distributors to change their packaging of ntp to take benefit of
    >it.


    >If NTP community says no to step b) we can choose a much easier way of solving
    >a) because we don't need build the library in a tremendously portable way, as
    >an example.


    >Lets say, I have narrowed this (in my mind) down to availablity of libntp.a
    >and/or libntp.so in common ntp installations for ready access by our
    >software. And my suspect is that due to the BSD-alike license normally
    >everybody just silently takes the code.


    >I believe older middleware from another vendor that I am familiar with did
    >that probably, unless of course they implemented the RFC themselves. But both
    >would be waste, and we want to be good Free Software citizens.


    >Best regards,
    >Kay Hayen


  2. Re: No libntp.so

    kayhayen@gmx.de (Kay Hayen) writes:

    >Hello Joseph,


    >Am Mittwoch, 27. August 2008 15:00:25 schrieb Joseph Gwinn:
    >> In article <200808260659.13498.kayhayen@gmx.de>,
    >>
    >> kayhayen@gmx.de (Kay Hayen) wrote:
    >> > Hello NTP-World :-),
    >> >
    >> > we are implementing a NTP supervision for our ATC middleware. Initially
    >> > we are doing it by repeated "ntpq" executions and textual evaluations of
    >> > the results. We have had to notice that pipes are very unsuited for the
    >> > task, so we really fork it and close its stdin to make it flush. It
    >> > works, but it is unconvincing in performance (latency).

    >>
    >> What exactly is "NTP supervision"? What problem does it solve?


    >I will gladly give examples of NTP functioning perfect and it still being
    >needed, bare with me if they lack in one way or the other:


    >a) Suppose you have a system of 10 hosts that must have near identical time,
    >even if not correct. You may be using PC hardware with non-optimal clocks
    >because DEC is no more. You want to know if one node has an offset higher
    >than say 200ms. Can happen, and then you want to know, you don't even want to
    >start the software if e.g. right after boot these conditions are not already
    >met.


    You send each host an ntp request packet and look at the times they report
    back. Accurate to a few 10s of micro (not milli) seconds if they are all on
    your local lan.


    >b) Suppose you have 2 hosts of these 10, and they are the only ones with
    >connection to an "NTP LAN". They peer the 5 or even 7 external NTP servers
    >that you have. For the case that the "external" NTP servers with low stratum
    >values failing, you make those 2 external hosts peer another, but at high
    >stratum value. But if that has happenen, you want to know, because isolation
    >from external time input will lead to trouble sooner or later.


    Set up a local gps timeserver, and use the external ones as fallback.
    The stratum is determined by the distance from an authoritative source. It
    is not reallysomething that you assign.
    I think by "peer" you mean"serve".

    But finding out that that an external source has failed is not something
    that requires 25ms accuracy. Or even seconds accuracy. Maybe hours.
    After all ntp does not query the external sources more than once every 20
    min, and then throws away 80% of those queries anyway. Ie, there is no
    rush.




    >c) Suppose you have bought that fine NTP clock from a fine vendor, but it
    >doesn't always work reliable. Some of them behave strange. I have seen my
    >share of them. I think I know of a clock that doesn't work correctly when
    >there is too much traffic on the LAN, it seems to be able to decode only so
    >much in a given time, then drop things. The NTP algorithms correctly flag
    >that clock (I don't know as what exactly they did, I didn't analyse that
    >myself), but it's certainly nice to see that reported on your SNMP system
    >where something becomes red instead of some log file only or false results
    >from the software.


    >I think "NTP supervision" mainly has to solve the problem of not allowing the
    >software to continue on system with currently bad clock input and interfacing
    >in a nice way with SNMP based supervision systems where a technical watch
    >operator can easily visualize the health status of a system and take
    >necessary actions.


    Sure. But the key issue with any time criticality is to make sure that the
    computers all share a time base.
    That you can do with ntp queries, not ntpq. Ie, the whole of ntp is set up
    precisely to answer with high accuracy, the main question you want to ask.
    And the structure of an ntp packet is open and well known and can be sent
    out by whatever software you want. It does not have to be ntpd which sends
    the query. The other machines just see the packet and respond with their
    own timestamps. The latency is the latency of your network, and the cost is
    a single packet each way. (about 16 bytes each way).


    This of course assumes thatthose machines are running ntpd, and will
    respond to queries, but since we are discussing ntpd it can.


  3. Re: No libntp.so

    martin.burnicki@meinberg.de (Martin Burnicki) writes:

    >Hello Kay,


    >Kay Hayen wrote:
    >> Hello Martin,
    >>
    >> thanks for the reply, you wrote:
    >>
    >>>As mentioned in another reply it depends on what ntpq has to do during
    >>>those 25 milliseconds. Please be aware that this may include DNS lookups
    >>>which might fail, so the the call would be take until the DNS lookup times
    >>>out.

    >>
    >> We are using "ntpq -n" which makes a huge difference for the peers command and
    >> make all DNS lookups obsolete. We are also not using DNS on these machines,
    >> but only a hosts file, so normally this couldn't play a role.


    >OK. However, if 25 milliseconds is still too long for other task you are
    >running you should do this asynchronously anyway.


    My suspicion is that are using those ntpq queries to try to figure out if
    the remote machines are "on time". Ie they use ntpq to query the time on
    the remote machine, and then read the time on their local machine to see if
    the remote machine is out of time sync. Then 25ms would be a long time.
    However, using ntpq to determine the time on the other machine is silly.



    >>>>As we would like do other important stuff in the same process, we would
    >>>>like to do that faster. So I had a look at the source and found that ntpq
    >>>>is using a libntp.a that nobody seems to package though. There is no
    >>>>libntp.so either, we had searched for those initially, but no luck.
    >>>Libntp just contains a couple of functions which are shared between several
    >>>programs of the NTP package, e.g. ntpd, ntpq and others. It is used during
    >>>the build process of the package in order to save compiule time.
    >>>
    >>>Even if you would be made available als shared object file it would not
    >>>provide the full ntpq functionality.

    >>
    >> I have not come to fully understand where the code is that is executed when I
    >> enter "peers", but it seemed that the encoding and decoding of packets could
    >> be found there at least.


    >Search the code for dopeers(), dogetpeers(), doprintpeers(). The packets
    >returned from ntpd already contain some ASCII text with the desired data.


    >> But checking now, the RFC 2553 is not the NTP one. Would you happen to know if
    >> the decoding/encoding of the packets are part of the library? I could not
    >> identify it right now.


    >This is mainly done in ntpq_subs.c which is not in the library. This is
    >why I think it would not help you just to distribute libntp as a .so file.


    >>>IMHO it would not make much sense to provide the existing libntp as shared
    >>>object library. There is some ongoing work which will encapsulate the ntpq
    >>>functionality in a shared object library, so it would be easier to use from
    >>>an own application.

    >>
    >> Can you point me to that work? It seems like we would want to contribute to
    >> that effort then.
    >>
    >>>However, those calls may also block, so the basic requirement to call those
    >>>functions from an own tread/task would be unchanged.

    >>
    >> One would hope that it could provide both kinds of APIs by allowing us to poll
    >> on the reply receiving socket. That's not normally very hard to refactor code
    >> that waits for a reply to give back control and continue when the reply comes
    >> in. Provided of course, the protocol allows questions and replies to be
    >> matched by another criterion than sequence, can they?


    >You should discuss this with the guy who's working on this. I'll ask him
    >to contact you. BTW, I wonder why your post to which I'm just replying
    >is not yet shown by the public news servers?


    >>>>The question of course would be: Will you merge such Makefile patches
    >>>>from us in the first place. And do you see any reason why this should not
    >>>>be done like this. After all, it has not been done for a long time
    >>>>already, so possibly this is on purpose?
    >>>>
    >>>>Obviously we hope you will allow us to be good Free Software citizens and
    >>>>let us drop the fork down the road. Will you?
    >>>The question is not basically whether a patch is accepted. In the first
    >>>place the question is whether a patch makes sense at all. As already
    >>>mentioned above I think it does not much sense just to provide libntp
    >>>as .so file.

    >>
    >> If it's only based on the fact that the NTP code isn't ready yet, we will be
    >> happy as it seems that we can work with you to get it there.
    >>
    >> It seems I was overly optimistic when I saw the listing of libntp directory,
    >> because it included an RFC in file name. Too bad that's not the NTP RFC. :-/


    >Right. The RFC for NTPv3 is 1305. See at the bottom of:
    >http://www.meinberg.de/english/info/ntp.htm


    >For NTPv4 there's no RFC but an IEEE draft. See:
    >http://www.ietf.org/proceedings/05nov/ntp.html




  4. Re: No libntp.so

    Kay Hayen wrote:
    > because DEC is no more. You want to know if one node has an offset higher
    > than say 200ms. Can happen, and then you want to know, you don't even want to


    Why do you believe that that offset represents a true error in the
    node's clock time when ntpd doesn't? If ntpd knew that the error was
    real, it should have removed it. This question may have a valid answer,
    but expect some resistance from the creators of ntpd, who believe that
    it is always giving the best possible estimate of the true time.

  5. Re: No libntp.so

    Bill,

    Unruh wrote:
    > martin.burnicki@meinberg.de (Martin Burnicki) writes:
    >
    >>Hello Kay,

    >
    >>Kay Hayen wrote:
    >>> Hello Martin,
    >>>
    >>> thanks for the reply, you wrote:
    >>>
    >>>>As mentioned in another reply it depends on what ntpq has to do during
    >>>>those 25 milliseconds. Please be aware that this may include DNS lookups
    >>>>which might fail, so the the call would be take until the DNS lookup
    >>>>times out.
    >>>
    >>> We are using "ntpq -n" which makes a huge difference for the peers
    >>> command and make all DNS lookups obsolete. We are also not using DNS on
    >>> these machines, but only a hosts file, so normally this couldn't play a
    >>> role.

    >
    >>OK. However, if 25 milliseconds is still too long for other task you are
    >>running you should do this asynchronously anyway.

    >
    > My suspicion is that are using those ntpq queries to try to figure out if
    > the remote machines are "on time". Ie they use ntpq to query the time on
    > the remote machine, and then read the time on their local machine to see
    > if the remote machine is out of time sync. Then 25ms would be a long time.
    > However, using ntpq to determine the time on the other machine is silly.


    In his first message Kay wrote:
    > It appears that the whole ntpq call is relatively slow when, when we do it
    > on e.g. RHEL 5.1 on modern Dual Core hardware, we get up to 25ms
    > execution time, which is very long for us.
    >
    > As we would like do other important stuff in the same process, we would
    > like to do that faster. [...]


    If I understand this correctly then he's just doing the ntpq query and some
    other processing in one loop. If the ntpq call takes 25 milliseconds to
    execute then the other processing is delayed by that amount of time.

    That's why I think the approach to run those ntpq queries in an extra
    thread/process would be better. BTW, this would also be a better approach
    if his application would just send time (client) requests to a server and
    wait for the response before continuing to do other processing.

    If this is done in an extra thread then both the query rate and the
    execution time have no effect on some other processing which needs to be
    done independently.

    If his applications does not just want to check the current time offset of a
    remote node but collect some more information then using the ntpq
    functionality is a good way to go. If the basic functionality would be in a
    shared object library then it would just be easier to use, e.g. those
    functions could just return a string buffer which could be parsed instead
    of letting ntpq print to stdout and then pipe this into your own
    application, which may introduce problems with line buffering.

    Martin
    --
    Martin Burnicki

    Meinberg Funkuhren
    Bad Pyrmont
    Germany

  6. Re: No libntp.so

    Hello Kay,

    Kay Hayen wrote:
    > Hello Martin,
    >
    >>>thanks for the reply, you wrote:
    >>>>As mentioned in another reply it depends on what ntpq has to do during
    >>>>those 25 milliseconds. Please be aware that this may include DNS lookups
    >>>>which might fail, so the the call would be take until the DNS lookup
    >>>>times out.
    >>>We are using "ntpq -n" which makes a huge difference for the peers
    >>>command and make all DNS lookups obsolete. We are also not using DNS on
    >>>these machines, but only a hosts file, so normally this couldn't play a
    >>>role.

    >>OK. However, if 25 milliseconds is still too long for other task you are
    >>running you should do this asynchronously anyway.

    >
    > Yes, I agree, and I would say it's best to break up doquery() as I saw it into
    > a part that sends and a part that processes the reply later. That form of
    > async will match our other control flows nicely and avoid the response delays
    > to punch through.


    Why splitting up doquery?

    If you would ever have to send several queries in parallel and then
    receive several replies you'd have to check which reply is associated to
    which query.

    If your programming environment supports threads then I'd suggest to
    start one thread for each server to be queried.

    In each thread you can send a request and then wait for a reply, and
    after a reply has been received or a timeout has occurred you can either
    exit that thread or just let it sleep for some minutes or whatever and
    then loop back to send the next query.

    The advantage of this is that you send the query from some free port and
    you receive the reply on the same port, so the reply is automatically
    associated to the correct request.

    >>>I have not come to fully understand where the code is that is executed
    >>>when I enter "peers", but it seemed that the encoding and decoding of
    >>>packets could be found there at least.

    >>Search the code for dopeers(), dogetpeers(), doprintpeers(). The packets
    >>returned from ntpd already contain some ASCII text with the desired data.

    >
    > Thanks for the pointers, that's great. I looked through it down to doquery and
    > well, dunno how to say that correctly, I think it's good quality code to work
    > with.
    >
    > It appeared to be quiet achievable to have a "libntpq.[a|so]" made. Note that
    > I realized, that I actually want that instead.
    >
    > One probably should have to handle error printing somewhat differently, and
    > change a few things in order to allow to split the job (optionally) into a
    > sending and receiving part, and let the drawing parts use these.
    >
    > But for the peers command example that seemed relatively straightforward stuff
    > and nothing that would harm any code, or so the hope. :-)


    Anyway, modifications to the public code base should be done in a way
    which is useful for your kind of application but also doesn't change the
    way ntpq works.

    As mentioned above, my first idea would be to do the thread handling in
    my application, and from my thread call a NTP library function which
    sends a query, receives the reply packet(s), and returns a buffer with
    the requested information to the caller.

    >>>But checking now, the RFC 2553 is not the NTP one. Would you happen to
    >>>know if the decoding/encoding of the packets are part of the library? I
    >>>could not identify it right now.

    >>This is mainly done in ntpq_subs.c which is not in the library. This is
    >>why I think it would not help you just to distribute libntp as a .so file.

    >
    > Right. Seems what I actually want is (much) of the ntpq functionality in
    > a "libntpq.[a|so]", which probably also uses "libntp.a", but that won't ever
    > matter and be visible on installations. So I probably would choose the
    > subject now as "No libntpq (yet)."


    Yep.

    >>>>IMHO it would not make much sense to provide the existing libntp as
    >>>>shared object library. There is some ongoing work which will encapsulate
    >>>>the ntpq functionality in a shared object library, so it would be easier
    >>>>to use from an own application.
    >>>Can you point me to that work? It seems like we would want to contribute
    >>>to that effort then.

    >
    > [...]
    >
    >>You should discuss this with the guy who's working on this. I'll ask him
    >>to contact you. [...]

    >
    > Yes, please do that, very welcome. We will have time available to work on
    > this, once we have reached our next milestone. And obviously we will want to
    > coordinate efforts and plans with him.


    I'm sure you have seen Heiko has already posted in the news group. The
    questions list is just a frontend to the news server.


    Martin
    --
    Martin Burnicki

    Meinberg Funkuhren
    Bad Pyrmont
    Germany

  7. Re: No libntp.so

    Martin Burnicki writes:

    >Bill,


    >Unruh wrote:
    >> martin.burnicki@meinberg.de (Martin Burnicki) writes:
    >>
    >>>Hello Kay,

    >>
    >>>Kay Hayen wrote:
    >>>> Hello Martin,
    >>>>
    >>>> thanks for the reply, you wrote:
    >>>>
    >>>>>As mentioned in another reply it depends on what ntpq has to do during
    >>>>>those 25 milliseconds. Please be aware that this may include DNS lookups
    >>>>>which might fail, so the the call would be take until the DNS lookup
    >>>>>times out.
    >>>>
    >>>> We are using "ntpq -n" which makes a huge difference for the peers
    >>>> command and make all DNS lookups obsolete. We are also not using DNS on
    >>>> these machines, but only a hosts file, so normally this couldn't play a
    >>>> role.

    >>
    >>>OK. However, if 25 milliseconds is still too long for other task you are
    >>>running you should do this asynchronously anyway.

    >>
    >> My suspicion is that are using those ntpq queries to try to figure out if
    >> the remote machines are "on time". Ie they use ntpq to query the time on
    >> the remote machine, and then read the time on their local machine to see
    >> if the remote machine is out of time sync. Then 25ms would be a long time.
    >> However, using ntpq to determine the time on the other machine is silly.


    >In his first message Kay wrote:
    >> It appears that the whole ntpq call is relatively slow when, when we do it
    >> on e.g. RHEL 5.1 on modern Dual Core hardware, we get up to 25ms
    >> execution time, which is very long for us.
    >>
    >> As we would like do other important stuff in the same process, we would
    >> like to do that faster. [...]


    >If I understand this correctly then he's just doing the ntpq query and some
    >other processing in one loop. If the ntpq call takes 25 milliseconds to
    >execute then the other processing is delayed by that amount of time.


    >That's why I think the approach to run those ntpq queries in an extra
    >thread/process would be better. BTW, this would also be a better approach
    >if his application would just send time (client) requests to a server and
    >wait for the response before continuing to do other processing.


    >If this is done in an extra thread then both the query rate and the
    >execution time have no effect on some other processing which needs to be
    >done independently.


    >If his applications does not just want to check the current time offset of a
    >remote node but collect some more information then using the ntpq
    >functionality is a good way to go. If the basic functionality would be in a
    >shared object library then it would just be easier to use, e.g. those
    >functions could just return a string buffer which could be parsed instead
    >of letting ntpq print to stdout and then pipe this into your own
    >application, which may introduce problems with line buffering.


    ntpq will not give the current time offset. It will give the offset last
    time packets were sent out and filtered which could be hours ago. There is
    not 25ms urgency. But then he has not responded as to what he needs from
    ntpq and why he needs a less than 25ms latency on that query, so anything
    we do is guessing. Modulo that your suggestion is undoubtedly better than
    what he is doing. But he seems fixated on NTP people fixing their software
    for his solution rather than finding the best solution for his problem.




  8. Re: No libntp.so

    Unruh wrote:

    > Martin Burnicki writes:
    >
    >>Bill,

    >
    >>Unruh wrote:
    >>> martin.burnicki@meinberg.de (Martin Burnicki) writes:
    >>>
    >>>>Hello Kay,
    >>>
    >>>>Kay Hayen wrote:
    >>>>> Hello Martin,
    >>>>>
    >>>>> thanks for the reply, you wrote:
    >>>>>
    >>>>>>As mentioned in another reply it depends on what ntpq has to do during
    >>>>>>those 25 milliseconds. Please be aware that this may include DNS
    >>>>>>lookups which might fail, so the the call would be take until the DNS
    >>>>>>lookup times out.
    >>>>>
    >>>>> We are using "ntpq -n" which makes a huge difference for the peers
    >>>>> command and make all DNS lookups obsolete. We are also not using DNS
    >>>>> on these machines, but only a hosts file, so normally this couldn't
    >>>>> play a role.
    >>>
    >>>>OK. However, if 25 milliseconds is still too long for other task you are
    >>>>running you should do this asynchronously anyway.
    >>>
    >>> My suspicion is that are using those ntpq queries to try to figure out
    >>> if the remote machines are "on time". Ie they use ntpq to query the time
    >>> on the remote machine, and then read the time on their local machine to
    >>> see if the remote machine is out of time sync. Then 25ms would be a long
    >>> time. However, using ntpq to determine the time on the other machine is
    >>> silly.

    >
    >>In his first message Kay wrote:
    >>> It appears that the whole ntpq call is relatively slow when, when we do
    >>> it on e.g. RHEL 5.1 on modern Dual Core hardware, we get up to 25ms
    >>> execution time, which is very long for us.
    >>>
    >>> As we would like do other important stuff in the same process, we would
    >>> like to do that faster. [...]

    >
    >>If I understand this correctly then he's just doing the ntpq query and
    >>some other processing in one loop. If the ntpq call takes 25 milliseconds
    >>to execute then the other processing is delayed by that amount of time.

    >
    >>That's why I think the approach to run those ntpq queries in an extra
    >>thread/process would be better. BTW, this would also be a better approach
    >>if his application would just send time (client) requests to a server and
    >>wait for the response before continuing to do other processing.

    >
    >>If this is done in an extra thread then both the query rate and the
    >>execution time have no effect on some other processing which needs to be
    >>done independently.

    >
    >>If his applications does not just want to check the current time offset of
    >>a remote node but collect some more information then using the ntpq
    >>functionality is a good way to go. If the basic functionality would be in
    >>a shared object library then it would just be easier to use, e.g. those
    >>functions could just return a string buffer which could be parsed instead
    >>of letting ntpq print to stdout and then pipe this into your own
    >>application, which may introduce problems with line buffering.

    >
    > ntpq will not give the current time offset. It will give the offset last
    > time packets were sent out and filtered which could be hours ago.


    I'm aware of this. In the text you quoted from my last post above I wrote:
    "If his applications does **not** just want to check the current time offset
    of a remote node but collect some more information ..."

    > There is
    > not 25ms urgency.


    Not from the point of view for ntpq, but for other tasks that the OP was
    going to run in the same program loop as the blocking ntpq query. I've
    already pointed out above that this is no good programming technique.

    > But then he has not responded as to what he needs from
    > ntpq and why he needs a less than 25ms latency on that query, so anything
    > we do is guessing.


    Basically it's his own decision which information he wants to collect from
    ntpd using ntpq functionality. As I've already mentioned earlier our NTP
    time server monitor program for Windows contains similar functionality.

    When I came in touch with NTP I first learned that you have to look at the
    output of "ntpq -p" if there's a '*' at the beginning of one line the NTP
    daemon uses that time source as system peer and claims to be synchronized.

    Some time later I learned that ntpd may not even claim to be synchronized if
    the '*' is available, instead you have to look for "state=4" in the output
    of "ntpq -c rv". Now in the current ntp-dev version the state variable has
    been removed so you have to find another way to check whether ntpd is
    synchronized or not.

    > Modulo that your suggestion is undoubtedly better than
    > what he is doing. But he seems fixated on NTP people fixing their software
    > for his solution rather than finding the best solution for his problem.


    I think this is just a little bit of misunderstanding because he's not yet
    too familiar with the way NTP works. We can just point him in the right
    direction.

    Martin
    --
    Martin Burnicki

    Meinberg Funkuhren
    Bad Pyrmont
    Germany

  9. Re: No libntp.so

    Kay Hayen wrote:
    > Hello NTP-World :-),
    >
    > we are implementing a NTP supervision for our ATC middleware. Initially we are
    > doing it by repeated "ntpq" executions and textual evaluations of the
    > results. We have had to notice that pipes are very unsuited for the task, so
    > we really fork it and close its stdin to make it flush. It works, but it is
    > unconvincing in performance (latency).
    >
    > It appears that the whole ntpq call is relatively slow when, when we do it on
    > e.g. RHEL 5.1 on modern Dual Core hardware, we get up to 25ms execution time,
    > which is very long for us.
    >
    > As we would like do other important stuff in the same process, we would like
    > to do that faster. So I had a look at the source and found that ntpq is using
    > a libntp.a that nobody seems to package though. There is no libntp.so either,
    > we had searched for those initially, but no luck.
    >
    > Our aims can be realized (and must be short term it seems) with a fork of your
    > work that builds a shared library for use in our software. While that is
    > workable, it is also bad thing to do in the first place.
    >
    > What we really would prefer instead, is to see is the addition of a target to
    > your makefile to make a libntp.so buildable. Then, we could already use the
    > original releases as from you. From there we would work with downstreams
    > (Debian and RHEL) to convince them to include libntp.so in the NTP packages
    > and (libntp.a and headers obviously) possible through some ntp-devel package,
    > at which point we could drop building from source again.
    >
    > The question of course would be: Will you merge such Makefile patches from us
    > in the first place. And do you see any reason why this should not be done
    > like this. After all, it has not been done for a long time already, so
    > possibly this is on purpose?
    >
    > Obviously we hope you will allow us to be good Free Software citizens and let
    > us drop the fork down the road. Will you?
    >
    > Thanks in advance,
    > Kay Hayen


    My apologies for my late response. I've been urgently dealing with some
    unrelated issues elsewhere and I am now more than a month behind on my
    email.

    I would strongly support this. One of the items on my list of things to
    do with NTP is to clean it up and organize it better. There's a lot of
    code that should be shared as far as possible that is not currently
    being shared between applications.

    Makefile changes are of course encouraged as long as the changes work
    across a wide range of O/S's and compilers. Harlan has the final say on
    this since that is one of his principal responsibilities and he has to
    be able to maintain it.

    Danny

  10. Re: No libntp.so

    Martin Burnicki wrote:
    > Kay,
    >
    > Kay Hayen wrote:
    >> Hello NTP-World :-),
    >>
    >> we are implementing a NTP supervision for our ATC middleware. Initially we
    >> are doing it by repeated "ntpq" executions and textual evaluations of the
    >> results. We have had to notice that pipes are very unsuited for the task,
    >> so we really fork it and close its stdin to make it flush. It works, but
    >> it is unconvincing in performance (latency).
    >>
    >> It appears that the whole ntpq call is relatively slow when, when we do it
    >> on e.g. RHEL 5.1 on modern Dual Core hardware, we get up to 25ms execution
    >> time, which is very long for us.

    >
    > As mentioned in another reply it depends on what ntpq has to do during those
    > 25 milliseconds. Please be aware that this may include DNS lookups which
    > might fail, so the the call would be take until the DNS lookup times out.
    >
    > A proper approch would be to start a thread or task which would do the ntpq
    > stuff asynchrounously and report the results to your main application when
    > the reply has become available.
    >


    Well any application that depends on using ntpq or any of its
    functionality is broken by definition unless it's used for monitoring
    purposes.

    >> As we would like do other important stuff in the same process, we would
    >> like to do that faster. So I had a look at the source and found that ntpq
    >> is using a libntp.a that nobody seems to package though. There is no
    >> libntp.so either, we had searched for those initially, but no luck.

    >


    Libntp is not intended for such usage. It's intended just to provide
    common functions across a number of applications in the ntp suite.

    > Libntp just contains a couple of functions which are shared between several
    > programs of the NTP package, e.g. ntpd, ntpq and others. It is used during
    > the build process of the package in order to save compiule time.
    >


    That's probably not an issue any more. It was when we had to build on
    Ultrix on the flock but most modern hardware it's not really an issue.

    >> Obviously we hope you will allow us to be good Free Software citizens and
    >> let us drop the fork down the road. Will you?

    >
    > The question is not basically whether a patch is accepted. In the first
    > place the question is whether a patch makes sense at all. As already
    > mentioned above I think it does not much sense just to provide libntp
    > as .so file.
    >
    > Martin


    Whether or not we build a libntp.so or libntp.a is not as important as
    it is to move more of the common functionality into the libntp library.

    Danny

  11. Re: No libntp.so

    Kay Hayen wrote:
    >> Please tell us what you are trying to do. You are telling us your solution
    >> ( which sounds really bad) instead of the problem you are trying to solve,
    >> and then trying to convince the whole communtity to impliment your
    >> solution.

    >
    > Uhm, well you are right.
    >
    > We need to assess NTP status information. We consider the details of every
    > peer of the system, reachability, jitter, etc.
    >
    > Ideally as it occurs with no delays. Ideally we would be able to poll() some
    > file descriptor whenever something has changed for an ntpd. I don't think
    > such an interface exists. Short of that we would check it with reasonable
    > frequency.
    >
    > We need to do it for many hosts to calculate a "CLOCK" status for the host,
    > and the clock status would e.g. be bad if an ntpd is synchronized to an
    > internal host, but not one the allowed external hosts. There is a set of
    > criterias subject to user configuration.
    >
    > We need to be able to actively restrict NTP servers that are out bounds with
    > offsets. Servers that e.g. mishandle a leap second and are 1 second off from
    > that time on, or have other malfunctions, will be disabled that way.
    >
    > We need to publish the information for all hosts via SNMP.
    >


    Heiko is in the process of implementing an SNMP MIB for NTP so you
    should just be able to leverage this when it's complete. Meinberg also
    has a monitoring application that you might want to consider.

    Danny

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2