pingtimes and anomaly detection - TCP-IP

This is a discussion on pingtimes and anomaly detection - TCP-IP ; Hi people ! For my diploma project i have a little partial problem. I want to know the quality of network between 2 points in internet. There is a constant udp traffic in both directions between this 2 points. (yes ...

+ Reply to Thread
Results 1 to 11 of 11

Thread: pingtimes and anomaly detection

  1. pingtimes and anomaly detection

    Hi people !

    For my diploma project i have a little partial problem. I want to know
    the quality of network between 2 points in internet. There is a
    constant udp traffic in both directions between this 2 points. (yes i
    know tcp-ip forum )
    The problem is that i have to monitor the packetloss, sometimes if we
    exceed the the available bandwidth, we have a drastic packet loss. Now
    the idea is to monitor the packet loss. A side effect is , that the
    ping times between both points rises if the bandwith is exceeded. So
    my idea was to monitor the ping times and if they rise drastic we
    simply lower the packets to sent.(audio and video data)
    My problem now is how to get rid of the anomalys in the ping time
    measurement. If we look at the link belows we can see a little graph
    with some ping times.
    http://www.manitoba-team.de/content/...pingzeiten.gif

    My first idea was to use some mathematical algorithm like the grubbs,
    dixon or nalimov test. I testet this algorithm but in my thought they
    are not useful for my requirement. They cant work because my ping
    times are not allocated like under a gauss bell. So i had to make it
    in another way. Because i dont found any other solutions i had this
    ideas below.

    The second idea: make classes of ping times. For example
    Class 0-50ms, Class 50-100ms, Class 100-200ms, Class200-400ms, Class
    400-800ms and Class >800ms
    If we now see that one or more Classes includes fewer than 10% of
    total measurement count, we can say: ok that class is full of
    anomalies. But this is not right in all cases. Maybe there are 10
    values straight then this is no abnormaly.

    So my third idea was to get a raster above this graph and count all
    values in each square. If some squares have nearly no values included
    i can say this are abnormalys.

    My question now is if there is another algorithmus. Some notes or
    ideas from you would be great !

    Some infos maybe useful
    we only want to use udp so we have to make the control connection over
    this udp stream. it works , but we have a drastic problem if we exceed
    the bandwidth. Then the control connection breaks and the connection
    is over. And Yes we have implemented resends and so, but if we exceed
    the bandwith we have to lower the audio and video bandwidth to keep
    the control connection.

  2. Re: pingtimes and anomaly detection

    On May 6, 7:23 am, rupat wrote:

    > For my diploma project i have a little partial problem. I want to know
    > the quality of network between 2 points in internet. There is a
    > constant udp traffic in both directions between this 2 points. (yes i
    > know tcp-ip forum )
    > The problem is that i have to monitor the packetloss, sometimes if we
    > exceed the the available bandwidth, we have a drastic packet loss. Now
    > the idea is to monitor the packet loss. A side effect is , that the
    > ping times between both points rises if the bandwith is exceeded. So
    > my idea was to monitor the ping times and if they rise drastic we
    > simply lower the packets to sent.(audio and video data)
    > My problem now is how to get rid of the anomalys in the ping time
    > measurement. If we look at the link belows we can see a little graph
    > with some ping times.http://www.manitoba-team.de/content/...pingzeiten.gif

    [snip]
    > we only want to use udp so we have to make the control connection over
    > this udp stream. it works , but we have a drastic problem if we exceed
    > the bandwidth. Then the control connection breaks and the connection
    > is over. And Yes we have implemented resends and so, but if we exceed
    > the bandwith we have to lower the audio and video bandwidth to keep
    > the control connection.


    This is precisely what TCP was designed to do -- dynamically adapt the
    data transfer rate to the available bandwidth. If you can't use TCP
    directly, you can at least take advantage of all the research that's
    been done on it. You can implement slow start, RTT measurement,
    delayed ACK, selective acknowledge, and so on. There's tons and tons
    of research and documentation on all of these methods and how well
    they work.

    It sounds like you're trying to reinvent TCP. That may be a mistake or
    there may be a good reason. But if you need TCP's transmit pacing,
    exponential backoff, and rate adaptation, you need to implement them.

    There are also other algorithms out there that provide these kinds of
    features. XTP, for example. I would advise against having two separate
    mechanisms unless there's some reason you really need that. Better to
    have the transmission protocol itself handle the rate adaptation, as
    TCP does, rather than trying to use some separate mechanism. And
    *definitely* don't use 'ping'.

    DS

  3. Re: pingtimes and anomaly detection

    On Wed, 07 May 2008 01:42:48 -0700, David Schwartz wrote:

    > There are also other algorithms out there that provide these kinds of
    > features. XTP, for example. I would advise against having two separate
    > mechanisms unless there's some reason you really need that. Better to
    > have the transmission protocol itself handle the rate adaptation, as TCP
    > does, rather than trying to use some separate mechanism. And
    > *definitely* don't use 'ping'.


    Just to expand on that and to repeat that, *definitely* don't use ping.

    - Many OSses give ping a low priority
    - Many networks use QoS and give ping a low priority (or in our case, a
    high priority)
    - Pings may take a completely different network path.

    I'm sure there are more reasons, but these should be enough to get
    thinking.

    We *do* use pings as a measurement tool. We have reasons that the above
    does not apply to our situation. Even then it is just a cheap way of
    estimating RTT and packet loss. As we don't need reliable measurements,
    but just an alert if RTT or packet loss is consistently high, it suffices
    for us. It will not suffice for your application, I can tell from
    experience it is not reliable, except in specific circumstances, and even
    then only for the big picture.

    M4

  4. Re: pingtimes and anomaly detection

    On 7 Mai, 11:43, Martijn Lievaart wrote:
    > On Wed, 07 May 2008 01:42:48 -0700, David Schwartz wrote:
    > > There are also other algorithms out there that provide these kinds of
    > > features. XTP, for example. I would advise against having two separate
    > > mechanisms unless there's some reason you really need that. Better to
    > > have the transmission protocol itself handle the rate adaptation, as TCP
    > > does, rather than trying to use some separate mechanism. And
    > > *definitely* don't use 'ping'.


    first of all we do not want to use ping we can take the "ping-
    times" from the responses of our packets from the control connection.


    >
    > Just to expand on that and to repeat that, *definitely* don't use ping.
    >
    > - Many OSses give ping a low priority
    > - Many networks use QoS and give ping a low priority (or in our case, a
    > high priority)
    > - Pings may take a completely different network path.
    >
    > I'm sure there are more reasons, but these should be enough to get
    > thinking.
    >
    > We *do* use pings as a measurement tool. We have reasons that the above
    > does not apply to our situation. Even then it is just a cheap way of
    > estimating RTT and packet loss. As we don't need reliable measurements,
    > but just an alert if RTT or packet loss is consistently high, it suffices
    > for us. It will not suffice for your application, I can tell from
    > experience it is not reliable, except in specific circumstances, and even
    > then only for the big picture.
    >
    > M4


    thats all facts i know and its allways boring to tell people that
    ping traffic is not the same lile realistic application traffic
    (different bandwith, packet size ....)

  5. Re: pingtimes and anomaly detection

    On 7 Mai, 10:42, David Schwartz wrote:
    > On May 6, 7:23 am, rupat wrote:
    >
    >
    >
    >
    >
    > > For my diploma project i have a little partial problem. I want to know
    > > the quality of network between 2 points in internet. There is a
    > > constant udp traffic in both directions between this 2 points. (yes i
    > > know tcp-ip forum )
    > > The problem is that i have to monitor the packetloss, sometimes if we
    > > exceed the the available bandwidth, we have a drastic packet loss. Now
    > > the idea is to monitor the packet loss. A side effect is , that the
    > > ping times between both points rises if the bandwith is exceeded. So
    > > my idea was to monitor the ping times and if they rise drastic we
    > > simply lower the packets to sent.(audio and video data)
    > > My problem now is how to get rid of the anomalys in the ping time
    > > measurement. If we look at the link belows we can see a little graph
    > > with some ping times.http://www.manitoba-team.de/content/...pingzeiten.gif

    > [snip]
    > > we only want to use udp so we have to make the control connection over
    > > this udp stream. it works , but we have a drastic problem if we exceed
    > > the bandwidth. Then the control connection breaks and the connection
    > > is over. And Yes we have implemented resends and so, but if we exceed
    > > the bandwith we have to lower the audio and video bandwidth to keep
    > > the control connection.

    >
    > This is precisely what TCP was designed to do -- dynamically adapt the
    > data transfer rate to the available bandwidth. If you can't use TCP
    > directly, you can at least take advantage of all the research that's
    > been done on it. You can implement slow start, RTT measurement,
    > delayed ACK, selective acknowledge, and so on. There's tons and tons
    > of research and documentation on all of these methods and how well
    > they work.
    >
    > It sounds like you're trying to reinvent TCP. That may be a mistake or
    > there may be a good reason. But if you need TCP's transmit pacing,
    > exponential backoff, and rate adaptation, you need to implement them.
    >

    Yes you´re right, but we have the problem that we have 2 different
    communications in one udp connection. We have traffic which is not so
    important (audio and video). Its no problem if we have a packetloss of
    30% or more. But on the same udp connection we have to transmit
    absolutly safe our control connection protocol. I think best way would
    be to implement something like that:

    if The Sender do not receive responses on the control messages he
    resends for some times his messages
    if still no responses arrives him he lowers the audio and video
    bandwith and resends again the control messages
    if messages responses are coming he can try to adapt the bandwith
    again in a defined time intervall. its like tcp
    if still no control message responses arrives we have to cut the
    connection.

    The main difference is that we do not react on packet loss of the
    audio and video data. its not important to get 100% of this packets
    through.

    Anonther problem is that the response messages cant receive because of
    overriding the upload bandwith on the receivers side, so the response
    packets gets dropped on the receivers side .
    Maybe they both have to fall down to a minimum of bandwith and then
    adapt in different times their upload so they know that the packetloss
    is the problem its own side and they have reached the maximum
    bandwith. ?

    Any Ideas are Welcome !

    And thanks to your help


    > There are also other algorithms out there that provide these kinds of
    > features. XTP, for example. I would advise against having two separate
    > mechanisms unless there's some reason you really need that. Better to
    > have the transmission protocol itself handle the rate adaptation, as
    > TCP does, rather than trying to use some separate mechanism. And
    > *definitely* don't use 'ping'.
    >
    > DS



  6. Re: pingtimes and anomaly detection

    "David Schwartz" wrote in message
    news:d8605ba1-cc0e-4bcf-9dfc-e691bbdd5629@x19g2000prg.googlegroups.com...
    > On May 6, 7:23 am, rupat wrote:
    >
    > > For my diploma project i have a little partial problem. I want to know
    > > the quality of network between 2 points in internet. There is a
    > > constant udp traffic in both directions between this 2 points. (yes i
    > > know tcp-ip forum )
    > > The problem is that i have to monitor the packetloss, sometimes if we
    > > exceed the the available bandwidth, we have a drastic packet loss. Now
    > > the idea is to monitor the packet loss. A side effect is , that the
    > > ping times between both points rises if the bandwith is exceeded. So
    > > my idea was to monitor the ping times and if they rise drastic we
    > > simply lower the packets to sent.(audio and video data)
    > > My problem now is how to get rid of the anomalys in the ping time
    > > measurement. If we look at the link belows we can see a little graph
    > > with some ping

    times.http://www.manitoba-team.de/content/...pingzeiten.gif
    > [snip]
    > > we only want to use udp so we have to make the control connection over
    > > this udp stream. it works , but we have a drastic problem if we exceed
    > > the bandwidth. Then the control connection breaks and the connection
    > > is over. And Yes we have implemented resends and so, but if we exceed
    > > the bandwith we have to lower the audio and video bandwidth to keep
    > > the control connection.

    >
    > This is precisely what TCP was designed to do -- dynamically adapt the
    > data transfer rate to the available bandwidth. If you can't use TCP
    > directly, you can at least take advantage of all the research that's
    > been done on it. You can implement slow start, RTT measurement,
    > delayed ACK, selective acknowledge, and so on. There's tons and tons
    > of research and documentation on all of these methods and how well
    > they work.


    if you are using UDP are you also using RTP?

    if so the associated RTCP control session will already be reporting what you
    want.
    >
    > It sounds like you're trying to reinvent TCP. That may be a mistake or
    > there may be a good reason. But if you need TCP's transmit pacing,
    > exponential backoff, and rate adaptation, you need to implement them.
    >
    > There are also other algorithms out there that provide these kinds of
    > features. XTP, for example. I would advise against having two separate
    > mechanisms unless there's some reason you really need that. Better to
    > have the transmission protocol itself handle the rate adaptation, as
    > TCP does, rather than trying to use some separate mechanism. And
    > *definitely* don't use 'ping'.
    >
    > DS

    --
    Regards

    stephen_hope@xyzworld.com - replace xyz with ntl



  7. Re: pingtimes and anomaly detection

    On 2008-05-07 05:43:00 -0400, Martijn Lievaart said:

    > - Many OSses give ping a low priority


    Martin, thanks so much for bringing this up - this is my favorite grind
    when people bring up ICMP-based ping measurements. I'll add $0.02 here
    in that all of my practical experience in large network enterprises has
    shown that ICMP does get a low priority, plurally across most OS's,
    even RTOS's like Cisco IOS, etc., etc. and can't be relied on for
    accurate measurement.

    Also, using a single point of measurement for RTT assumes that every
    packet is taking .5*RTT to get to a destination and back - this is
    never universally true, and you can and will get one-way latencies that
    are far from 50/50. This is one of the most interesting areas of
    network traffic research today.

    /dmfh

    --
    _ __ _
    __| |_ __ / _| |_ 01100100 01101101
    / _` | ' \| _| ' \ 01100110 01101000
    \__,_|_|_|_|_| |_||_| dmfh(-2)dmfh.cx


  8. Re: pingtimes and anomaly detection

    On Fri, 09 May 2008 23:35:45 -0400, Digital Mercenary For Honor wrote:

    > On 2008-05-07 05:43:00 -0400, Martijn Lievaart
    > said:
    >
    >> - Many OSses give ping a low priority

    >
    > Martin, thanks so much for bringing this up - this is my favorite grind


    That's Martijn, it's a Dutch name.

    > when people bring up ICMP-based ping measurements. I'll add $0.02 here
    > in that all of my practical experience in large network enterprises has
    > shown that ICMP does get a low priority, plurally across most OS's, even
    > RTOS's like Cisco IOS, etc., etc. and can't be relied on for accurate
    > measurement.


    While it's not what the OP asked, I'll repeat it here yet again, so
    posterity will know for once and for all.

    Especially Ciscos (and other network gear?) give pings a very low
    priority, and I've seen the same on Windows, but less on Unixy machines.

    Many SOHO routers have some form of QoS build in, throwing your
    measurements way of from what you expect to measure.

    You can use ping as a indication of trouble, if, and only if you know
    your network treats ping exactly the same as the traffic you're
    interested in, and if, and only if, you know how your network reacts when
    no problems exist.

    Packet loss is especially telling, if you lose pings, your application
    will also see packet loss. Theoretically the pings could be dropped while
    your traffic does get handled, but I have yet to see it. Note that some
    packet loss is normal, if you lose one ping in a row, frequently (even
    every few seconds), there's probably nothing to worry about. If you see
    multiple pings lost in a row, you more often than not have a problem.

    High RTTs indicate that something is highly loaded. Which may be fine, or
    it may not be. Some router may be heavily loaded, but still functioning
    fine.

    In my experience, on a network I manage, that intentionally treats ping
    exactly the same as Citrix traffic, ping times are a very good indication
    of the performance the users perceive. But this was especially set up to
    behave this way and is not universally true. And we know the false
    positives.

    Bottom line, use can use ping as a diagnostic tool, but only if you know
    what you are looking at.

    >
    > Also, using a single point of measurement for RTT assumes that every
    > packet is taking .5*RTT to get to a destination and back - this is never
    > universally true, and you can and will get one-way latencies that are
    > far from 50/50. This is one of the most interesting areas of network
    > traffic research today.


    Actually, on a not too large WAN, packets tend to take the same routes,
    or the differences are insignificant. But different return paths is
    normal for the Internet and must always be assumed for larger networks
    unless known to be untrue.

    On smaller networks, it can occur, but is more seldom and more often than
    not an indication of a misconfiguration.

    M4

  9. Re: pingtimes and anomaly detection

    Digital Mercenary For Honor wrote:
    > On 2008-05-07 05:43:00 -0400, Martijn Lievaart said:


    > > - Many OSses give ping a low priority


    > Martin, thanks so much for bringing this up - this is my favorite
    > grind when people bring up ICMP-based ping measurements. I'll add
    > $0.02 here in that all of my practical experience in large network
    > enterprises has shown that ICMP does get a low priority, plurally
    > across most OS's, even RTOS's like Cisco IOS, etc., etc. and can't
    > be relied on for accurate measurement.


    Well, it is an accurate measurement - of ICMP round-trip latency. It
    just isn't an accurate estimation/simulation of RTT for something
    else. Would it not though be a _conservative_ estimation of the RTT?

    Heck, even without being consigned to a lower priority in the cloud,
    an ICMP echo isn't going to be an accurate estimate of
    application-layer RTT anyway since those have processing times ping
    will never have. (And to be complete, neither will netperf TCP_RR
    although it won't ostensibly have the other issues of ping.)

    rick jones
    --
    oxymoron n, Hummer H2 with California Save Our Coasts and Oceans plates
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  10. Re: pingtimes and anomaly detection

    Rick Jones wrote:

    > Well, it is an accurate measurement - of ICMP round-trip latency. It
    > just isn't an accurate estimation/simulation of RTT for something
    > else. Would it not though be a _conservative_ estimation of the RTT?


    The theoretical answer is that ICMP might take a different path
    entirely. For example, a site might have one central machine near
    their network edge that responds to all ICMP echo requests while the
    link from their network edge to the site where the machine you are
    pinging is might be down or overloaded. (This doesn't sound that
    likely, but it's one of the reasons you can't rely on it in general.)

    > Heck, even without being consigned to a lower priority in the cloud,
    > an ICMP echo isn't going to be an accurate estimate of
    > application-layer RTT anyway since those have processing times ping
    > will never have. (And to be complete, neither will netperf TCP_RR
    > although it won't ostensibly have the other issues of ping.)


    For many UNIX machines, the time from an inbound ICMP echo request to
    an outbound echo reply might be much lower than anything else, because
    ICMP replies are generated entirely in the kernel.

    DS

  11. Re: pingtimes and anomaly detection

    On Mon, 12 May 2008 17:20:47 -0700, David Schwartz wrote:

    > For many UNIX machines, the time from an inbound ICMP echo request to an
    > outbound echo reply might be much lower than anything else, because ICMP
    > replies are generated entirely in the kernel.


    Enter hping2 :-)

    M4


+ Reply to Thread