pingtimes and anomaly detection

Hi people !

For my diploma project i have a little partial problem. I want to know
the quality of network between 2 points in internet. There is a
constant udp traffic in both directions between this 2 points. (yes i
know tcp-ip forum ;))
The problem is that i have to monitor the packetloss, sometimes if we
exceed the the available bandwidth, we have a drastic packet loss. Now
the idea is to monitor the packet loss. A side effect is , that the
ping times between both points rises if the bandwith is exceeded. So
my idea was to monitor the ping times and if they rise drastic we
simply lower the packets to sent.(audio and video data)
My problem now is how to get rid of the anomalys in the ping time
measurement. If we look at the link belows we can see a little graph
with some ping times.
[url]http://www.manitoba-team.de/content/uploads/bude/pingzeiten.gif[/url]

My first idea was to use some mathematical algorithm like the grubbs,
dixon or nalimov test. I testet this algorithm but in my thought they
are not useful for my requirement. They cant work because my ping
times are not allocated like under a gauss bell. So i had to make it
in another way. Because i dont found any other solutions i had this
ideas below.

The second idea: make classes of ping times. For example
Class 0-50ms, Class 50-100ms, Class 100-200ms, Class200-400ms, Class
400-800ms and Class >800ms
If we now see that one or more Classes includes fewer than 10% of
total measurement count, we can say: ok that class is full of
anomalies. But this is not right in all cases. Maybe there are 10
values straight then this is no abnormaly.

So my third idea was to get a raster above this graph and count all
values in each square. If some squares have nearly no values included
i can say this are abnormalys.

My question now is if there is another algorithmus. Some notes or
ideas from you would be great !

Some infos maybe useful
we only want to use udp so we have to make the control connection over
this udp stream. it works , but we have a drastic problem if we exceed
the bandwidth. Then the control connection breaks and the connection
is over. And Yes we have implemented resends and so, but if we exceed
the bandwith we have to lower the audio and video bandwidth to keep
the control connection.

On May 6, 7:23 am, rupat <r.b...@gmx.de> wrote:
[color=blue]
> For my diploma project i have a little partial problem. I want to know
> the quality of network between 2 points in internet. There is a
> constant udp traffic in both directions between this 2 points. (yes i
> know tcp-ip forum ;))
> The problem is that i have to monitor the packetloss, sometimes if we
> exceed the the available bandwidth, we have a drastic packet loss. Now
> the idea is to monitor the packet loss. A side effect is , that the
> ping times between both points rises if the bandwith is exceeded. So
> my idea was to monitor the ping times and if they rise drastic we
> simply lower the packets to sent.(audio and video data)
> My problem now is how to get rid of the anomalys in the ping time
> measurement. If we look at the link belows we can see a little graph
> with some ping times.[url]http://www.manitoba-team.de/content/uploads/bude/pingzeiten.gif[/url][/color]
[snip][color=blue]
> we only want to use udp so we have to make the control connection over
> this udp stream. it works , but we have a drastic problem if we exceed
> the bandwidth. Then the control connection breaks and the connection
> is over. And Yes we have implemented resends and so, but if we exceed
> the bandwith we have to lower the audio and video bandwidth to keep
> the control connection.[/color]

This is precisely what TCP was designed to do -- dynamically adapt the
data transfer rate to the available bandwidth. If you can't use TCP
directly, you can at least take advantage of all the research that's
been done on it. You can implement slow start, RTT measurement,
delayed ACK, selective acknowledge, and so on. There's tons and tons
of research and documentation on all of these methods and how well
they work.

It sounds like you're trying to reinvent TCP. That may be a mistake or
there may be a good reason. But if you need TCP's transmit pacing,
exponential backoff, and rate adaptation, you need to implement them.

There are also other algorithms out there that provide these kinds of
features. XTP, for example. I would advise against having two separate
mechanisms unless there's some reason you really need that. Better to
have the transmission protocol itself handle the rate adaptation, as
TCP does, rather than trying to use some separate mechanism. And
*definitely* don't use 'ping'.

DS

On Wed, 07 May 2008 01:42:48 -0700, David Schwartz wrote:
[color=blue]
> There are also other algorithms out there that provide these kinds of
> features. XTP, for example. I would advise against having two separate
> mechanisms unless there's some reason you really need that. Better to
> have the transmission protocol itself handle the rate adaptation, as TCP
> does, rather than trying to use some separate mechanism. And
> *definitely* don't use 'ping'.[/color]

Just to expand on that and to repeat that, *definitely* don't use ping.

- Many OSses give ping a low priority
- Many networks use QoS and give ping a low priority (or in our case, a
high priority)
- Pings may take a completely different network path.

I'm sure there are more reasons, but these should be enough to get
thinking.

We *do* use pings as a measurement tool. We have reasons that the above
does not apply to our situation. Even then it is just a cheap way of
estimating RTT and packet loss. As we don't need reliable measurements,
but just an alert if RTT or packet loss is consistently high, it suffices
for us. It will not suffice for your application, I can tell from
experience it is not reliable, except in specific circumstances, and even
then only for the big picture.

M4

On 7 Mai, 11:43, Martijn Lievaart <m...@rtij.nl.invlalid> wrote:[color=blue]
> On Wed, 07 May 2008 01:42:48 -0700, David Schwartz wrote:[color=green]
> > There are also other algorithms out there that provide these kinds of
> > features. XTP, for example. I would advise against having two separate
> > mechanisms unless there's some reason you really need that. Better to
> > have the transmission protocol itself handle the rate adaptation, as TCP
> > does, rather than trying to use some separate mechanism. And
> > *definitely* don't use 'ping'.[/color][/color]

first of all we do not want to use ping ;) we can take the "ping-
times" from the responses of our packets from the control connection.

[color=blue]
>
> Just to expand on that and to repeat that, *definitely* don't use ping.
>
> - Many OSses give ping a low priority
> - Many networks use QoS and give ping a low priority (or in our case, a
> high priority)
> - Pings may take a completely different network path.
>
> I'm sure there are more reasons, but these should be enough to get
> thinking.
>
> We *do* use pings as a measurement tool. We have reasons that the above
> does not apply to our situation. Even then it is just a cheap way of
> estimating RTT and packet loss. As we don't need reliable measurements,
> but just an alert if RTT or packet loss is consistently high, it suffices
> for us. It will not suffice for your application, I can tell from
> experience it is not reliable, except in specific circumstances, and even
> then only for the big picture.
>
> M4[/color]

thats all facts i know ;) and its allways boring to tell people that
ping traffic is not the same lile realistic application traffic
(different bandwith, packet size ....)

On 7 Mai, 10:42, David Schwartz <dav...@webmaster.com> wrote:[color=blue]
> On May 6, 7:23 am, rupat <r.b...@gmx.de> wrote:
>
>
>
>
>[color=green]
> > For my diploma project i have a little partial problem. I want to know
> > the quality of network between 2 points in internet. There is a
> > constant udp traffic in both directions between this 2 points. (yes i
> > know tcp-ip forum ;))
> > The problem is that i have to monitor the packetloss, sometimes if we
> > exceed the the available bandwidth, we have a drastic packet loss. Now
> > the idea is to monitor the packet loss. A side effect is , that the
> > ping times between both points rises if the bandwith is exceeded. So
> > my idea was to monitor the ping times and if they rise drastic we
> > simply lower the packets to sent.(audio and video data)
> > My problem now is how to get rid of the anomalys in the ping time
> > measurement. If we look at the link belows we can see a little graph
> > with some ping times.[url]http://www.manitoba-team.de/content/uploads/bude/pingzeiten.gif[/url][/color]
> [snip][color=green]
> > we only want to use udp so we have to make the control connection over
> > this udp stream. it works , but we have a drastic problem if we exceed
> > the bandwidth. Then the control connection breaks and the connection
> > is over. And Yes we have implemented resends and so, but if we exceed
> > the bandwith we have to lower the audio and video bandwidth to keep
> > the control connection.[/color]
>
> This is precisely what TCP was designed to do -- dynamically adapt the
> data transfer rate to the available bandwidth. If you can't use TCP
> directly, you can at least take advantage of all the research that's
> been done on it. You can implement slow start, RTT measurement,
> delayed ACK, selective acknowledge, and so on. There's tons and tons
> of research and documentation on all of these methods and how well
> they work.
>
> It sounds like you're trying to reinvent TCP. That may be a mistake or
> there may be a good reason. But if you need TCP's transmit pacing,
> exponential backoff, and rate adaptation, you need to implement them.
>[/color]
Yes you´re right, but we have the problem that we have 2 different
communications in one udp connection. We have traffic which is not so
important (audio and video). Its no problem if we have a packetloss of
30% or more. But on the same udp connection we have to transmit
absolutly safe our control connection protocol. I think best way would
be to implement something like that:

if The Sender do not receive responses on the control messages he
resends for some times his messages
if still no responses arrives him he lowers the audio and video
bandwith and resends again the control messages
if messages responses are coming he can try to adapt the bandwith
again in a defined time intervall. its like tcp
if still no control message responses arrives we have to cut the
connection.

The main difference is that we do not react on packet loss of the
audio and video data. its not important to get 100% of this packets
through.

Anonther problem is that the response messages cant receive because of
overriding the upload bandwith on the receivers side, so the response
packets gets dropped on the receivers side .
Maybe they both have to fall down to a minimum of bandwith and then
adapt in different times their upload so they know that the packetloss
is the problem its own side and they have reached the maximum
bandwith. ?

Any Ideas are Welcome !

And thanks to your help

[color=blue]
> There are also other algorithms out there that provide these kinds of
> features. XTP, for example. I would advise against having two separate
> mechanisms unless there's some reason you really need that. Better to
> have the transmission protocol itself handle the rate adaptation, as
> TCP does, rather than trying to use some separate mechanism. And
> *definitely* don't use 'ping'.
>
> DS[/color]

"David Schwartz" <davids@webmaster.com> wrote in message
news:d8605ba1-cc0e-4bcf-9dfc-e691bbdd5629@x19g2000prg.googlegroups.com...[color=blue]
> On May 6, 7:23 am, rupat <r.b...@gmx.de> wrote:
>[color=green]
> > For my diploma project i have a little partial problem. I want to know
> > the quality of network between 2 points in internet. There is a
> > constant udp traffic in both directions between this 2 points. (yes i
> > know tcp-ip forum ;))
> > The problem is that i have to monitor the packetloss, sometimes if we
> > exceed the the available bandwidth, we have a drastic packet loss. Now
> > the idea is to monitor the packet loss. A side effect is , that the
> > ping times between both points rises if the bandwith is exceeded. So
> > my idea was to monitor the ping times and if they rise drastic we
> > simply lower the packets to sent.(audio and video data)
> > My problem now is how to get rid of the anomalys in the ping time
> > measurement. If we look at the link belows we can see a little graph
> > with some ping[/color][/color]
times.[url]http://www.manitoba-team.de/content/uploads/bude/pingzeiten.gif[/url][color=blue]
> [snip][color=green]
> > we only want to use udp so we have to make the control connection over
> > this udp stream. it works , but we have a drastic problem if we exceed
> > the bandwidth. Then the control connection breaks and the connection
> > is over. And Yes we have implemented resends and so, but if we exceed
> > the bandwith we have to lower the audio and video bandwidth to keep
> > the control connection.[/color]
>
> This is precisely what TCP was designed to do -- dynamically adapt the
> data transfer rate to the available bandwidth. If you can't use TCP
> directly, you can at least take advantage of all the research that's
> been done on it. You can implement slow start, RTT measurement,
> delayed ACK, selective acknowledge, and so on. There's tons and tons
> of research and documentation on all of these methods and how well
> they work.[/color]

if you are using UDP are you also using RTP?

if so the associated RTCP control session will already be reporting what you
want.[color=blue]
>
> It sounds like you're trying to reinvent TCP. That may be a mistake or
> there may be a good reason. But if you need TCP's transmit pacing,
> exponential backoff, and rate adaptation, you need to implement them.
>
> There are also other algorithms out there that provide these kinds of
> features. XTP, for example. I would advise against having two separate
> mechanisms unless there's some reason you really need that. Better to
> have the transmission protocol itself handle the rate adaptation, as
> TCP does, rather than trying to use some separate mechanism. And
> *definitely* don't use 'ping'.
>
> DS[/color]
--
Regards

[email]stephen_hope@xyzworld.com[/email] - replace xyz with ntl

On 2008-05-07 05:43:00 -0400, Martijn Lievaart <m@rtij.nl.invlalid> said:
[color=blue]
> - Many OSses give ping a low priority[/color]

Martin, thanks so much for bringing this up - this is my favorite grind
when people bring up ICMP-based ping measurements. I'll add $0.02 here
in that all of my practical experience in large network enterprises has
shown that ICMP does get a low priority, plurally across most OS's,
even RTOS's like Cisco IOS, etc., etc. and can't be relied on for
accurate measurement.

Also, using a single point of measurement for RTT assumes that every
packet is taking .5*RTT to get to a destination and back - this is
never universally true, and you can and will get one-way latencies that
are far from 50/50. This is one of the most interesting areas of
network traffic research today.

/dmfh

--
_ __ _
__| |_ __ / _| |_ 01100100 01101101
/ _` | ' \| _| ' \ 01100110 01101000
\__,_|_|_|_|_| |_||_| dmfh(-2)dmfh.cx

On Fri, 09 May 2008 23:35:45 -0400, Digital Mercenary For Honor wrote:
[color=blue]
> On 2008-05-07 05:43:00 -0400, Martijn Lievaart <m@rtij.nl.invlalid>
> said:
>[color=green]
>> - Many OSses give ping a low priority[/color]
>
> Martin, thanks so much for bringing this up - this is my favorite grind[/color]

That's Martijn, it's a Dutch name.
[color=blue]
> when people bring up ICMP-based ping measurements. I'll add $0.02 here
> in that all of my practical experience in large network enterprises has
> shown that ICMP does get a low priority, plurally across most OS's, even
> RTOS's like Cisco IOS, etc., etc. and can't be relied on for accurate
> measurement.[/color]

While it's not what the OP asked, I'll repeat it here yet again, so
posterity will know for once and for all.

Especially Ciscos (and other network gear?) give pings a very low
priority, and I've seen the same on Windows, but less on Unixy machines.

Many SOHO routers have some form of QoS build in, throwing your
measurements way of from what you expect to measure.

You can use ping as a indication of trouble, if, and only if you know
your network treats ping exactly the same as the traffic you're
interested in, and if, and only if, you know how your network reacts when
no problems exist.

Packet loss is especially telling, if you lose pings, your application
will also see packet loss. Theoretically the pings could be dropped while
your traffic does get handled, but I have yet to see it. Note that some
packet loss is normal, if you lose one ping in a row, frequently (even
every few seconds), there's probably nothing to worry about. If you see
multiple pings lost in a row, you more often than not have a problem.

High RTTs indicate that something is highly loaded. Which may be fine, or
it may not be. Some router may be heavily loaded, but still functioning
fine.

In my experience, on a network I manage, that intentionally treats ping
exactly the same as Citrix traffic, ping times are a very good indication
of the performance the users perceive. But this was especially set up to
behave this way and is not universally true. And we know the false
positives.

Bottom line, use can use ping as a diagnostic tool, but only if you know
what you are looking at.
[color=blue]
>
> Also, using a single point of measurement for RTT assumes that every
> packet is taking .5*RTT to get to a destination and back - this is never
> universally true, and you can and will get one-way latencies that are
> far from 50/50. This is one of the most interesting areas of network
> traffic research today.[/color]

Actually, on a not too large WAN, packets tend to take the same routes,
or the differences are insignificant. But different return paths is
normal for the Internet and must always be assumed for larger networks
unless known to be untrue.

On smaller networks, it can occur, but is more seldom and more often than
not an indication of a misconfiguration.

M4

Digital Mercenary For Honor <dmfh(-2-)nospamr3m0v3th1s.dmfh.cx> wrote:[color=blue]
> On 2008-05-07 05:43:00 -0400, Martijn Lievaart <m@rtij.nl.invlalid> said:[/color]
[color=blue][color=green]
> > - Many OSses give ping a low priority[/color][/color]
[color=blue]
> Martin, thanks so much for bringing this up - this is my favorite
> grind when people bring up ICMP-based ping measurements. I'll add
> $0.02 here in that all of my practical experience in large network
> enterprises has shown that ICMP does get a low priority, plurally
> across most OS's, even RTOS's like Cisco IOS, etc., etc. and can't
> be relied on for accurate measurement.[/color]

Well, it is an accurate measurement - of ICMP round-trip latency. It
just isn't an accurate estimation/simulation of RTT for something
else. Would it not though be a _conservative_ estimation of the RTT?

Heck, even without being consigned to a lower priority in the cloud,
an ICMP echo isn't going to be an accurate estimate of
application-layer RTT anyway since those have processing times ping
will never have. (And to be complete, neither will netperf TCP_RR
although it won't ostensibly have the other issues of ping.)

rick jones
--
oxymoron n, Hummer H2 with California Save Our Coasts and Oceans plates
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Rick Jones wrote:
[color=blue]
> Well, it is an accurate measurement - of ICMP round-trip latency. It
> just isn't an accurate estimation/simulation of RTT for something
> else. Would it not though be a _conservative_ estimation of the RTT?[/color]

The theoretical answer is that ICMP might take a different path
entirely. For example, a site might have one central machine near
their network edge that responds to all ICMP echo requests while the
link from their network edge to the site where the machine you are
pinging is might be down or overloaded. (This doesn't sound that
likely, but it's one of the reasons you can't rely on it in general.)
[color=blue]
> Heck, even without being consigned to a lower priority in the cloud,
> an ICMP echo isn't going to be an accurate estimate of
> application-layer RTT anyway since those have processing times ping
> will never have. (And to be complete, neither will netperf TCP_RR
> although it won't ostensibly have the other issues of ping.)[/color]

For many UNIX machines, the time from an inbound ICMP echo request to
an outbound echo reply might be much lower than anything else, because
ICMP replies are generated entirely in the kernel.

DS

On Mon, 12 May 2008 17:20:47 -0700, David Schwartz wrote:
[color=blue]
> For many UNIX machines, the time from an inbound ICMP echo request to an
> outbound echo reply might be much lower than anything else, because ICMP
> replies are generated entirely in the kernel.[/color]

Enter hping2 :-)

M4