I have loaded router (~650 Mbps In+Out), based on 2xAMD Opteron 248, Sun Fire
X4100. HPET timer available (TSC seems not available on this platform).
Network interfaces is onboard, connected over PCI-X.

Right now i am using only one processor, cause using only one interface and
interrupts stick to it. Other is almost not used.
At peak time i notice in mpstat, that this processor is almost "dead", and if
i run minor application consuming resources - ping over this router will be
terrible. For me it is clear - system overloaded. I did oprofile, and here is
result (at low load time, but at peak time it is very similar).

CPU: AMD64 processors, speed 2193.74 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit
mask of 0x00 (No unit mask) count 100000
CPU_CLK_UNHALT...|
samples| %|
------------------
2679376 71.9851 vmlinux
287212 7.7163 e1000
278674 7.4870 ip_tables
259923 6.9832 nf_conntrack
29699 0.7979 iptable_nat
26752 0.7187 nf_nat
26093 0.7010 nf_conntrack_ipv4
16525 0.4440 iptable_mangle
14988 0.4027 oprofiled


CPU: AMD64 processors, speed 2193.74 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit
mask of 0x00 (No unit mask) count 100000
samples % symbol name
1031727 37.1736 getnstimeofday
230457 8.3035 __napi_schedule
122154 4.4013 __do_softirq
110036 3.9647 dev_queue_xmit
88800 3.1995 net_rx_action
71163 2.5640 ip_route_input
52232 1.8819 local_bh_enable
43804 1.5783 get_next_timer_interrupt
43387 1.5633 ip_forward
35501 1.2791 nf_iterate
35212 1.2687 __slab_alloc
34652 1.2485 default_idle
32375 1.1665 kfree
28127 1.0134 kmem_cache_alloc

What is bothering me, why getnstimeofday called so much? Even i remove HTB
shaper, it still takes 30-40% of whole vmlinux time. From other
applications - only zebra is running.
Any ideas?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Friday 22 August 2008, Denys Fedoryshchenko wrote:

Most significant event types where i notice getnstimeofday at top of list.

Additions:
Counted MEMORY_REQUESTS events (Memory requests by type) with a unit mask of
0x01 (Requests to non-cacheable (UC) memory) count 5000
samples % samples % symbol name
129 31.0843 596 31.1879 getnstimeofday
54 13.0120 251 13.1345 __napi_schedule
36 8.6747 178 9.3145 default_idle
34 8.1928 164 8.5819 irq_entries_start
23 5.5422 143 7.4830 __do_softirq

and
CPU: AMD64 processors, speed 2193.74 MHz (estimated)
Counted INTERRUPTS_MASKED_CYCLES events (Cycles with interrupts masked (IF=0))
with a unit mask of 0x00 (No unit mask) count 5000
samples % symbol name
630015 62.4741 getnstimeofday
28634 2.8394 get_next_timer_interrupt
23279 2.3084 __slab_alloc
15775 1.5643 schedule
14765 1.4641 __slab_free
11154 1.1061 native_read_tsc
10953 1.0861 kmem_cache_alloc
10918 1.0827 tick_nohz_stop_sched_tick
10752 1.0662 update_wall_time
10430 1.0343 net_rx_action
10220 1.0134 __do_softirq
9895 0.9812 __update_sched_clock
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On 22-08-2008 03:57, Denys Fedoryshchenko wrote:
> I have loaded router (~650 Mbps In+Out), based on 2xAMD Opteron 248, Sun Fire
> X4100. HPET timer available (TSC seems not available on this platform).
> Network interfaces is onboard, connected over PCI-X.
>
> Right now i am using only one processor, cause using only one interface and
> interrupts stick to it. Other is almost not used.
> At peak time i notice in mpstat, that this processor is almost "dead", and if
> i run minor application consuming resources - ping over this router will be
> terrible. For me it is clear - system overloaded. I did oprofile, and here is
> result (at low load time, but at peak time it is very similar).
....
> CPU: AMD64 processors, speed 2193.74 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit
> mask of 0x00 (No unit mask) count 100000
> samples % symbol name
> 1031727 37.1736 getnstimeofday
> 230457 8.3035 __napi_schedule
> 122154 4.4013 __do_softirq
> 110036 3.9647 dev_queue_xmit
....
> What is bothering me, why getnstimeofday called so much? Even i remove HTB
> shaper, it still takes 30-40% of whole vmlinux time. From other
> applications - only zebra is running.
> Any ideas?

This function is really used in many places, and these profiles are
not enough at least to me, but it seems you could have a lot of
softirqs (and probably hrtimers) scheduling, so maybe you should try
if e.g. disabling hrtimers or changing kernel HZ makes any difference.

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Tuesday 26 August 2008, Jarek Poplawski wrote:
>
> This function is really used in many places, and these profiles are
> not enough at least to me, but it seems you could have a lot of
> softirqs (and probably hrtimers) scheduling, so maybe you should try
> if e.g. disabling hrtimers or changing kernel HZ makes any difference.
>
> Jarek P.
One user is shapers, it is ok for me.
I am not sure, but maybe another user is softlockup debug option... and if
there is a lot of task switches maybe it will cause excessive load of timers
slow?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Tue, Aug 26, 2008 at 01:29:53PM +0300, Denys Fedoryshchenko wrote:
> On Tuesday 26 August 2008, Jarek Poplawski wrote:
> >
> > This function is really used in many places, and these profiles are
> > not enough at least to me, but it seems you could have a lot of
> > softirqs (and probably hrtimers) scheduling, so maybe you should try
> > if e.g. disabling hrtimers or changing kernel HZ makes any difference.
> >
> > Jarek P.
> One user is shapers, it is ok for me.

The question is if you really need so exact shaping at a cost of
higher system load.

> I am not sure, but maybe another user is softlockup debug option... and if
> there is a lot of task switches maybe it will cause excessive load of timers
> slow?

Maybe. Anyway, you could try if lower HZ (with longer jiffies) can help
with processing more skbs without rescheduling.

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Tuesday 26 August 2008, Jarek Poplawski wrote:
> The question is if you really need so exact shaping at a cost of
> higher system load.
Thats maybe another reason to have your patch in mainline :-)
I will try it today with this case, if it will help.

Maybe it can be optional, and enabled via kernel parameter and /sys , so it
can be useful in case of crashes when TSC used and when timer is too slow.
Because it is not so useful just to disable hrtimers completely, if you need
them for some other task...

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Tue, Aug 26, 2008 at 01:49:09PM +0300, Denys Fedoryshchenko wrote:
> On Tuesday 26 August 2008, Jarek Poplawski wrote:
> > The question is if you really need so exact shaping at a cost of
> > higher system load.
> Thats maybe another reason to have your patch in mainline :-)

We should be first sure when it's really needed.

> I will try it today with this case, if it will help.
>
> Maybe it can be optional, and enabled via kernel parameter and /sys , so it
> can be useful in case of crashes when TSC used and when timer is too slow.
> Because it is not so useful just to disable hrtimers completely, if you need
> them for some other task...

Maybe it could be enough to use current parameters like: "highres=off"
according to Documentation/kernel-parameters.txt?

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Tuesday 26 August 2008, Jarek Poplawski wrote:
> Hmm.. it isn't actually answer to your question, sorry. As I said
> before I think we need to have more people interested in using such
> additional options, and btw. I understood from your message that
> disabling htb didn't solve the problem?
>
> Jarek P.
Only HTB - no. If i disable softlockup debug - seems the load is less (i must
make sure), and if i remove HTB - it is becoming low. I will try to give
exact numbers in recent days.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Tue, Aug 26, 2008 at 11:07:46AM +0000, Jarek Poplawski wrote:
> On Tue, Aug 26, 2008 at 01:49:09PM +0300, Denys Fedoryshchenko wrote:
....
> > Maybe it can be optional, and enabled via kernel parameter and /sys , so it
> > can be useful in case of crashes when TSC used and when timer is too slow.
> > Because it is not so useful just to disable hrtimers completely, if you need
> > them for some other task...
>
> Maybe it could be enough to use current parameters like: "highres=off"
> according to Documentation/kernel-parameters.txt?

Hmm.. it isn't actually answer to your question, sorry. As I said
before I think we need to have more people interested in using such
additional options, and btw. I understood from your message that
disabling htb didn't solve the problem?

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Tue, Aug 26, 2008 at 02:16:32PM +0300, Denys Fedoryshchenko wrote:
> On Tuesday 26 August 2008, Jarek Poplawski wrote:
> > Hmm.. it isn't actually answer to your question, sorry. As I said
> > before I think we need to have more people interested in using such
> > additional options, and btw. I understood from your message that
> > disabling htb didn't solve the problem?
> >
> > Jarek P.
> Only HTB - no. If i disable softlockup debug - seems the load is less (i must
> make sure), and if i remove HTB - it is becoming low. I will try to give
> exact numbers in recent days.
>

So maybe you could try again this htb patch for limiting
qdisc_watchdog_schedule()?

Jarek P.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Tuesday 26 August 2008, Jarek Poplawski wrote:

> So maybe you could try again this htb patch for limiting
> qdisc_watchdog_schedule()?
>
> Jarek P.
Yes, and i am going to take snapshops from system load with different boot
flags. It will take time but, cause it is major router.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Fri, Aug 22, 2008 at 04:57:40AM +0300, Denys Fedoryshchenko (denys@visp.net.lb) wrote:
> I have loaded router (~650 Mbps In+Out), based on 2xAMD Opteron 248, Sun Fire
> X4100. HPET timer available (TSC seems not available on this platform).
> Network interfaces is onboard, connected over PCI-X.
>
> Right now i am using only one processor, cause using only one interface and
> interrupts stick to it. Other is almost not used.
> At peak time i notice in mpstat, that this processor is almost "dead", and if
> i run minor application consuming resources - ping over this router will be
> terrible. For me it is clear - system overloaded. I did oprofile, and here is
> result (at low load time, but at peak time it is very similar).

Do you have any packet sockets in this system? Like running dhcp daemon?

--
Evgeniy Polyakov
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Evgeniy Polyakov a écrit :
> On Fri, Aug 22, 2008 at 04:57:40AM +0300, Denys Fedoryshchenko (denys@visp.net.lb) wrote:
>> I have loaded router (~650 Mbps In+Out), based on 2xAMD Opteron 248, Sun Fire
>> X4100. HPET timer available (TSC seems not available on this platform).
>> Network interfaces is onboard, connected over PCI-X.
>>
>> Right now i am using only one processor, cause using only one interface and
>> interrupts stick to it. Other is almost not used.
>> At peak time i notice in mpstat, that this processor is almost "dead", and if
>> i run minor application consuming resources - ping over this router will be
>> terrible. For me it is clear - system overloaded. I did oprofile, and here is
>> result (at low load time, but at peak time it is very similar).
>
> Do you have any packet sockets in this system? Like running dhcp daemon?
>

Another way to see this problem can be to start a sniffer on the machine, even with a
restrictive pcap filter, to check if performance change or not. (It should decrease)

For example, I believe that running "ping" could have the same effect
(increasing netstamp_needed variable : every incoming packet has to be timestamped)

So beware of pings, traceroute and other networking tools...



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Tue, Aug 26, 2008 at 10:44:56PM +0200, Eric Dumazet (dada1@cosmosbay.com) wrote:
> >Do you have any packet sockets in this system? Like running dhcp daemon?
> >
>
> Another way to see this problem can be to start a sniffer on the machine,
> even with a restrictive pcap filter, to check if performance change or not.
> (It should decrease)

Or just check /proc/net/packet iirc.
Anyway, having at least one packet socket ends up with timestamping of
each packet, so you will get fair load of getnstimeofday() in that case.

> For example, I believe that running "ping" could have the same effect
> (increasing netstamp_needed variable : every incoming packet has to be
> timestamped)
>
> So beware of pings, traceroute and other networking tools...

Yup, this innocent toys can end up with this such behaviour on modern
highly loaded machines.

--
Evgeniy Polyakov
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Tuesday 26 August 2008, Evgeniy Polyakov wrote:
> On Tue, Aug 26, 2008 at 10:44:56PM +0200, Eric Dumazet (dada1@cosmosbay.com)
wrote:
> > >Do you have any packet sockets in this system? Like running dhcp daemon?
No, nothing at all.
> >
> > Another way to see this problem can be to start a sniffer on the machine,
> > even with a restrictive pcap filter, to check if performance change or
> > not. (It should decrease)
Yes, when i run tcpdump even without promisc at peak time, machine will be
almost dead. Transit traffic will be 100ms+. I know that it is timestamping
packets. Same almost for any libpcap app.

>
> Or just check /proc/net/packet iirc.
> Anyway, having at least one packet socket ends up with timestamping of
> each packet, so you will get fair load of getnstimeofday() in that case.
There is very short list of tasks. Attached.
/proc/net/packet clean, nothing there.

>
> > For example, I believe that running "ping" could have the same effect
> > (increasing netstamp_needed variable : every incoming packet has to be
> > timestamped)
Even answering icmp timestamp request will take resources.
> >
> > So beware of pings, traceroute and other networking tools...
When i am measuring performance - they are all off.
>
> Yup, this innocent toys can end up with this such behaviour on modern
> highly loaded machines.



tcp 0 0 127.0.0.1:2600 0.0.0.0:* LISTEN 3167/zebra
tcp 0 0 0.0.0.0:2601 0.0.0.0:* LISTEN 3167/zebra
tcp 0 0 0.0.0.0:2602 0.0.0.0:* LISTEN 3174/ripd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 3549/sshd
tcp 0 0 194.146.153.17:22 194.146.154.161:37549 ESTABLISHED11593/sshd
tcp 0 0 194.146.153.17:22 192.168.0.92:45891 ESTABLISHED11803/sshd
tcp 0 0 127.0.0.1:42537 127.0.0.1:2600 ESTABLISHED3174/ripd
tcp 0 0 194.146.153.17:22 194.146.153.18:51810 ESTABLISHED11799/sshd
tcp 0 0 127.0.0.1:2600 127.0.0.1:42537 ESTABLISHED3167/zebra
udp 0 0 0.0.0.0:520 0.0.0.0:* 3174/ripd
udp 0 0 0.0.0.0:161 0.0.0.0:* 3194/snmpd
udp 0 0 0.0.0.0:67 0.0.0.0:* 3207/udhcpd
udp 111360 0 0.0.0.0:49619 0.0.0.0:* 2449/syslogd

On Wed, Aug 27, 2008 at 03:09:17PM +0300, Denys Fedoryshchenko (denys@visp.net.lb) wrote:
> On Tuesday 26 August 2008, Evgeniy Polyakov wrote:
> > On Tue, Aug 26, 2008 at 10:44:56PM +0200, Eric Dumazet (dada1@cosmosbay.com)
> wrote:
> > > >Do you have any packet sockets in this system? Like running dhcp daemon?
> No, nothing at all.

Can you put debug print into
net_enable_timestamp()/net_disable_timestamp() to determine if someone
enabled timestamp socket option?

> tcp 0 0 127.0.0.1:2600 0.0.0.0:* LISTEN 3167/zebra
> tcp 0 0 0.0.0.0:2601 0.0.0.0:* LISTEN 3167/zebra
> tcp 0 0 0.0.0.0:2602 0.0.0.0:* LISTEN 3174/ripd
> tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 3549/sshd
> tcp 0 0 194.146.153.17:22 194.146.154.161:37549 ESTABLISHED11593/sshd
> tcp 0 0 194.146.153.17:22 192.168.0.92:45891 ESTABLISHED11803/sshd
> tcp 0 0 127.0.0.1:42537 127.0.0.1:2600 ESTABLISHED3174/ripd
> tcp 0 0 194.146.153.17:22 194.146.153.18:51810 ESTABLISHED11799/sshd
> tcp 0 0 127.0.0.1:2600 127.0.0.1:42537 ESTABLISHED3167/zebra
> udp 0 0 0.0.0.0:520 0.0.0.0:* 3174/ripd
> udp 0 0 0.0.0.0:161 0.0.0.0:* 3194/snmpd
> udp 0 0 0.0.0.0:67 0.0.0.0:* 3207/udhcpd

This one looks suspicious ^^^^^^^^^^

> udp 111360 0 0.0.0.0:49619 0.0.0.0:* 2449/syslogd


--
Evgeniy Polyakov
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Evgeniy Polyakov writes:
>
> Yup, this innocent toys can end up with this such behaviour on modern
> highly loaded machines.

I and also other people had some patches to move the time stamp
measuring into the socket. This way the time stamping didn't need to
be enabled on all packets, only on those that actually end up at a
socket that requires the time stamp.

Unfortunately DaveM didn't like it because some bank wanted
different semantics, see the discussion in
http://thread.gmane.org/gmane.linux.network/91679

Perhaps you can find out which bank it was and send them a bill for
your CPU time ;-)

-Andi

--
ak@linux.intel.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Wednesday 27 August 2008, Evgeniy Polyakov wrote:
> Can you put debug print into
> net_enable_timestamp()/net_disable_timestamp() to determine if someone
> enabled timestamp socket option?
OK, i will do that on next system reboot.

> > 0.0.0.0:* 3207/udhcpd
>
> This one looks suspicious
> ^^^^^^^^^^
It is busybox udhcpd... i guess it is innocent. Even i kill it - it doesn't
change anything at all.
Only who possible listen multicast socket - it is ripd, i cannot kill him. But
i think it doesn't matter much too...

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

On Wed, Aug 27, 2008 at 05:00:35PM +0300, Denys Fedoryshchenko (denys@visp.net.lb) wrote:
> > > 0.0.0.0:* 3207/udhcpd
> >
> > This one looks suspicious
> > ^^^^^^^^^^
> It is busybox udhcpd... i guess it is innocent. Even i kill it - it doesn't
> change anything at all.
> Only who possible listen multicast socket - it is ripd, i cannot kill him. But
> i think it doesn't matter much too...

It depends... If it turns timestamps on, then you will have this
behaviour. Please check if timestamps are actually enabled, so we could
remove one (im)possible case.

--
Evgeniy Polyakov
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Andi Kleen wrote:
> Evgeniy Polyakov writes:
>
>>Yup, this innocent toys can end up with this such behaviour on modern
>>highly loaded machines.
>
>
> I and also other people had some patches to move the time stamp
> measuring into the socket. This way the time stamping didn't need to
> be enabled on all packets, only on those that actually end up at a
> socket that requires the time stamp.
>
> Unfortunately DaveM didn't like it because some bank wanted
> different semantics, see the discussion in
> http://thread.gmane.org/gmane.linux.network/91679
>
> Perhaps you can find out which bank it was and send them a bill for
> your CPU time ;-)

Those banks really want to crank down on latency - to the point they
start disabling interrupt coalescing. I bet they'd toss anything out
they could to shave another microsecond.

rick jones
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/