netstat -s output: "packets pruned" and "packets collapsed" - TCP-IP

This is a discussion on netstat -s output: "packets pruned" and "packets collapsed" - TCP-IP ; Hi All, We seem to have networking related problems running some locally developed applications on a pair of identical Linux (2.6.18-1.2239.fc5) server systems. While trying to track down the source of these problems, I found some output from the "netstat ...

+ Reply to Thread
Results 1 to 3 of 3

Thread: netstat -s output: "packets pruned" and "packets collapsed"

  1. netstat -s output: "packets pruned" and "packets collapsed"

    Hi All,

    We seem to have networking related problems running some locally
    developed applications on a pair of identical Linux
    (2.6.18-1.2239.fc5) server systems. While trying to track down the
    source of these problems, I found some output from the "netstat -s"
    command that looks worrying e.g.
    ...
    4087 packets pruned from receive queue because of socket buffer
    overrun
    3827052 packets collapsed in receive queue due to low socket
    buffer
    ...
    (complete sample output from netstat -s is included below).

    These variables first begin to be reported when when our daemons
    (applications) are started and their values ramp up from then on
    (obviously :-)).

    Despite some pretty intensive research I have found practically zero
    information about these variables. Does anyone out there have some
    tips for me on what these actually mean? For example, I notice that
    they are both, per their names, explicitly socket related. This lead
    me to try increasing the size of some of the system networking buffers
    e.g.
    /proc/sys/net/ipv4/tcp_rmem default value = 4096 87380
    4194304
    /proc/sys/net/ipv4/tcp_wmem default value = 4096 16384
    4194304

    And:
    /proc/sys/net/core/rmem_max default size = 131071
    /proc/sys/net/core/wmem_max default size = 131071

    I doubled these sizes and then doubled them again, without any
    apparent improvement in the rate at which the above errors were
    occurring. Note though that "tcp_moderate_rcvbuf" is also set to 1 on
    these systems which, as I understand it, means buffer sizes are auto-
    tuned (could it be then that my changes had no effect - is it
    necessary to turn this auto-tuning off, before changing the above
    sizes manually?)

    I also tried a different approach of greping the source tree for any
    calls to "setsockopt" looking for, for example, small buffer sizes
    being set, but none were apparent (I did find options such as:
    SO_SNDTIMEO, SO_RCVTIMEO and TCP_NODELAY being set).

    At this point I am really just assuming that these values are not
    normal and are in fact associated with our actual problems. Please
    take a look at the output below and let me know what you think.

    Thanks in advance!

    Yours,
    Robb.

    +-----------------------------------------------------------+
    | "I've seen things you people wouldn't believe. |
    | Attack ships on fire off the shoulder of Orion. I watched |
    | C-Beams glitter in the dark near the Tannhauser Gate. All |
    | those moments will be lost in time, like tears in rain. |
    | Time to die." -- Roy Batty, Nexus6, N6MAA10816, Combat |
    +-----------------------------------------------------------+


    Ip:
    4097694726 total packets received
    155085 with invalid addresses
    2492354975 forwarded
    0 incoming packets discarded
    1605183310 incoming packets delivered
    4042938962 requests sent out
    Icmp:
    38331 ICMP messages received
    9 input ICMP message failed.
    ICMP input histogram:
    destination unreachable: 36
    echo requests: 25554
    echo replies: 12741
    26659 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
    destination unreachable: 1105
    echo replies: 25554
    Tcp:
    362332 active connections openings
    404536 passive connection openings
    1580 failed connection attempts
    7458 connection resets received
    187 connections established
    1600328035 segments received
    1550250974 segments send out
    260697 segments retransmited
    0 bad segments received.
    7298 resets sent
    Udp:
    66672 packets received
    1080 packets to unknown port received.
    0 packet receive errors
    66988 packets sent
    TcpExt:
    5354 invalid SYN cookies received
    613 resets received for embryonic SYN_RECV sockets
    4087 packets pruned from receive queue because of socket buffer
    overrun
    398014 TCP sockets finished time wait in fast timer
    158 time wait sockets recycled by time stamp
    39735643 delayed acks sent
    544 delayed acks further delayed because of locked socket
    Quick ack mode was activated 3938 times
    547091 times the listen queue of a socket overflowed
    547091 SYNs to LISTEN sockets ignored
    10641526 packets directly queued to recvmsg prequeue.
    19159885 packets directly received from backlog
    1523726509 packets directly received from prequeue
    1070393298 packets header predicted
    8431813 packets header predicted and directly queued to user
    14447834 acknowledgments not containing data received
    817036278 predicted acknowledgments
    111 times recovered from packet loss due to SACK data
    3887 congestion windows recovered after partial ack
    164 TCP data loss events
    71 timeouts after SACK recovery
    157 fast retransmits
    27 forward retransmits
    58 retransmits in slow start
    259192 other TCP timeouts
    2 sack retransmits failed
    3827052 packets collapsed in receive queue due to low socket
    buffer
    36049 DSACKs sent for old packets
    28 DSACKs sent for out of order packets
    4 DSACKs received
    1852 connections reset due to unexpected data
    894 connections reset due to early user close
    2 connections aborted due to timeout


  2. Re: netstat -s output: "packets pruned" and "packets collapsed"

    On Fri, 20 Jul 2007 09:26:27 -0700, roybatty wrote:
    ....
    > Despite some pretty intensive research I have found practically zero
    > information about these variables. Does anyone out there have some
    > tips for me on what these actually mean?


    There is (reference) documentation in the kernel sources:

    linux/Documentation/networking/ip-sysctl.txt

    > /proc/sys/net/ipv4/tcp_rmem
    > /proc/sys/net/ipv4/tcp_wmem
    > /proc/sys/net/core/rmem_max
    > /proc/sys/net/core/wmem_max


    You can Google for tuning tips, but most material you find appears to
    be "cargo-culted". One piece that at least /looks/ reasonable:

    Wadenstein. M.: TCP performance tuning - how to tune linux
    http://www.acc.umu.se/~maswan/linux-netperf.txt

    /Jorgen

    --
    // Jorgen Grahn \X/ snipabacken.dyndns.org> R'lyeh wgah'nagl fhtagn!

  3. Re: netstat -s output: "packets pruned" and "packets collapsed"

    I had similar problems. The folowing sysctls seem to have fixed it:

    sysctl -q -w net.ipv4.tcp_adv_win_scale=1
    sysctl -q -w net.ipv4.tcp_app_win=30

    they control somehow how a part of the receive window is keept for buffering delayed reads by the application due to scheduling (?)

    ML

+ Reply to Thread