In the last post, I mentioned I'd demonstrate how we can probe TCP sliding window behaviour. When examining performance problems relating to sliding window on a system, the key thing to determine is if we are encountering situations where either:
  • the send window of any TCP connections is zero. This indicates we can no longer send data since the peer connection is advertising a zero-sized window.
  • the receive window of any TCP connections is zero. This indicates we can no longer receive data since we are advertising a zero-sized window.

If either of these situations occurs regularly, we will then need to dig down and find out why the send or receive windows are filling up, but the first step is to determine if these events are occuring, and if so, how long are we stuck unable to send or receive new data? We can do this by timing the interval between zero window advertisement send and new data receive (for receive window) and receipt of a zero window advertisement and new data send (for send window). Here is a script which averages zero receive window idletime by remote address/port. This is the average time between when we advertise a zero window and when we resume receiving new data.

#!/usr/sbin/dtrace -s#pragma D option quiettcp:::send/ args[4]->tcp_window == 0 && (args[4]->tcp_flags & TH_RST) == 0 /{ rwndclosed[args[1]->cs_cid] = timestamp; rwndrnxt[args[1]->cs_cid] = args[3]->tcps_rnxt; @numrwndclosed[args[2]->ip_daddr, args[4]->tcp_dport] = count();}tcp:::receive/ rwndclosed[args[1]->cs_cid] && args[4]->tcp_seq >= rwndrnxt[args[1]->cs_cid] /{ @meantimeclosed[args[2]->ip_saddr, args[4]->tcp_sport] = avg(timestamp - rwndclosed[args[1]->cs_cid]); @stddevtimeclosed[args[2]->ip_saddr, args[4]->tcp_sport] = stddev(timestamp - rwndclosed[args[1]->cs_cid]); rwndclosed[args[1]->cs_cid] = 0; rwndrnxt[args[1]->cs_cid] = 0;}END{ printf("%-20s %-8s %-25s %-8s %-8s\n", "Remote host", "Port", "TCP Avg RwndClosed(ns)", "StdDev", "Num"); printa("%-20s %-8d %@-25d %@-8d %@-8d\n", @meantimeclosed, @stddevtimeclosed, @numrwndclosed);}Here's the output showing one receive window close event, triggered by opening a hi-res YouTube video in a browser.

# dtrace -s tcp_rwndclosed.d^CRemote host Port TCP Avg RwndClosed(ns) StdDev Num 80 26914620 0 1 We can see we were stuck unable to receive new data for 0.269 seconds since our receive window was 0. Here's the send window equivalent:

#!/usr/sbin/dtrace -s#pragma D option quiettcp:::receive/ args[4]->tcp_window == 0 && (args[4]->tcp_flags & TH_RST) == 0 /{ swndclosed[args[1]->cs_cid] = timestamp; swndsnxt[args[1]->cs_cid] = args[3]->tcps_snxt; @numswndclosed[args[2]->ip_saddr, args[4]->tcp_sport] = count();}tcp:::send/ swndclosed[args[1]->cs_cid] && args[4]->tcp_seq >= swndsnxt[args[1]->cs_cid] /{ @meantimeclosed[args[2]->ip_daddr, args[4]->tcp_dport] = avg(timestamp - swndclosed[args[1]->cs_cid]); @stddevtimeclosed[args[2]->ip_daddr, args[4]->tcp_dport] = stddev(timestamp - swndclosed[args[1]->cs_cid]); swndclosed[args[1]->cs_cid] = 0; swndsnxt[args[1]->cs_cid] = 0;}END{ printf("%-20s %-8s %-25s %-8s %-8s\n", "Remote host", "Port", "TCP Avg SwndClosed(ns)", "StdDev", "Num"); printa("%-20s %-8d %@-25d %@-8d %@-8d\n", @meantimeclosed, @stddevtimeclosed, @numswndclosed);}If a lot of zero-window events are occuring, a good next step would be to use the acknowledgement RTT latency measurement script in the previous entry - this will help determine if the ACK RTT is sufficiently high such that the send window fills up before any data is acknowledged.

Read More about [DTrace TCP provider and TCP's sliding window...