Weird TCP/IP problem - TCP-IP

This is a discussion on Weird TCP/IP problem - TCP-IP ; Here's a summary of a capture of a failed connection. I captured both ends and both traces look identical except for the source/destination address swap. This is for an SSL connection on port 443. Client Server 1 ------SYN------> 2 3 ...

+ Reply to Thread
Results 1 to 10 of 10

Thread: Weird TCP/IP problem

  1. Weird TCP/IP problem

    Here's a summary of a capture of a failed connection.
    I captured both ends and both traces look identical except
    for the source/destination address swap. This is for an
    SSL connection on port 443.

    Client Server

    1 ------SYN------>
    2 <---SYN,ACK-----
    3 ------ACK------> (so far so good)
    4 <---SYN,ACK----- (huh?)
    5 ------ACK------> dup of frame 3 according to wireshark
    6 <---SYN,ACK-----
    7 ------ACK------> dup of frame 3

  2. Re: Weird TCP/IP problem

    In article <1221537151_2922@news.usenet.com>,
    Jim Garrison wrote:

    > Here's a summary of a capture of a failed connection.
    > I captured both ends and both traces look identical except
    > for the source/destination address swap. This is for an
    > SSL connection on port 443.
    >
    > Client Server
    >
    > 1 ------SYN------>
    > 2 <---SYN,ACK-----
    > 3 ------ACK------> (so far so good)
    > 4 <---SYN,ACK----- (huh?)
    > 5 ------ACK------> dup of frame 3 according to wireshark
    > 6 <---SYN,ACK-----
    > 7 ------ACK------> dup of frame 3
    > .
    > .
    > .
    > ad nauseum, at ever increasing delays between packets
    >
    > Can someone explain what's going on here?
    >
    > The server is Fedora Core 5 (yes, I know it's ancient) and
    > the client is Firefox 3.0.1 on Windows XP SP3. This
    > problem started happening after upgrading Firefox, but
    > since it's the server that keeps resending SYN/ACK packets
    > I doubt the problem is on the client. The server configuration
    > has been unchanged for 2 years except that I switched from
    > a self-signed cert to a commercial cert a few weeks ago.
    > The commercial cert came with an "intermediate bundle".
    > Could that somehow be related to this problem?


    I doubt it, since your problem is occurring at the TCP layer, not the
    SSL layer.

    I can think of a couple of things:

    1) Is the checksum on the ACK good? If not, it will be discarded by the
    server.

    2) Is the serial number in the ACK correct? If there's a serial number
    gap, the server will retransmit the SYN/ACK.

    3) Do you have a firewall enabled on the server? I assume Wireshark
    captures packets before the firewall gets its hands on them, so a packet
    can appear in Wireshark, but never be seen by the TCP driver. If the
    ACKs get filtered, the SYN/ACKs will be retransmitted.

    How much delay is there before each of the repeated SYN/ACKs?

    --
    Barry Margolin, barmar@alum.mit.edu
    Arlington, MA
    *** PLEASE don't copy me on replies, I'll read them in the group ***

  3. Re: Weird TCP/IP problem

    Barry Margolin wrote:
    > I can think of a couple of things:
    >
    > 1) Is the checksum on the ACK good? If not, it will be discarded by the
    > server.


    All the checksums are good according to Wireshark

    > 2) Is the serial number in the ACK correct? If there's a serial number
    > gap, the server will retransmit the SYN/ACK.


    Again, according to Wireshark, the serial numbers are correct

    > How much delay is there before each of the repeated SYN/ACKs?


    The first three packets occur within the first 54ms.

    The fourth packet (the first repeated SYN/ACK) comes almost exactly
    3 seconds after packet number 3. Following that, the delays between
    successive SYN/ACKs are 6, 12 and 20 seconds (at which point I stopped
    the connection at the browser).

    > 3) Do you have a firewall enabled on the server? I assume Wireshark
    > captures packets before the firewall gets its hands on them, so a packet
    > can appear in Wireshark, but never be seen by the TCP driver. If the
    > ACKs get filtered, the SYN/ACKs will be retransmitted.


    The server is Linux and I'm capturing with tcpdump. The firewall is
    standard iptables, and hasn't changed in a year or so.

    I switched off Wireshark's sequence analysis and can see that
    the server is behaving as if it never received the first ack.
    All the syn/acks sent by the server have the same sequence and
    ack fields. I know the "missing" ack is arriving at the server
    because it shows up in the tcpdum capture.

    The weirdest part is that restarting the connection attempt at
    the client almost always succeeds. This is a very intermittent
    problem and occurs about 10% of the time.

    Hmmm....

    $ uptime
    19:27:11 up 432 days, 11:40, 1 user, load average: 1.00, 1.00, 1.00

    Maybe it's time to reboot the server?



  4. Re: Weird TCP/IP problem

    Jim Garrison wrote:

    > Here's a summary of a capture of a failed connection.
    > I captured both ends and both traces look identical except
    > for the source/destination address swap. This is for an
    > SSL connection on port 443.
    >
    > Client Server
    >
    > 1 ------SYN------>
    > 2 <---SYN,ACK-----
    > 3 ------ACK------> (so far so good)
    > 4 <---SYN,ACK----- (huh?)
    > 5 ------ACK------> dup of frame 3 according to wireshark
    > 6 <---SYN,ACK-----
    > 7 ------ACK------> dup of frame 3
    > .
    > .
    > .
    > ad nauseum, at ever increasing delays between packets
    >
    > Can someone explain what's going on here?
    >
    > The server is Fedora Core 5 (yes, I know it's ancient) and
    > the client is Firefox 3.0.1 on Windows XP SP3. This
    > problem started happening after upgrading Firefox, but
    > since it's the server that keeps resending SYN/ACK packets
    > I doubt the problem is on the client. The server configuration
    > has been unchanged for 2 years except that I switched from
    > a self-signed cert to a commercial cert a few weeks ago.
    > The commercial cert came with an "intermediate bundle".
    > Could that somehow be related to this problem?



    Seems like a simple packet loss (FW, some filtering etc.) problem. What
    if pkt#3 (the ack) never made it to the other side? The other guys
    will assume syn+ack never made it and will send a dup out. Windows
    will retransmit the syn or syn+ack at 3 second, 6 second then 9 second
    interval (IIRC) and then give up.


    --

    hsb


    "Somehow I imagined this experience would be more rewarding" Calvin
    ************************************************** ******************
    Due to the volume of email that I receive, I may not be able to
    reply to emails sent to my account. Please post a followup instead.
    ************************************************** ******************

  5. Re: Weird TCP/IP problem

    In article ,
    "Hansang Bae" wrote:

    > Jim Garrison wrote:
    >
    > > Here's a summary of a capture of a failed connection.
    > > I captured both ends and both traces look identical except
    > > for the source/destination address swap. This is for an
    > > SSL connection on port 443.
    > >
    > > Client Server
    > >
    > > 1 ------SYN------>
    > > 2 <---SYN,ACK-----
    > > 3 ------ACK------> (so far so good)
    > > 4 <---SYN,ACK----- (huh?)
    > > 5 ------ACK------> dup of frame 3 according to wireshark
    > > 6 <---SYN,ACK-----
    > > 7 ------ACK------> dup of frame 3
    > > .
    > > .
    > > .
    > > ad nauseum, at ever increasing delays between packets
    > >
    > > Can someone explain what's going on here?
    > >
    > > The server is Fedora Core 5 (yes, I know it's ancient) and
    > > the client is Firefox 3.0.1 on Windows XP SP3. This
    > > problem started happening after upgrading Firefox, but
    > > since it's the server that keeps resending SYN/ACK packets
    > > I doubt the problem is on the client. The server configuration
    > > has been unchanged for 2 years except that I switched from
    > > a self-signed cert to a commercial cert a few weeks ago.
    > > The commercial cert came with an "intermediate bundle".
    > > Could that somehow be related to this problem?

    >
    >
    > Seems like a simple packet loss (FW, some filtering etc.) problem. What
    > if pkt#3 (the ack) never made it to the other side? The other guys
    > will assume syn+ack never made it and will send a dup out. Windows
    > will retransmit the syn or syn+ack at 3 second, 6 second then 9 second
    > interval (IIRC) and then give up.


    He said he did packet captures at both ends, and all the packets were
    the same in both. So nothing is being lost in the network.

    That's why I think the ACK packet is being filtered out in the server's
    TCP/IP stack.

    --
    Barry Margolin, barmar@alum.mit.edu
    Arlington, MA
    *** PLEASE don't copy me on replies, I'll read them in the group ***

  6. Re: Weird TCP/IP problem



    > > Jim Garrison wrote:
    > > > Here's a summary of a capture of a failed connection.
    > > > I captured both ends and both traces look identical except
    > > > for the source/destination address swap. This is for an
    > > > SSL connection on port 443.
    > > >
    > > > Client Server
    > > >
    > > > 1 ------SYN------>
    > > > 2 <---SYN,ACK-----
    > > > 3 ------ACK------> (so far so good)
    > > > 4 <---SYN,ACK----- (huh?)
    > > > 5 ------ACK------> dup of frame 3 according to wireshark
    > > > 6 <---SYN,ACK-----
    > > > 7 ------ACK------> dup of frame 3
    > > > .





    Barry Margolin wrote:
    > He said he did packet captures at both ends, and all the packets were
    > the same in both. So nothing is being lost in the network.
    > That's why I think the ACK packet is being filtered out in the
    > server's TCP/IP stack.



    Hmmm. I don't know that I've ever seen that happen. My first guess
    would be he really didn't capture on both *FAR* ends. So the ACK got
    lost *past* where the packet capture occurred. That seems more likely
    to me. I'd love to find out what really happened.

    --

    hsb


    "Somehow I imagined this experience would be more rewarding" Calvin
    ************************************************** ******************
    Due to the volume of email that I receive, I may not be able to
    reply to emails sent to my account. Please post a followup instead.
    ************************************************** ******************

  7. Re: Weird TCP/IP problem

    Hansang Bae wrote:
    >>> Jim Garrison wrote:
    >>>> Here's a summary of a capture of a failed connection.
    >>>> I captured both ends and both traces look identical except
    >>>> for the source/destination address swap. This is for an
    >>>> SSL connection on port 443.
    >>>>
    >>>> Client Server
    >>>>
    >>>> 1 ------SYN------>
    >>>> 2 <---SYN,ACK-----
    >>>> 3 ------ACK------> (so far so good)
    >>>> 4 <---SYN,ACK----- (huh?)
    >>>> 5 ------ACK------> dup of frame 3 according to wireshark
    >>>> 6 <---SYN,ACK-----
    >>>> 7 ------ACK------> dup of frame 3
    >>>> .

    >
    >
    >
    >
    > Barry Margolin wrote:
    >> He said he did packet captures at both ends, and all the packets were
    >> the same in both. So nothing is being lost in the network.
    >> That's why I think the ACK packet is being filtered out in the
    >> server's TCP/IP stack.

    >
    >
    > Hmmm. I don't know that I've ever seen that happen. My first guess
    > would be he really didn't capture on both *FAR* ends. So the ACK got
    > lost *past* where the packet capture occurred. That seems more likely
    > to me. I'd love to find out what really happened.
    >


    I did capture at both ends. On the local (Windoze) side, using
    Wireshark. The remote system is Fedora Core 5 (yes, I know that's
    old) with tcpdump. The problem is intermittent, and I've not been
    able to pin it down. It seems to affect only SSL-encrypted
    connections (https, secure-POP) so I suspect something in the SSL
    library on the server.

    This suspicion is based on two facts: (1) SSH is not affected, only
    SSL-based protocols, and (2) the problem started immediately after I
    replaced my old self-signed cert with a 'real' commercial one. The
    major difference is that the commercial cert comes with an
    "intermediate bundle" that is required for operation. Perhaps
    something in the SSL library is occasionally choking on that...?

    I plan to upgrade the server to Fedora Core 9 but can't get to it for
    a few weeks.

  8. Re: Weird TCP/IP problem

    In article <1222412387_87134@news.usenet.com>,
    Jim Garrison wrote:

    > I did capture at both ends. On the local (Windoze) side, using
    > Wireshark. The remote system is Fedora Core 5 (yes, I know that's
    > old) with tcpdump. The problem is intermittent, and I've not been
    > able to pin it down. It seems to affect only SSL-encrypted
    > connections (https, secure-POP) so I suspect something in the SSL
    > library on the server.
    >
    > This suspicion is based on two facts: (1) SSH is not affected, only
    > SSL-based protocols, and (2) the problem started immediately after I
    > replaced my old self-signed cert with a 'real' commercial one. The
    > major difference is that the commercial cert comes with an
    > "intermediate bundle" that is required for operation. Perhaps
    > something in the SSL library is occasionally choking on that...?
    >
    > I plan to upgrade the server to Fedora Core 9 but can't get to it for
    > a few weeks.


    SSL is in the application layer, it shouldn't be able to affect the
    transport layer.

    I still suspect some kind of packet filtering.

    --
    Barry Margolin, barmar@alum.mit.edu
    Arlington, MA
    *** PLEASE don't copy me on replies, I'll read them in the group ***

  9. Re: Weird TCP/IP problem

    Barry Margolin wrote:
    > SSL is in the application layer, it shouldn't be able to affect the
    > transport layer.
    >
    > I still suspect some kind of packet filtering.


    Based on what I know about TCP/IP (I'm not an expert) I would
    tend to agree with you. But I can't think of any scenario in
    which failure would be intermittent (10-20% of the time) and
    random. Since I built the server I can assert that it's
    running only standard iptables without any sort of automatic
    traffic-based blocking. If you have any other suggestions,
    I'm all ears (er... eyes :-)


  10. Re: Weird TCP/IP problem

    Jim Garrison wrote:
    > Based on what I know about TCP/IP (I'm not an expert) I would
    > tend to agree with you. But I can't think of any scenario in
    > which failure would be intermittent (10-20% of the time) and
    > random. Since I built the server I can assert that it's
    > running only standard iptables without any sort of automatic
    > traffic-based blocking. If you have any other suggestions,
    > I'm all ears (er... eyes :-)



    Wow, you have an odd problem indeed. Whatever you find, please post a
    follow up. One final shot in the doc, is it possible that you have
    multiple NICs and the ACK (syn, syn+ack, then ACK) is getting lost
    somehow do to a driver issue? grasping at straws here.

    --

    hsb


    "Somehow I imagined this experience would be more rewarding" Calvin
    ************************************************** ******************
    Due to the volume of email that I receive, I may not be able to
    reply to emails sent to my account. Please post a followup instead.
    ************************************************** ******************

+ Reply to Thread