Odd socket stream happening more and more often - TCP-IP

This is a discussion on Odd socket stream happening more and more often - TCP-IP ; Hi, I'm having some issues with an application. Both machines are FreeBSD (Client is 5.3-RELEASE-p10, Server is 5.5-RELEASE-p8). The client and server are perl 5.8.6 applications. The client contacts the server on a specific port, the server spits out about ...

+ Reply to Thread
Results 1 to 3 of 3

Thread: Odd socket stream happening more and more often

  1. Odd socket stream happening more and more often

    Hi,

    I'm having some issues with an application. Both machines are
    FreeBSD (Client is 5.3-RELEASE-p10, Server is 5.5-RELEASE-p8). The
    client and server are perl 5.8.6 applications. The client contacts the
    server on a specific port, the server spits out about 50 lines of text
    and closes the connection. (Monitoring application). They've been
    working together almost perfectly for almost 3 years (Almost perfectly
    means that every once in a great while there was a warning that it
    couldn't get the data it needed, but it happened so infrequently that
    it wasn't a concern). The client polls the server 3 times every 5
    minutes. Every 10 minutes there is 1 extra poll. There are absolutely
    no firewall issues on the server.

    Now, its happening 3-4 times a day, and is a great concern.

    I can provide TCPDUMPs of the actual session. What follows is
    from the servers point of view

    When all is fine, the sequence looks like the following.
    (C=client, S=server) Its exactly the same time after time after time.

    C: SYN
    S: SYN,ACK to previous frame (Ack #1)
    C: ACK to previous frame (Seq #1, Ack #1)
    S: PSH,ACK to previous frame, contains some data (Seq #1, Ack #1)
    S: FIN,PSH,ACK contains some data (Seq #22, Ack #1)
    C: ACK to previous frame (Seq #1, Ack #1030)
    C: FIN,ACK (Seq #1, Ack #1030)
    S: ACK to previous frame (Seq #1030, Ack #2)

    When its gone bad, it does something like (C=client, S=server)

    C: SYN
    S: SYN,ACK to previous frame (Ack #1)
    C: ACK to previous frame (Seq #1, Ack #1)
    S: PSH,ACK to previous frame, contains some data (Seq #1, Ack #1)
    C: ACK to previous frame (Seq #1, Ack #22)
    S: PSH,ACK to previous frame, contains some data (Seq #22, Ack #1)
    C: FIN,ACK to previous frame (Seq #1, Ack #705)
    S: ACK to previous frame (Seq #705, Ack #2)
    S: PSH,ACK with more data (Seq #705, Ack #2)
    C: RST (Seq #2)

    Not all the data was transmitted. Then what happens is the
    next few requests during that "interval" look like :

    C: SYN
    S: RST,ACK to previous frame (Seq #1, Ack #1)

    C: SYN
    S: RST,ACK to previous frame (Seq #1, Ack #1)

    However, when the next interval happens 5 minutes later, all
    is well.

    It appears from my interpretation that the client goes away in
    the first bad trace, and then stops answering the port on the
    subsequent ones. I can't find anything from the server that it
    restarted its process, etc.

    Is my determination correct, and where to look if not?

    Thanks, Tuc

  2. Re: Odd socket stream happening more and more often

    Tuc writes:

    [...]

    > Not all the data was transmitted. Then what happens is the
    > next few requests during that "interval" look like :
    >
    > C: SYN
    > S: RST,ACK to previous frame (Seq #1, Ack #1)
    >
    > C: SYN
    > S: RST,ACK to previous frame (Seq #1, Ack #1)


    This is consistent with your server crashing, and being restarted a
    short while later. Using logging in your server would be a good way
    to figure out exactly what's going on.

    Hope this helps,

    ----Scott.

  3. Re: Odd socket stream happening more and more often

    On 2008-04-08 09:15:45 -0400, Tuc said:

    > Hi,
    >
    > I'm having some issues with an application. Both machines are
    > FreeBSD (Client is 5.3-RELEASE-p10, Server is 5.5-RELEASE-p8). The
    > client and server are perl 5.8.6 applications. The client contacts the


    I hope you've resolved your issue by now, but if not, I'm wondering:

    - Is there only a LAN-sized network between these hosts, or another
    network with considerable distance between these two hosts?

    - Looking at the data you provided, it looks like roughly only half of
    the data is going through chunked in a smaller size (see where you have
    ACK #1030 in the "good" example, and ACK #705 in the "bad" example?).
    This makes me wonder if the source of the data the server is providing
    is constrained somehow. I'm making the assumption that you're using
    Ethernet, and that your MTU for the physical interface is 1500 or
    thereabouts bytes, so if your application could fully pack a PDU full
    of 1500 bytes, it would, but it isn't - it seems that (1030-22) is the
    data you're passing, and in the bad example, you seem to only be able
    to pass (705-22) bytes.

    All this is guesswork from the data you provided. Lots of bright people
    in this group, sure as the conversation goes along we'll get to the
    bottom of it.

    /dmfh

    --

    __| |_ __ / _| |_ 01100100 01101101
    / _` | ' \| _| ' \ 01100110 01101000
    \__,_|_|_|_|_| |_||_| dmfh(-2)dmfh.cx


+ Reply to Thread