TCP Blackhole - TCP-IP

This is a discussion on TCP Blackhole - TCP-IP ; Hello all, I work for an company that has a website that our members use to conduct their business. Sometimes we will get complaints about a customer not being able to load our website. A traceroute and ping works fine, ...

+ Reply to Thread
Results 1 to 8 of 8

Thread: TCP Blackhole

  1. TCP Blackhole

    Hello all,

    I work for an company that has a website that our members use to
    conduct their business. Sometimes we will get complaints about a
    customer not being able to load our website. A traceroute and ping
    works fine, but they cannot load the page in their browser using the
    DNS name or IP. A reboot of the customers router or having their ISP
    give them a new IP address resolves the issue. Fair enough... but I
    don't accept this as a solution. Further troubleshooting reveals the
    following:

    Capturing packets on Customers Computer, Our Firewall, Our Server.

    Customer sends SYN, I see the SYN make it in our firewall and sent to
    the server. I see the SYN received by the server. Server sends a
    SYN,ACK to customer. I see SYN,ACK leave our firewall destined for
    customers computer. SYN,ACK never makes it to customers computer.

    Customers computer sends 2 more SYN's. I see these in our firewall and
    server, I see the server respond, I see it leave the firewall, but the
    customer never receives it.

    It sounds like it is being stuck in a blackhole... but why? By who?
    How does it get tagged to be blackholed?

    This is normal HTTP traffic of port 80 and it is RFC compliant. Has
    anyone had any experiences like this? Know of any appliance that does
    such a thing?

    This happens more than it should.. maybe once a month.

    Any thoughts, insight, or opinions are greatly appreciated.

    -Ed

  2. Re: TCP Blackhole

    On Mon, 18 Aug 2008 11:44:01 -0700, Eduard wrote:

    > Hello all,
    >
    > I work for an company that has a website that our members use to conduct
    > their business. Sometimes we will get complaints about a customer not
    > being able to load our website. A traceroute and ping works fine, but
    > they cannot load the page in their browser using the DNS name or IP. A
    > reboot of the customers router or having their ISP give them a new IP
    > address resolves the issue. Fair enough... but I don't accept this as a
    > solution. Further troubleshooting reveals the following:
    >
    > Capturing packets on Customers Computer, Our Firewall, Our Server.
    >
    > Customer sends SYN, I see the SYN make it in our firewall and sent to
    > the server. I see the SYN received by the server. Server sends a SYN,ACK
    > to customer. I see SYN,ACK leave our firewall destined for customers
    > computer. SYN,ACK never makes it to customers computer.
    >
    > Customers computer sends 2 more SYN's. I see these in our firewall and
    > server, I see the server respond, I see it leave the firewall, but the
    > customer never receives it.
    >
    > It sounds like it is being stuck in a blackhole... but why? By who? How
    > does it get tagged to be blackholed?
    >
    > This is normal HTTP traffic of port 80 and it is RFC compliant. Has
    > anyone had any experiences like this? Know of any appliance that does
    > such a thing?
    >
    > This happens more than it should.. maybe once a month.
    >
    > Any thoughts, insight, or opinions are greatly appreciated.


    The one time I saw behavior like this was with a firewall that insisted
    that any packet with a destination address ending in .255 was a broadcast
    address and should be dropped.


    Stoopid piece of ****, it lets the SYN come in and then blocks the
    outgoing SYN-ACK. Well what do you expect, the version before that either
    blocked or passed *all* ICMP messages. MTU problems were rampant.


    I don't expect this is your problem, but its the only thing I can think
    of.

    M4

  3. Re: TCP Blackhole

    Eduard wrote:
    > I work for an company that has a website that our members use to
    > conduct their business. Sometimes we will get complaints about a
    > customer not being able to load our website. A traceroute and ping
    > works fine, but they cannot load the page in their browser using the
    > DNS name or IP. A reboot of the customers router or having their ISP
    > give them a new IP address resolves the issue. Fair enough... but I
    > don't accept this as a solution. Further troubleshooting reveals the
    > following:


    > Capturing packets on Customers Computer, Our Firewall, Our Server.


    > Customer sends SYN, I see the SYN make it in our firewall and sent to
    > the server. I see the SYN received by the server. Server sends a
    > SYN,ACK to customer. I see SYN,ACK leave our firewall destined for
    > customers computer. SYN,ACK never makes it to customers computer.


    > Customers computer sends 2 more SYN's. I see these in our firewall and
    > server, I see the server respond, I see it leave the firewall, but the
    > customer never receives it.


    > It sounds like it is being stuck in a blackhole... but why? By who?
    > How does it get tagged to be blackholed?


    > This is normal HTTP traffic of port 80 and it is RFC compliant. Has
    > anyone had any experiences like this? Know of any appliance that does
    > such a thing?


    > This happens more than it should.. maybe once a month.


    > Any thoughts, insight, or opinions are greatly appreciated.


    Is the customers "router" really one of those switch/NAT/router
    combinations that folks just short-hand to "router?" Is the firmware
    up-to-date on that beast?

    Any way to get traces from both sides of that device?

    rick jones
    --
    The computing industry isn't as much a game of "Follow The Leader" as
    it is one of "Ring Around the Rosy" or perhaps "Duck Duck Goose."
    - Rick Jones
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  4. Re: TCP Blackhole

    In article <3fe48607-c265-407f-bcba-6e3b0fe47830@r35g2000prm.googlegroups.com>,
    Eduard writes:
    > Hello all,
    >
    > I work for an company that has a website that our members use to
    > conduct their business. Sometimes we will get complaints about a
    > customer not being able to load our website. A traceroute and ping
    > works fine, but they cannot load the page in their browser using the
    > DNS name or IP. A reboot of the customers router or having their ISP
    > give them a new IP address resolves the issue.


    Does the customer use a NAT modem? Those things can be buggy.

  5. Re: TCP Blackhole

    >
    > Is the customers "router" really one of those switch/NAT/router
    > combinations that folks just short-hand to "router?" *Is the firmware
    > up-to-date on that beast?
    >


    It is one of those combo devices. I don't know about the firmware. I
    know that if they receive a new IP from the ISP they are able to log
    in.

    > Any way to get traces from both sides of that device?
    >

    I am able to collect traces from the inside, but I would have to work
    with their ISP to get a trace on the other side. I would be interested
    to see if the packets from my servers make it to the Internet facing
    interface or are dropped prior to it.

    What about MTU?

    Thanks again!

    > rick jones
    > --
    > The computing industry isn't as much a game of "Follow The Leader" as
    > it is one of "Ring Around the Rosy" or perhaps "Duck Duck Goose."
    > * * * * * * * * * * * * * * * * * * * * * * * * * * - Rick Jones
    > these opinions are mine, all mine; HP might not want them anyway...
    > feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...



  6. Re: TCP Blackhole

    Eduard wrote:
    > What about MTU?


    Unless it is exceedingly small, or something is trying to wedge data
    in a SYN, SYN|ACK or ACK of SYN|ACK those segments are generally so
    much smaller than the MTU that it isn't a problem. Issues with MTU
    and resulting TCP MSS and IP fragmentation and ICMP's being blocked
    tend to come when the data actually starts to flow.

    rick jones
    --
    The glass is neither half-empty nor half-full. The glass has a leak.
    The real question is "Can it be patched?"
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  7. Re: TCP Blackhole

    On Aug 19, 1:12*pm, v...@calcite.rhyolite.com (Vernon Schryver) wrote:
    > In article <88078e73-5673-406c-a436-32244d0ce...@t1g2000pra.googlegroups.com>,


    > The cause might not be in the customer's router. *The symptoms could
    > happen if the customer's IP address is dynamically assigned and one or
    > more of the ISP's routers still think packets (including SYN-ACKs)
    > should be sent no where or to the previous user of the IP address.
    > Power-cycling the customer's router or NAT box often affects more than
    > just the customer's router or NAT box. *That is forces a new PPP IPCP
    > and probably a DHCP cycle, and either of those might trigger routing
    > updates to the ISP's routers. *The basic idea there is that something
    > like a RIP flash update got lost.
    >

    This clears up some of the issue for me. This makes sense.

    > I would start diagnosing the problem by trying to get all of the boxes
    > in the working path to respond to something while things are broken.
    > If the customer's router or NAT box answers something, whether ICMP
    > echo requests or some flavor of UDP, TCP, or ICMP traceroute while
    > things are broken, then depending on IP addresses and masks assigned
    > to interfaces in the router/NAT box, you might know that the problem
    > is probably in the router/NAT box or the customer's host or internal
    > network. *Of course, if the router/NAT box won't answer remote packets
    > even while things are working, this idea is useless.
    >
    > Finding the cause of the problem without snooping for packets all along
    > the path is likely to be very difficult, especially when all you can
    > do is poke from afar.
    >
    > Even if you do diagnose the problem, I bet you won't be able to fix it.
    > If it's a problem in the customer's ISP's equipment, the customer's ISP
    > will never believe you. *


    Oh, they never do! They always tend to blame someone else... I have to
    send them the packet captures and explain exactly what is happening
    before they will believe it.

    -Ed

  8. Re: TCP Blackhole

    In article ,
    Eduard wrote:

    >> Even if you do diagnose the problem, I bet you won't be able to fix it.
    >> If it's a problem in the customer's ISP's equipment, the customer's ISP
    >> will never believe you.


    >Oh, they never do! They always tend to blame someone else... I have to
    >send them the packet captures and explain exactly what is happening
    >before they will believe it.


    Makers (really end user technical support) of consumer grade "routers"
    are the same or worse.

    Even if they say they believe you, the odds are good that they're being
    agreeable to end a too long phone call or trouble ticket. That goes
    double when you are not a customer of the ISP or box maker, but only
    another vendor with a problem that affects your product. Even if they
    do believe you (and less likely, understand well enough to make their
    belief meaningful), end user technical support people must convince
    people with the power to do something about the problem. If and when
    someone with passwords to ISP routers or write access to router source
    trees is finally convinced, understands the problem, and miraculously
    starts the source or configuration control grinding out a fix, it will
    almost certainly be weeks to many months before the fix reaches your
    customer. If the official fix ever does reach your customer, you'll
    have watch for backsliding. An ISP router misconfiguration often results
    from patching some other issue that your fix will have broken.

    That's not cyncism, but experience including decades on the equipment
    vendor side as someone who had trouble believing bug reports from
    the technical support organization.

    I'm not trying criticise the way things are or even describe them, but
    to emphasize that you are on your own when dealing with problems that
    happen once a month and are fixed by rebooting, and especially when
    you're not the customer who might buy more of whatever is broken.
    --


    Vernon Schryver vjs@rhyolite.com

+ Reply to Thread