Very limited port redirection setup with pf not working - BSD

This is a discussion on Very limited port redirection setup with pf not working - BSD ; Hi, I have a problem with configuring pf and would very much appreciate some help. Some background is needed: I have to migrate some web applications from one machine to another, and I decided to install an OpenBSD box so ...

+ Reply to Thread
Results 1 to 12 of 12

Thread: Very limited port redirection setup with pf not working

  1. Very limited port redirection setup with pf not working

    Hi,

    I have a problem with configuring pf and would very much appreciate
    some help.

    Some background is needed: I have to migrate some web applications from
    one machine to another, and I decided to install an OpenBSD box so I
    can give it the name of the old web apps machine over night (in DNS),
    and then redirect ports gradually from the old web apps machine towards
    the new machine as I migrate the individual web apps living on
    individual ports over the course of some days without any impact on the
    users.

    I won't need any firewall functionality because the entire excercise
    happens inside a firewalled environment, and I don't think I'd need any
    NAT since I have no one-to-many IP relations involved. It should be a
    very straight forward setup with a few lines of pf.conf for all I can
    see, but it doesn't work.

    The old and new web app machine are say 10.4.11.26 and 10.4.10.235
    respectively. Those addresses are usually a 255.255.0.0 network (so,
    both in the same network). My OpenBSD box is 10.4.12.142, which usually
    also has a 16 bit netmask (meaning, all three would be in the same
    network).

    I must admit I am not an expert when it comes to routing, so I have put
    three network cards into the OpenBSD box and hooked all of them up to
    the LAN, and I have configured them as 10.4.11.28/24, 10.4.10.237/24
    and 10.4.12.142/24. So I tell the box to treat each part of what
    usually is a 16 bit network as a class C (24 bit) network.

    I don't know if this is a smart thing to do, but I figured the netmask
    only tells the box which network portion is attached to what interface?
    Essentially it seems that I achieve exactly what I want, I can
    communicate (say, ping) my old web app machine (10.4.11.26, through
    network card #1) from the OpenBSD box, and my new web app machine
    (10.4.10.235, through card #2) as well, and it can reach all other
    clients which are all in the 10.4.12.xxx range.

    Vice versa, both web app boxes can ping my OpenBSD box, and my openbsd
    box can be reached by clients, say 10.4.12.1 for example.

    For short, every machine can talk to the OpenBSD box, and be talked to
    from the OpenBSD box, and OpenBSD knows how to route.

    Then I made a very small pf.conf:

    --- snip ---

    ext_if="xl0" # interface for inbound traffic
    old_if="xl1" # interface for old web apps machine
    new_if="xl2" # interface for new web apps machine
    old_ip="10.4.11.26" # old web apps IP for rdr rules target
    new_ip="10.4.10.235" # new web apps IP for rdr rules target

    set block-policy return

    rdr on $ext_if proto tcp to port 80 -> $old_ip port 80

    pass in keep state
    pass out keep state

    --- snip ---

    This should, me thought, allow a client (say 10.4.12.1) to connect to
    the OpenBSD box itself (10.4.12.142) on port 80 and allow me to
    redirect this to the web server on the old web apps machine, port 80
    (and later, with each software migrated, to redirect to the new app
    servers IP instead).

    But it doesn't work - I can't see the web page from a client at
    10.4.12.1. In tcpdump -i pflog0 -l when adding "log" to the pass out
    and pass in rules, I see that my workstation tries to connect to the
    old web server (successfully redirected it seems), like so:

    17:11:53.730275 myclient.mydomain.com.1059 >
    oldwebsrv.mydomain.com.www: [|tcp] (DF)

    but no reply ever seems to come from the old web server, through the
    OpenBSD box and back to my client.

    I believe that the webserver tries to reply directly to the client at
    that point and the packets get dropped by the client because they're
    out of sequence or so. But I have no idea how to make this work.

    Could anyone with more routing skills kindly offer some advice for the
    above scenario and how to best build a machine that allows me to
    migrate from one web apps machine to another by bending ports one after
    another?

    Many thanks,

    Carsten
    The Netherlands


  2. Re: Very limited port redirection setup with pf not working



    > but no reply ever seems to come from the old web server, through the
    > OpenBSD box and back to my client.
    >
    > I believe that the webserver tries to reply directly to the client at
    > that point and the packets get dropped by the client because they're
    > out of sequence or so. But I have no idea how to make this work.


    it sounds like you need to set a route from your webserver back to the obsd
    box.
    stefan.



  3. Re: Very limited port redirection setup with pf not working

    Hi Stefan, and others,

    First, many thanks for your reply.

    But sorry, I don't quite understand this (being not an expert with
    routing).

    I believed that I set a route to allow one machine to reach another.
    The point at hand is, my webserver can already reach the OpenBSD box.

    Unlike the OpenBSD box, the webserver operates with a 16 bit netmask
    for the 10.4. network that itself is in, so it can reach (ping etc) the
    OpenBSD box at 10.4.12.142 with no problems. So, if the packets coming
    port-redirected from the OpenBSD box would look right, the web server
    should already send a reply back to the OpenBSD box as far as I can
    see?

    Maybe you or someone on the list could kindly explain this proposal and
    maybe even help me out with a sample route command for this scenario -
    it would be greatly appreciated as I have to alter a production
    environment and wouldn't want to bring that down by "trying things
    out".

    Thanks
    Carsten


  4. Re: Very limited port redirection setup with pf not working

    Some more info: It appears that the port-redirected packages from the
    OpenBSD box never reach the webserver in the first place, although the
    OpenBSD box can generally reach the web server! Puzzles me.

    Client IP: 10.4.10.234
    OpenBSD receiving if (to client): 10.4.10.142
    OpenBSD if for redirection to webserver: 10.4.12.237 (xl2)
    Webserver if: 10.4.12.235

    Connecting from the client to port 80 of the openbsd box gets
    redirected to port 80 on webserver fine:

    tcpdump: WARNING: pflog0: no IPv4 address assigned
    tcpdump: listening on pflog0, link-type PFLOG
    10:45:30.690277 10.4.12.234.4399 > 10.4.10.235.80: [|tcp] (DF)

    and on the interface itself (different tcpdump run, thats why different
    high port)

    tcpdump: listening on xl2, link-type EN10MB
    10:38:16.914070 10.4.12.234.4388 > 10.4.10.235.80: S
    2959675908:2959675908(0) wi

    but on the web server the command

    tcpdump -i eth0 -l -n | grep 10.4.10.23 | grep -v ssh

    which should show an incoming connection from 10.4.10.234 that isn't
    ssh, I see nothing incoming at the same time - and that despite being
    able to reach

    10.4.12.235:80

    from the OpenBSD box with lynx or telnet 80.

    I am totally puzzled by this.

    I am wondering now: I could simplify my setup even more and get rid of
    the different subnet masks by using just one network card on the
    OpenBSD box, maybe that solves the problem? But is it possible to use
    just one network card for port redirect? Like saying in pf.conf
    "everything coming in on xl0 on port 80, redirect this to another
    machine in the same subnets port 80 via the same interface"?

    In that case, I could have one card only as 10.4.xxx.xxx/16 that can
    reach all machines at once.

    Or do I need two network cards and/or two different target subnets for
    port redirection to work?

    Many thanks


  5. Re: Very limited port redirection setup with pf not working

    Sorry, have an error in my second-to last port. The correct IP
    addresses are:

    Client IP: 10.4.12.234 OpenBSD receiving if (to client): 10.4.12.142
    OpenBSD if for redirection to webserver: 10.4.10.237 (xl2) Webserver
    if: 10.4.10.235

    Apologies.


  6. Re: Very limited port redirection setup with pf not working

    google@humanprototype.com wrote:
    > Hi Stefan, and others,
    >
    > First, many thanks for your reply.
    >
    > But sorry, I don't quite understand this (being not an expert with
    > routing).

    I'm not an expert either but some simple reasoning:
    If the webserver returns a packet, it has to know a route to the client.
    You say that the webserver knows the OBSD box, but does it know how to reach
    the client ?
    If not so, maybe the webserver needs the OBSD box to be its router (set as
    default gateway or having a routing rule that tells the webserver to send
    packets destined for the client network to the OBSD box). I think that is
    what Stefan meant to say, no ?

    HTH, Jaap

  7. Re: Very limited port redirection setup with pf not working

    wrote in message
    news:1132822831.900534.164920@g47g2000cwa.googlegr oups.com...
    > Sorry, have an error in my second-to last port. The correct IP
    > addresses are:
    >
    > Client IP: 10.4.12.234 OpenBSD receiving if (to client): 10.4.12.142
    > OpenBSD if for redirection to webserver: 10.4.10.237 (xl2) Webserver
    > if: 10.4.10.235


    try this on your webserver:
    route add default 10.4.10.237

    stefan



  8. Re: Very limited port redirection setup with pf not working

    Hi,

    Thanks for the replies.

    Jaap:

    > You say that the webserver knows the
    > OBSD box, but does it know how to reach
    > the client ?


    Yes. The webservers at 10.4.12.235 and 10.4.11.26 know 10.4.x.x as a 16
    bit network, hence it can reach the clients, which are all 10.4.12.x.x.

    The clients are accessing the web server (at least the old one) as part
    of daily business so that routing is fine.

    Stefan:

    Thanks, I understand now what you are suggesting. Still I can't make
    the OpenBSD box the default route for the web server - it needs the
    default route to be set to the company gateway or some of the web apps
    running there will break. I might try this as a last resort thing, but
    I can't really risk breaking the apps for the users.

    I have throught this through in the meantime and I discover further
    flaws in my original plan. E.g. the packets from the client, through
    the OpenBSD box to the web server will leave the OpenBSD box on one
    interface and come back on another (due to my rather "excotic"
    subnetting), this will probably break the statefullness anyway. I don't
    think that statefullness spawns over multiple interfaces.

    I will now try to use one network card in the OpenBSD box, and try to
    use 10.4.x.x/16 on that one so I can address both the client and the
    two web servers with one network card and have all machines in the same
    subnet. I will then try if port forwarding from the OpenBSD box to the
    web server works this way. If that works, its the easier and better
    way. If not, I am lost

    Will keep you all posted. If anyone knows already that the approach
    with one common subnet and port forwarding wouldn't work however, I'd
    be happy to hear about it.

    Cheers,
    Carsten


  9. Re: Very limited port redirection setup with pf not working

    Got it! Has to do with routing, but default gateway didn't help.

    I'll explain:

    I added the default gw as you suggested, but the web server still had a
    way of reaching the client directly (by being in the same 10.4.x.x/16
    network) so it never used the default gateway to reach the clients.

    However, then I added this:

    route add -net 10.4.12.0 netmask 255.255.255.0 gw 10.4.12.142

    to force all packages coming from clients, going through the OpenBSD
    box, and back to the clients on the web server to use the OpenBSD box
    as gateway.

    This works I think because the route above is more stringent (for a 24
    bit subnet) than the 16 bit network direct route it has, and hence ...
    it works - I can go, on my client, to http://10.4.12.142 and see the
    web page the same way I can do by going to http://10.4.12.235.

    Many thanks for all replies.


  10. Re: Very limited port redirection setup with pf not working

    Happy too soon. It works now, I can bend ports each way, and I can
    reach the web servers from clients directly (and the return traffic
    through the OpenBSD box) or through the OpenBSD box entirely. Totally
    my desired setup.

    But I end up getting only half web pages now. It connects, but pages
    are returned half, or miss elements towards the bottom of the page.
    Seems that now I run into to some weird "timing" or sequence number
    problems with the tcp stack

    Lost again ...


  11. Re: Very limited port redirection setup with pf not working

    And this is solved, too.

    Its not possible to connect to the webserver directly with the route
    going back through the OpenBSD box, it gets problems with TCP sequence
    numbers it seems.

    Done some tests now with going through the OpenBSD box exclusively and
    this works fine.

    Hence, I will now first add all ports to be redirected (to the old
    webserver), THEN add the route on the old webserver from it to clients
    in 10.4.12 through the BSD box, then change DNS to give the BSD box the
    name of the old web server, and then gradually migrate my software and
    bend ports.

    All solved - thanks for getting me on the right track with the routing.

    Cheers,
    Carsten


  12. Re: Very limited port redirection setup with pf not working

    > I believed that I set a route to allow one machine to reach another.
    > The point at hand is, my webserver can already reach the OpenBSD box.


    yes, as your webserver and the obsd box are in the same subnet, thats fine.

    > Unlike the OpenBSD box, the webserver operates with a 16 bit netmask
    > for the 10.4. network that itself is in, so it can reach (ping etc) the
    > OpenBSD box at 10.4.12.142 with no problems. So, if the packets coming
    > port-redirected from the OpenBSD box would look right, the web server
    > should already send a reply back to the OpenBSD box as far as I can
    > see?


    no. let me explain.
    the client sends a http request to the obsd box. rdr of obsd replaces the
    destination ip of those packets with the ip of your internal webserver,
    thats ok.
    but, rdr will NOT touch the source ip adress.
    thus, your webserver will have to send a reply targeted to your external
    client. the webserver cant do that, as he would have to adress a subnet
    which he doesnt have a route for.

    so, log on to your webserver and try a default route directed at the
    internal interface of your obsd box.
    route add default 10.4.11.x
    x i dont know, i couldnt find the full ip adress of your obsd box inside the
    10.4.11 subnet in your original posting.
    if your webserver is windows, the command could be different depending on
    version. i have seen this:
    route add 0.0.0.0 mask 0.0.0.0 10.4.11.x

    stefan




+ Reply to Thread