ipfw drive me crazy - BSD

This is a discussion on ipfw drive me crazy - BSD ; Hello I observe a strange behavior with ipfw/natd and fwd command. the same packet , fwd to a same address, use a different outgoing interface if it is nated. Here is the sample: freebsd, 3 interfaces : - bge1: 10.10.21.2 ...

+ Reply to Thread
Results 1 to 7 of 7

Thread: ipfw drive me crazy

  1. ipfw drive me crazy

    Hello

    I observe a strange behavior with ipfw/natd and fwd command.
    the same packet , fwd to a same address, use a different outgoing interface
    if it is nated.

    Here is the sample:

    freebsd, 3 interfaces :
    - bge1: 10.10.21.2 connected to a local LAN 10.10.21/0
    - bge0: switch: 10.10.20.1 for DMZ, and 192.168.0.101 for a new internet
    router at 192.168.0.254
    - tun0: internet public address for a PPP adsl modem

    the freebsd is the default gateway for 10.10.21/24 network
    the tun0 is the default gateway interface inside the freebsd.

    the test i run is :
    10.10.21.1 request a http connection on port 8080
    without additionnal config, the request will come in bge1, go out tun0

    i try to trap the request to use 192.168.0.254 gateway
    the target ip 192.168.0.254 is on a lan connected to bge0.

    ipfw add 00150 fwd 192.168.0.254 log ip from any to any dst-port 8080

    the test was successfull, tcpdump show that :
    - incoming packet from 10.10.21.1 to external ip, 8080 on bge1
    - outgoing packet to 192.168.0.254 via bge0
    just a little strange behavior in ifpw log which show outgoing packet on
    tun0
    i think it's strange because 192.168.0.254 is on lan connected to bge0 wich
    have ip 192.168.0.101

    so now, just missing to nat the incoming packet
    ipfw add 00149 divert 3617 log ip from 10.10.0.0/16 to not 10.10.0.0/16
    dst-port 8080

    the test now give me headache.
    The log show than the packet is well catched & diverted, with same strange
    behavior: out via tun0
    (strange because the target ip 192.168.0.254 is on a lan connected to bge0)
    and tcpdump show:
    - incoming packet from 10.10.21.1 to external ip, 8080 on bge1
    - outgoing natd packet to 192.168.0.254 via tun0 (instead of bge0???)

    so for resume:
    if i do nothing, a packet in bge1 is going out on tun9
    i want to catch packet in bge1, fwd to gateway on bge0
    - if i just fwd, tcpdump say : ok it work, ipfw say: fwd is ok but on wrong
    interface
    - if i fwd&nat, tcpdump say : wrong interface, ipfw say: fwd is ok but on
    wrong interface

    and now ive got headache


    log from my test follow:
    ================================================== ===========
    interface:
    bge0: flags=8943 mtu 1500
    options=1a
    inet 10.10.20.1 netmask 0xffffff00 broadcast 10.10.20.255
    inet 192.168.0.101 netmask 0xffffff00 broadcast 192.168.0.255
    ether 00:30:48:88:5f:f2
    media: Ethernet autoselect (100baseTX )
    status: active
    bge1: flags=8843 mtu 1500
    options=1a
    inet 10.10.21.2 netmask 0xffffff00 broadcast 10.10.21.255
    ether 00:30:48:88:5f:f3
    media: Ethernet autoselect (100baseTX )
    status: active
    tun0: flags=8151 mtu 1492
    inet 80.11.76.251 --> 80.11.76.129 netmask 0xffffffff


    Route:
    Destination,Gateway, interface
    default, AMarseille-111-1-3, tun0
    10.10.1/24, link#3,em0
    10.10.20/24,link#1,bge0
    10.10.21/24, link#2,bge1
    AMarseille-111-1-3,AMarseille-111-1-3,tun0
    192.168.0,link#1,bge0

    natd:
    natd -o 3617 -alias_address 192.168.0.101
    (i used -o port because i m going to nat packet on input interface, and the
    option -reverse of natd cause core dump)

    ================================================== ==================
    test #1: 10.10.21.1 request a tcp to 8080 on a external web, forward every
    incoming packet for 8080
    ipfw add 00150 fwd 192.168.0.254 log ip from any to any dst-port 8080

    log:
    Mar 23 09:11:50 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP
    10.10.21
    ..1:3798 194.167.78.73:8080 in via bge1
    Mar 23 09:11:50 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP
    10.10.21
    ..1:3798 194.167.78.73:8080 out via tun0

    i do not understand why there is 2 lines for a single packet, and why is
    show tun0 as out interface
    but trace with tcpdump show that the packet :
    - is coming in bge1, out bge0, nothing as tun0
    (192.168.0.254 is connected on bge0)

    tcppdump bge1:
    09:11:50.131590 00:e0:18:53:3b:a1 > 00:30:48:88:5f:f3, ethertype IPv4
    (0x0800),
    length 60: IP 10.10.21.1.3798 > 194.167.78.73.8080: S
    1223470518:1223470518(0) w
    in 8192
    09:11:53.391659 00:e0:18:53:3b:a1 > 00:30:48:88:5f:f3, ethertype IPv4
    (0x0800),
    length 60: IP 10.10.21.1.3798 > 194.167.78.73.8080: S
    1223470518:1223470518(0) w
    in 8192

    tcpdump bge0
    09:11:50.132040 00:30:48:88:5f:f2 > 00:07:cb:24:2b:c8, ethertype IPv4
    (0x0800),
    length 58: IP 10.10.21.1.3798 > 194.167.78.73.8080: S
    1223470518:1223470518(0) w
    in 8192
    09:11:53.391740 00:30:48:88:5f:f2 > 00:07:cb:24:2b:c8, ethertype IPv4
    (0x0800),
    length 58: IP 10.10.21.1.3798 > 194.167.78.73.8080: S
    1223470518:1223470518(0) w
    in 8192

    tcpdump tun0: nothing

    ================================================== ==================
    test #2: same as test#1
    ipfw add 00150 fwd 192.168.0.254 log ip from any to any dst-port 8080

    and added natd before forwarding
    ipfw 00149 divert 3617 log ip from 10.10.0.0/16 to not 10.10.0.0/16 dst-port
    8080

    log:
    Mar 23 09:09:14 servidea kernel: ipfw: 149 Divert 3617 TCP 10.10.21.1:3757
    194.1
    67.78.73:8080 in via bge1
    Mar 23 09:09:14 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP
    192.168.
    0.101:3757 194.167.78.73:8080 in via bge1
    Mar 23 09:09:14 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP
    192.168.
    0.101:3757 194.167.78.73:8080 out via tun0
    Mar 23 09:09:17 servidea kernel: ipfw: 149 Divert 3617 TCP 10.10.21.1:3757
    194.1
    67.78.73:8080 in via bge1
    Mar 23 09:09:17 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP
    192.168.
    0.101:3757 194.167.78.73:8080 in via bge1
    Mar 23 09:09:17 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP
    192.168.
    0.101:3757 194.167.78.73:8080 out via tun0

    still same strange thing: 1 packet in bge1 cause 2 line forward

    tcpdump: bge1:
    09:09:14.346305 00:e0:18:53:3b:a1 > 00:30:48:88:5f:f3, ethertype IPv4
    (0x0800),
    length 60: IP 10.10.21.1.3757 > 194.167.78.73.8080: S
    1223469999:1223469999(0) w
    in 8192
    09:09:17.528385 00:e0:18:53:3b:a1 > 00:30:48:88:5f:f3, ethertype IPv4
    (0x0800),
    length 60: IP 10.10.21.1.3757 > 194.167.78.73.8080: S
    1223469999:1223469999(0) w
    in 8192

    tcpdump bge0: nothing

    tcpdump tun0:
    09:09:14.346459 AF 2 44: IP 192.168.0.101.3757 > 194.167.78.73.8080: S
    122346999
    9:1223469999(0) win 8192
    09:09:17.528522 AF 2 44: IP 192.168.0.101.3757 > 194.167.78.73.8080: S
    122346999
    9:1223469999(0) win 8192





  2. Re: ipfw drive me crazy

    patrice wrote:
    > Hello
    >
    > I observe a strange behavior with ipfw/natd and fwd command.
    > the same packet , fwd to a same address, use a different outgoing interface
    > if it is nated.
    >


    This is driving mee crazy too...

    > Here is the sample:
    >
    > freebsd, 3 interfaces :
    > - bge1: 10.10.21.2 connected to a local LAN 10.10.21/0
    > - bge0: switch: 10.10.20.1 for DMZ, and 192.168.0.101 for a new internet
    > router at 192.168.0.254
    > - tun0: internet public address for a PPP adsl modem
    >
    > the freebsd is the default gateway for 10.10.21/24 network
    > the tun0 is the default gateway interface inside the freebsd.
    >
    > the test i run is :
    > 10.10.21.1 request a http connection on port 8080
    > without additionnal config, the request will come in bge1, go out tun0
    >
    > i try to trap the request to use 192.168.0.254 gateway
    > the target ip 192.168.0.254 is on a lan connected to bge0.
    >
    > ipfw add 00150 fwd 192.168.0.254 log ip from any to any dst-port 8080
    >
    > the test was successfull, tcpdump show that :
    > - incoming packet from 10.10.21.1 to external ip, 8080 on bge1
    > - outgoing packet to 192.168.0.254 via bge0
    > just a little strange behavior in ifpw log which show outgoing packet on
    > tun0
    > i think it's strange because 192.168.0.254 is on lan connected to bge0 wich
    > have ip 192.168.0.101
    >
    > so now, just missing to nat the incoming packet
    > ipfw add 00149 divert 3617 log ip from 10.10.0.0/16 to not 10.10.0.0/16
    > dst-port 8080
    >
    > the test now give me headache.
    > The log show than the packet is well catched & diverted, with same strange
    > behavior: out via tun0
    > (strange because the target ip 192.168.0.254 is on a lan connected to bge0)
    > and tcpdump show:
    > - incoming packet from 10.10.21.1 to external ip, 8080 on bge1
    > - outgoing natd packet to 192.168.0.254 via tun0 (instead of bge0???)
    >
    > so for resume:
    > if i do nothing, a packet in bge1 is going out on tun9
    > i want to catch packet in bge1, fwd to gateway on bge0
    > - if i just fwd, tcpdump say : ok it work, ipfw say: fwd is ok but on wrong
    > interface
    > - if i fwd&nat, tcpdump say : wrong interface, ipfw say: fwd is ok but on
    > wrong interface
    >
    > and now ive got headache
    >
    >
    > log from my test follow:
    > ================================================== ===========
    > interface:
    > bge0: flags=8943 mtu 1500
    > options=1a
    > inet 10.10.20.1 netmask 0xffffff00 broadcast 10.10.20.255
    > inet 192.168.0.101 netmask 0xffffff00 broadcast 192.168.0.255
    > ether 00:30:48:88:5f:f2
    > media: Ethernet autoselect (100baseTX )
    > status: active
    > bge1: flags=8843 mtu 1500
    > options=1a
    > inet 10.10.21.2 netmask 0xffffff00 broadcast 10.10.21.255
    > ether 00:30:48:88:5f:f3
    > media: Ethernet autoselect (100baseTX )
    > status: active
    > tun0: flags=8151 mtu 1492
    > inet 80.11.76.251 --> 80.11.76.129 netmask 0xffffffff
    >
    >
    > Route:
    > Destination,Gateway, interface
    > default, AMarseille-111-1-3, tun0
    > 10.10.1/24, link#3,em0
    > 10.10.20/24,link#1,bge0
    > 10.10.21/24, link#2,bge1
    > AMarseille-111-1-3,AMarseille-111-1-3,tun0
    > 192.168.0,link#1,bge0
    >
    > natd:
    > natd -o 3617 -alias_address 192.168.0.101
    > (i used -o port because i m going to nat packet on input interface, and the
    > option -reverse of natd cause core dump)
    >
    > ================================================== ==================
    > test #1: 10.10.21.1 request a tcp to 8080 on a external web, forward every
    > incoming packet for 8080
    > ipfw add 00150 fwd 192.168.0.254 log ip from any to any dst-port 8080
    >
    > log:
    > Mar 23 09:11:50 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP
    > 10.10.21
    > .1:3798 194.167.78.73:8080 in via bge1
    > Mar 23 09:11:50 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP
    > 10.10.21
    > .1:3798 194.167.78.73:8080 out via tun0


    The man page (FreeBSD 4.5 that i have on paper - I'm running 6.2) say

    `If the IP is not a local address then the port number (if specified) is
    ignored and the rule only applies to packets leaving the system.'

    Strange is'nt it. In the man of 6.2 this restriction is not mentioned. I
    think maybe a bug is lingering here, the first line of the log don't
    really apply the forward but log the packet anyway and when the packet
    try to leave via tun0, the forward is applied.

    any comment ....

    Henri
    >
    > i do not understand why there is 2 lines for a single packet, and why is
    > show tun0 as out interface
    > but trace with tcpdump show that the packet :
    > - is coming in bge1, out bge0, nothing as tun0
    > (192.168.0.254 is connected on bge0)
    >
    > tcppdump bge1:
    > 09:11:50.131590 00:e0:18:53:3b:a1 > 00:30:48:88:5f:f3, ethertype IPv4
    > (0x0800),
    > length 60: IP 10.10.21.1.3798 > 194.167.78.73.8080: S
    > 1223470518:1223470518(0) w
    > in 8192
    > 09:11:53.391659 00:e0:18:53:3b:a1 > 00:30:48:88:5f:f3, ethertype IPv4
    > (0x0800),
    > length 60: IP 10.10.21.1.3798 > 194.167.78.73.8080: S
    > 1223470518:1223470518(0) w
    > in 8192
    >
    > tcpdump bge0
    > 09:11:50.132040 00:30:48:88:5f:f2 > 00:07:cb:24:2b:c8, ethertype IPv4
    > (0x0800),
    > length 58: IP 10.10.21.1.3798 > 194.167.78.73.8080: S
    > 1223470518:1223470518(0) w
    > in 8192
    > 09:11:53.391740 00:30:48:88:5f:f2 > 00:07:cb:24:2b:c8, ethertype IPv4
    > (0x0800),
    > length 58: IP 10.10.21.1.3798 > 194.167.78.73.8080: S
    > 1223470518:1223470518(0) w
    > in 8192
    >
    > tcpdump tun0: nothing
    >
    > ================================================== ==================
    > test #2: same as test#1
    > ipfw add 00150 fwd 192.168.0.254 log ip from any to any dst-port 8080
    >
    > and added natd before forwarding
    > ipfw 00149 divert 3617 log ip from 10.10.0.0/16 to not 10.10.0.0/16 dst-port
    > 8080
    >
    > log:
    > Mar 23 09:09:14 servidea kernel: ipfw: 149 Divert 3617 TCP 10.10.21.1:3757
    > 194.1
    > 67.78.73:8080 in via bge1
    > Mar 23 09:09:14 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP
    > 192.168.
    > 0.101:3757 194.167.78.73:8080 in via bge1
    > Mar 23 09:09:14 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP
    > 192.168.
    > 0.101:3757 194.167.78.73:8080 out via tun0
    > Mar 23 09:09:17 servidea kernel: ipfw: 149 Divert 3617 TCP 10.10.21.1:3757
    > 194.1
    > 67.78.73:8080 in via bge1
    > Mar 23 09:09:17 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP
    > 192.168.
    > 0.101:3757 194.167.78.73:8080 in via bge1
    > Mar 23 09:09:17 servidea kernel: ipfw: 150 Forward to 192.168.0.254 TCP
    > 192.168.
    > 0.101:3757 194.167.78.73:8080 out via tun0
    >
    > still same strange thing: 1 packet in bge1 cause 2 line forward
    >
    > tcpdump: bge1:
    > 09:09:14.346305 00:e0:18:53:3b:a1 > 00:30:48:88:5f:f3, ethertype IPv4
    > (0x0800),
    > length 60: IP 10.10.21.1.3757 > 194.167.78.73.8080: S
    > 1223469999:1223469999(0) w
    > in 8192
    > 09:09:17.528385 00:e0:18:53:3b:a1 > 00:30:48:88:5f:f3, ethertype IPv4
    > (0x0800),
    > length 60: IP 10.10.21.1.3757 > 194.167.78.73.8080: S
    > 1223469999:1223469999(0) w
    > in 8192
    >
    > tcpdump bge0: nothing
    >
    > tcpdump tun0:
    > 09:09:14.346459 AF 2 44: IP 192.168.0.101.3757 > 194.167.78.73.8080: S
    > 122346999
    > 9:1223469999(0) win 8192
    > 09:09:17.528522 AF 2 44: IP 192.168.0.101.3757 > 194.167.78.73.8080: S
    > 122346999
    > 9:1223469999(0) win 8192
    >
    >
    >
    >


  3. Re: ipfw drive me crazy

    In article Henri Hennebert
    writes:
    >patrice wrote:
    >>
    >> so for resume:
    >> if i do nothing, a packet in bge1 is going out on tun9
    >> i want to catch packet in bge1, fwd to gateway on bge0
    >> - if i just fwd, tcpdump say : ok it work, ipfw say: fwd is ok but on wrong
    >> interface
    >> - if i fwd&nat, tcpdump say : wrong interface, ipfw say: fwd is ok but on
    >> wrong interface


    It can probably be made to work with natd too, but I would suggest that
    you try to do the nat'ing with ipnat (from ipfilter) instead - that way
    the "policy routing" and the nat'ing is clearly seperated, and you don't
    have to try to figure out the complex "divert and come back and do you
    need to do the fwd before or after or both":-) logic.

    >The man page (FreeBSD 4.5 that i have on paper - I'm running 6.2) say
    >
    >`If the IP is not a local address then the port number (if specified) is
    >ignored and the rule only applies to packets leaving the system.'
    >
    >Strange is'nt it. In the man of 6.2 this restriction is not mentioned. I
    >think maybe a bug is lingering here, the first line of the log don't
    >really apply the forward but log the packet anyway and when the packet
    >try to leave via tun0, the forward is applied.


    I don't know if this has anything to do with the original problem, but
    that line is still there in the 6.2 man page at www.freebsd.org at
    least. But it refers to the port you can specify as a parameter to
    "fwd":

    fwd | forward ipaddr[,port]
    ^^^^^

    - i.e. it has nothing to do with any ports you may specify in the rule
    body, to have it match specfic packets. This port allows you to forward
    packets to a different port on the same host - this would typically be
    used in a "transparent proxy" setup. Since, as the man page says, "The
    fwd action does not change the contents of the packet at all", there's
    no way you could forward a packet to a different port on a different
    host - this is what the "port is ignored" text is about.

    --Per Hedeland
    per@hedeland.org

  4. Re: ipfw drive me crazy

    Per Hedeland wrote:
    > In article Henri Hennebert
    > writes:
    >> patrice wrote:
    >>> so for resume:
    >>> if i do nothing, a packet in bge1 is going out on tun9
    >>> i want to catch packet in bge1, fwd to gateway on bge0
    >>> - if i just fwd, tcpdump say : ok it work, ipfw say: fwd is ok but on wrong
    >>> interface
    >>> - if i fwd&nat, tcpdump say : wrong interface, ipfw say: fwd is ok but on
    >>> wrong interface

    >
    > It can probably be made to work with natd too, but I would suggest that
    > you try to do the nat'ing with ipnat (from ipfilter) instead - that way
    > the "policy routing" and the nat'ing is clearly seperated, and you don't
    > have to try to figure out the complex "divert and come back and do you
    > need to do the fwd before or after or both":-) logic.
    >
    >> The man page (FreeBSD 4.5 that i have on paper - I'm running 6.2) say
    >>
    >> `If the IP is not a local address then the port number (if specified) is
    >> ignored and the rule only applies to packets leaving the system.'


    My point was not about the port but `leaving the system'.

    I realize after posting that 5.3 (I believe it is the version that
    patrice is running - see other threads) is using ipfw2 witch is a new
    implementation and so the restriction about `leaving' is probably not there.

    Anyway this trace is strange, at least for me.

    >>
    >> Strange is'nt it. In the man of 6.2 this restriction is not mentioned. I
    >> think maybe a bug is lingering here, the first line of the log don't
    >> really apply the forward but log the packet anyway and when the packet
    >> try to leave via tun0, the forward is applied.

    >
    > I don't know if this has anything to do with the original problem, but
    > that line is still there in the 6.2 man page at www.freebsd.org at
    > least. But it refers to the port you can specify as a parameter to
    > "fwd":
    >
    > fwd | forward ipaddr[,port]
    > ^^^^^
    >
    > - i.e. it has nothing to do with any ports you may specify in the rule
    > body, to have it match specfic packets. This port allows you to forward
    > packets to a different port on the same host - this would typically be
    > used in a "transparent proxy" setup. Since, as the man page says, "The
    > fwd action does not change the contents of the packet at all", there's
    > no way you could forward a packet to a different port on a different
    > host - this is what the "port is ignored" text is about.


    I agree

    Henri
    >
    > --Per Hedeland
    > per@hedeland.org


  5. Re: ipfw drive me crazy

    "Henri Hennebert" a écrit dans le message de
    news:eu33gi$2r83$1@morzine.restart.bel...
    > This is driving mee crazy too...
    >


    found, no crazy inside, just bug:
    forwarding packet with local ip src does not work in 5.3...

    http://www.freebsd.org/cgi/query-pr....71910&cat=kern

    have to patch, rebuild, and retest...



  6. Re: ipfw drive me crazy

    patrice wrote:
    > "Henri Hennebert" a écrit dans le message de
    > news:eu33gi$2r83$1@morzine.restart.bel...
    >> This is driving mee crazy too...
    >>

    >
    > found, no crazy inside, just bug:
    > forwarding packet with local ip src does not work in 5.3...
    >
    > http://www.freebsd.org/cgi/query-pr....71910&cat=kern
    >
    > have to patch, rebuild, and retest...
    >
    >

    Thank you for this follow-up

    Henri

  7. Re: ipfw drive me crazy

    "patrice" a écrit dans le message de
    news:4610f7b9$0$2484$426a34cc@news.free.fr...
    > found, no crazy inside, just bug:
    > forwarding packet with local ip src does not work in 5.3...
    >
    > http://www.freebsd.org/cgi/query-pr....71910&cat=kern
    >
    > have to patch, rebuild, and retest...
    >


    ok, with patch; all is fine.

    so for routing packet on an alternate gateway 192.168.0.254 (not the default
    one), connected to 192.168.0.101 (interface bge0)
    the LAN is 10.10/16
    # nat
    natd -i 3615 -o 3616 -alias_address 192.168.0.101
    # de-nat returning packet
    ipfw add divert 3615 ip from any to 192.168.0.101 in via bge0
    # nat packet we want to send to alternate gateway
    ipfw add divert 3616 ip from any to not 10.10.0.0/16 80, want>
    # send nated packet to gateway
    ipfw add fwd 192.168.0.254 ip from 192.168.0.101 to not me 80, written previous line>

    if you add some "always redirected ip" with route add 192.168.0.254 :
    #nat outgoing packet
    ipfw add divert 3616 all from 10.10.0.0/16 to not 10.10.0.0/16 out via bge0





+ Reply to Thread