Hello,

I've been experiencing some strange behaviour with my netfilter/
iptables (v1.2.11) setup. I have an Xscale 425 board with kernel
2.4.17 and would I'm having difficulties with the NATting.

The intention is to do NATting for hosts on the eth0 interface, so
that packets either get out through eth1 (DHCP) or through ath0 (also
DHCP).

eth0 has a static IP address 192.168.15.1/24
eth1 obtains an IP address 192.168.3.X/24
ath0 obtains an IP address 10.0.254.X/24

In my /etc/init.d/ directory there is a script named firewall, which
contains the following commands when called with start as argument:

---beginning of the firewall snippet---
/bin/logger "Starting the firewall"
# Disable IP forwarding until the rules are in place.
echo 0 > /proc/sys/net/ipv4/ip_forward

# Flush the tables and delete the non-builtin chains
# to ensure that we are starting from scratch.
{
for i in filter nat mangle
do
$IPT --table $i --flush
$IPT --table $i --delete-chain
done

$IPT --table filter --append FORWARD --in-interface eth0 --jump DROP

$IPT --table nat --append POSTROUTING --jump LOG --log-prefix "n:
POSTROUTING "
$IPT --table mangle --append POSTROUTING --jump LOG --log-prefix "m:
POSTROUTING "

$IPT --table nat --append POSTROUTING --out-interface ath0 --
destination 0/0 --jump MASQUERADE
$IPT --table nat --append POSTROUTING --out-interface eth1 --
destination 0/0 --jump MASQUERADE

$IPT --table filter --append FORWARD --match state --state ESTABLISHED
--jump LOG --log-prefix "f: FORWARD: established "
$IPT --table filter --append FORWARD --proto udp --jump DROP
$IPT --table filter --policy FORWARD ACCEPT

$IPT --table mangle --append FORWARD --jump LOG --log-prefix "m:
FORWARD "

$IPT --table filter --append INPUT --dport !443 --in-interface eth0 --
match state --state INVALID --jump LOG --log-prefix "f: INPUT: invalid
"
$IPT --table filter --append INPUT --dport !443 --in-interface eth0 --
match state --state NEW --jump LOG --log-prefix "f: INPUT: new "
$IPT --table filter --append INPUT --dport !443 --match state --state
RELATED --jump LOG --log-prefix "f: INPUT: related "
$IPT --table filter --append INPUT --dport !443 --match state --state
ESTABLISHED --jump LOG --log-prefix "f: INPUT: established "
$IPT --table filter --append INPUT --proto udp --jump DROP
$IPT --table filter --policy INPUT ACCEPT

$IPT --table mangle --append INPUT --jump LOG --log-prefix "m: INPUT "

$IPT --table nat --append PREROUTING --jump LOG --log-prefix "n:
PREROUTING "
$IPT --table mangle --append PREROUTING --jump LOG --log-prefix "m:
PREROUTING "

$IPT --table filter --append FORWARD --match state --state RELATED --
jump LOG --log-prefix "f: FORWARD: related "
$IPT --table filter --append FORWARD --match state --state NEW --jump
LOG --log-prefix "f: FORWARD: new "
$IPT --table filter --append FORWARD --match state --state INVALID --
jump LOG --log-prefix "f: FORWARD: invalid "
} >> /tmp/fw_debug

echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/ip_dynaddr

$IPT --table filter --delete FORWARD --in-interface eth0 --jump DROP
---end of the firewall snippet---

Note: $IPT has value /usr/bin/iptables

The test is to start a ping before the board is even powered, power
the board, and wait for responses. But they never arrive. Unless I
stop the ping, let it timeout (I guess it is 30 seconds - linked to
conntrack?) and restart it. Then everything is fine.

More in detail:
I have my pc (192.168.15.15) connected to the board on the eth0
interface, and I start a ping to 172.16.19.1, which is reached through
10.0.254.7, which is on the board's ath0 interface, and is configured
with a static route created when the ath0 interface gets its DHCP
lease.
When I do a capture on the target host, I see the ICMP requests coming
in, but with the original (unNATted) source address, and hence replies
don't get delivered correctly:

12800.135975 192.168.15.15 -> 172.16.19.1 ICMP Echo (ping) request
12800.136052 172.16.19.1 -> 192.168.15.15 ICMP Echo (ping) reply
12805.636036 192.168.15.15 -> 172.16.19.1 ICMP Echo (ping) request
12805.636129 172.16.19.1 -> 192.168.15.15 ICMP Echo (ping) reply
12811.136063 192.168.15.15 -> 172.16.19.1 ICMP Echo (ping) request
12811.136157 172.16.19.1 -> 192.168.15.15 ICMP Echo (ping) reply
12816.636105 192.168.15.15 -> 172.16.19.1 ICMP Echo (ping) request
12816.636184 172.16.19.1 -> 192.168.15.15 ICMP Echo (ping) reply

The situation remains like this forever. The conntrack entries prove
that things are going wrong:
xs$ cat /proc/net/ip_conntrack
icmp 1 28 src=192.168.15.15 dst=172.16.19.1 type=8 code=0 id=1280
[UNREPLIED] src=172.16.19.1 dst=192.168.15.15 type=0 code=0 id=1280
use=1
tcp 6 431999 ESTABLISHED src=192.168.15.15 dst=192.168.15.1
sport=4010 dport=23 src=192.168.15.1 dst=192.168.15.15 sport=23
dport=4010 [ASSURED] use=1

If I stop the ping, let it timeout during 30 seconds (which is -
probably not coincidentally - the conntrack timeout) and start it
again, packets arrive with the source address translated:
13039.912480 10.0.254.76 -> 172.16.19.1 ICMP Echo (ping) request
13039.914390 172.16.19.1 -> 10.0.254.76 ICMP Echo (ping) reply
13040.914396 10.0.254.76 -> 172.16.19.1 ICMP Echo (ping) request
13040.914470 172.16.19.1 -> 10.0.254.76 ICMP Echo (ping) reply

Also, the rules in the script should clearly show which path the
packets are taking thorugh the different chains and tables. One would
expect that packets for forwarding proceed like this (as found in the
iptables tutorial):
raw PREROUTING -> mangle PREROUTING -> nat PREROUTING -> mangle
FORWARD -> filter FORWARD -> mangle POSTROUTING -> nat POSTROUTING

But, the only logged entries found are these:

m: PREROUTING IN=eth0 OUT= MAC=00:07:d5:01:39:c1:00:50:da:
56:1b:c9:08:00 SRC=192.168.15.15 DST=172.16.19.1 LEN=60 TOS=0x00
PREC=0x00 TTL=128 ID=10391 PROTO=ICMP TYPE=8 CODE=0 ID=1280
SEQ=13827
f: FORWARD: new IN=eth0 OUT=ath0 SRC=192.168.15.15 DST=172.16.19.1
LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=10391 PROTO=ICMP TYPE=8 CODE=0
ID=1280 SEQ=13827

It seems to me the POSTROUTING chain is not traversed in this case.
Even more: the whole nat table is not applying.

When looking at the logged entries for a successful ping, a whole lot
more can be seen:
m: PREROUTING IN=eth0 OUT= MAC=00:07:d5:01:39:c1:00:50:da:
56:1b:c9:08:00 SRC=192.168.15.15 DST=172.16.19.1 LEN=60 TOS=0x00
PREC=0x00 TTL=128 ID=10839 PROTO=ICMP TYPE=8 CODE=0 ID=1280 SEQ=14595
n: PREROUTING IN=eth0 OUT= MAC=00:07:d5:01:39:c1:00:50:da:
56:1b:c9:08:00 SRC=192.168.15.15 DST=172.16.19.1 LEN=60 TOS=0x00
PREC=0x00 TTL=128 ID=10839 PROTO=ICMP TYPE=8 CODE=0 ID=1280
SEQ=14595
f: FORWARD: new IN=eth0 OUT=ath0 SRC=192.168.15.15 DST=172.16.19.1
LEN=60 TOS=0x00 PREC=0x00 TTL=127 ID=10839 PROTO=ICMP TYPE=8 CODE=0
ID=1280 SEQ=14595
n: POSTROUTING IN= OUT=ath0 SRC=192.168.15.15 DST=172.16.19.1 LEN=60
TOS=0x00 PREC=0x00 TTL=127 ID=10839 PROTO=ICMP TYPE=8 CODE=0 ID=1280
SEQ=14595
m: PREROUTING IN=ath0 OUT= MAC=00:0e:8e:07:ba:27:00:c0:df:0f:91:7e:
08:00 SRC=172.16.19.1 DST=10.0.254.76 LEN=60 TOS=0x00 PREC=0x00 TTL=64
ID=65223 PROTO=ICMP TYPE=0 CODE=0 ID=1280 SEQ=14595

Does anybody have any idea what I'm missing here? This problem has
been keeping me awake for some time now, but I haven't made any
improvement. I've tried reversing the insertion of the iptables rules,
doing a "pre-drop" when rules aren't ready yet and removing it in the
end, sleeping 2 seconds before anabling IP forwarding, ... nothing
seems to solve the behaviour.

All tips are very very welcome.
Thank you.

Jarl