I'm still working on porting our unicast stream servers behind PF
firewall runs on openbsd3.8.
That is a hidden, bridged firewall.

Now, Im testing with only one server, which is streaming from tcp port
443 to nearly 500-1000 instannt browser embedded java clients.
Traffic makes up to 2 ~ 3Mbps now, but in real envriron. it wil take
up to 20Mbps

At begginning, there was high numerical differences among the OS's
ESTABLISHED netstat table, # of states in PF and # of connected hosts
of out server application, wihch is stable for years.
and my state table overwhelmed.

I customized PF rules as follows. And Differences has become
negligable.

FURTHERMORE, I have noticed some unexpected BLOCKs at pflogd0
Because the traffic, I have targetted to pass over PF is 10 times
bigger than the situation at now;
I'm cautions about those BLOCKED PACKETS.

There are 2 sample symthoms below ,

symptom #1:
rule 10/(match) block in on fxp0: 212.175.131.3.48012 >server1.443: F
0:0(0) ack 1 win 8621
rule 10/(match) block in on fxp0: 85.103.159.194.1722 > server1.443: P
22321829:22321963(134) ack 1951655540 win 16744
rule 10/(match) block in on fxp0: 88.225.20.206.11274 > server1.443: R
2749563789:2749563789(0) win 64631

esp. This host one more connection on state table
rule 10/(match) block in on fxp0: 85.104.140.148.1113 > server1.443: F
2282441566:2282441566(0) ack 646237210 win 65020
and 4 more blocks with ( F 0:0(0) ack 1 win 65020)


symptom #2:
rule 11/(match) block out on fxp0: server1.443 > 85.100.93.186.2522:
[|tcp] (DF)

# Now I block 1 ~ 5 connections per minute with avg 2 conn pr min
( I'm not sure those have SYN flag set. I hope those have not and
dont know how to investigate


Especially the first one!
I explaind the secound one , BLOCK OUT, with the time limits i have
custmized on tcp options of the rule ( tcp.established 300)
because my box has already killed the state,because of unactivity OS
behind FW tries to sending data.
-- because my server streams realtime quatos, 5 min inactivity is
unusal. ---


BUT, i have sone doubts about the first one related to INBOUND BLOCKS
-- sometimes those hosts ( blocked) has already one or more states in
my table ( but it is verry common , more than one client sharing the
same LAN connect to our servers). But blocked source ports are
different from the ones in state table, - ofcourse -
-- some of the hosts could never establish a connection yet, I have
checked that those are all acknowledged. Now, they are trying but
has not complaint yet
-- none of the limits have exceeded. (max, max-src-states,
max-src-conn ..etc)


What may cause these problems ?
-- QUEUE limits ?
-- TCP options ( S/SA) ?
-- or what can else , which I have missed ?

Thank you for your help and forgive my this long msg takes time.


ilker.arab...@gmail.com
App. Developer



#pftop -vqueue
QUEUE BW SCH PRIO PKTS BYTES DROP_P
DROP_B QLEN BORROW SUSPEN P/S B/S
any_eurojava 14M cbq 2 2644516 558110K
6 1019 0 45521 11386 1314 403213



Some of My pf.conf

Options
-------------------
set block-policy drop
set state-policy floating

#set optimization aggressive
set optimization normal

set limit states 100000

set timeout { tcp.first 120, tcp.opening 30, tcp.established 3600}
set timeout { tcp.closing 300, tcp.finwait 45, tcp.closed 60 }
set timeout { udp.first 60, udp.single 30, udp.multiple 60 }
set timeout { icmp.first 20, icmp.error 10 }
set timeout { other.first 60, other.single 30, other.multiple 60 }
set timeout { adaptive.start 10000, adaptive.end 30000 }

set loginterface $ext_if

set skip on lo0
set skip on $int_if

scrub in on $ext_if all no-df
scrub on $ext_if all reassemble tcp

################# QUEUEING CLASSES ################
altq on $ext_if cbq bandwidth 20Mb queue {
internet,throttled,firewall,secure_out,trusted_in}

# Main Internet services of servers -
queue internet bandwidth 16Mb {any_eurojava,any_web,any_commons}
# EuroJava Service for any customers
queue any_eurojava bandwidth 14Mb priority 2 qlimit 500
cbq(borrow ecn)
queue any_web bandwidth 1Mb priority 4 cbq(borrow ecn)
queue any_commons bandwidth 1Mb priority 4 cbq(default borrow
ecn)
queue firewall bandwidth 1Mb priority 1 cbq(borrow ecn) #
firewall management queue
# secure servers, behind FW , established out queue
queue secure_out bandwidth 1Mb priority 5 cbq(borrow ecn)
# trusted hosts' traffic on un-common services
queue trusted_in bandwidth 1Mb priority 3 cbq(borrow ecn)
# throttled access for flooding hosts on throttled queuee
queue throttled bandwidth 1Mb priority 7 cbq(red)
################################################## ######
## RELATED RULES TO MY PROBLEM

# Rule 10
block in log on $ext_if all


# Rule 11
block out log on $ext_if all

#
block in log quick on $ext_if from to any

#

# my main PASS rule for tcp in to port 443

pass in on $ext_if proto tcp from any to port
{$eurojava_ports} flags S/SA \
tag CHECK_MORE keep state ( max 5000, max-src-states 100, \
tcp.established 300, tcp.closing 10, tcp.finwait 10, tcp.closed
10 \
max-src-conn 100, overload ) \
queue any_eurojava
# CHECK rule for flooting customers
pass in on $ext_if from tagged CHECK_MORE tag THROTTLED
keep state \
( max 500, tcp.established 60, tcp.closing 10, tcp.closed 5,
tcp.finwait 10 \
max-src-conn-rate 50/5, overload flush global)
\
queue throttled


## OUT BOUND TRAFFIC OF SECURED SERVERS
pass out on $ext_if proto tcp from to any flags S/SA modulate
state queue secure_out