PF with 20Mbps data streaming - BSD

This is a discussion on PF with 20Mbps data streaming - BSD ; I'm still working on porting our unicast stream servers behind PF firewall runs on openbsd3.8. That is a hidden, bridged firewall. Now, Im testing with only one server, which is streaming from tcp port 443 to nearly 500-1000 instannt browser ...

+ Reply to Thread
Results 1 to 4 of 4

Thread: PF with 20Mbps data streaming

  1. PF with 20Mbps data streaming

    I'm still working on porting our unicast stream servers behind PF
    firewall runs on openbsd3.8.
    That is a hidden, bridged firewall.

    Now, Im testing with only one server, which is streaming from tcp port
    443 to nearly 500-1000 instannt browser embedded java clients.
    Traffic makes up to 2 ~ 3Mbps now, but in real envriron. it wil take
    up to 20Mbps

    At begginning, there was high numerical differences among the OS's
    ESTABLISHED netstat table, # of states in PF and # of connected hosts
    of out server application, wihch is stable for years.
    and my state table overwhelmed.

    I customized PF rules as follows. And Differences has become
    negligable.

    FURTHERMORE, I have noticed some unexpected BLOCKs at pflogd0
    Because the traffic, I have targetted to pass over PF is 10 times
    bigger than the situation at now;
    I'm cautions about those BLOCKED PACKETS.

    There are 2 sample symthoms below ,

    symptom #1:
    rule 10/(match) block in on fxp0: 212.175.131.3.48012 >server1.443: F
    0:0(0) ack 1 win 8621
    rule 10/(match) block in on fxp0: 85.103.159.194.1722 > server1.443: P
    22321829:22321963(134) ack 1951655540 win 16744
    rule 10/(match) block in on fxp0: 88.225.20.206.11274 > server1.443: R
    2749563789:2749563789(0) win 64631

    esp. This host one more connection on state table
    rule 10/(match) block in on fxp0: 85.104.140.148.1113 > server1.443: F
    2282441566:2282441566(0) ack 646237210 win 65020
    and 4 more blocks with ( F 0:0(0) ack 1 win 65020)


    symptom #2:
    rule 11/(match) block out on fxp0: server1.443 > 85.100.93.186.2522:
    [|tcp] (DF)

    # Now I block 1 ~ 5 connections per minute with avg 2 conn pr min
    ( I'm not sure those have SYN flag set. I hope those have not and
    dont know how to investigate


    Especially the first one!
    I explaind the secound one , BLOCK OUT, with the time limits i have
    custmized on tcp options of the rule ( tcp.established 300)
    because my box has already killed the state,because of unactivity OS
    behind FW tries to sending data.
    -- because my server streams realtime quatos, 5 min inactivity is
    unusal. ---


    BUT, i have sone doubts about the first one related to INBOUND BLOCKS
    -- sometimes those hosts ( blocked) has already one or more states in
    my table ( but it is verry common , more than one client sharing the
    same LAN connect to our servers). But blocked source ports are
    different from the ones in state table, - ofcourse -
    -- some of the hosts could never establish a connection yet, I have
    checked that those are all acknowledged. Now, they are trying but
    has not complaint yet
    -- none of the limits have exceeded. (max, max-src-states,
    max-src-conn ..etc)


    What may cause these problems ?
    -- QUEUE limits ?
    -- TCP options ( S/SA) ?
    -- or what can else , which I have missed ?

    Thank you for your help and forgive my this long msg takes time.


    ilker.arabaci@gmail.com
    App. Developer



    #pftop -vqueue
    QUEUE BW SCH PRIO PKTS BYTES DROP_P
    DROP_B QLEN BORROW SUSPEN P/S B/S
    any_eurojava 14M cbq 2 2644516 558110K
    6 1019 0 45521 11386 1314 403213



    Some of My pf.conf

    Options
    -------------------
    set block-policy drop
    set state-policy floating

    #set optimization aggressive
    set optimization normal

    set limit states 100000

    set timeout { tcp.first 120, tcp.opening 30, tcp.established 3600}
    set timeout { tcp.closing 300, tcp.finwait 45, tcp.closed 60 }
    set timeout { udp.first 60, udp.single 30, udp.multiple 60 }
    set timeout { icmp.first 20, icmp.error 10 }
    set timeout { other.first 60, other.single 30, other.multiple 60 }
    set timeout { adaptive.start 10000, adaptive.end 30000 }

    set loginterface $ext_if

    set skip on lo0
    set skip on $int_if

    scrub in on $ext_if all no-df
    scrub on $ext_if all reassemble tcp

    ################# QUEUEING CLASSES ################
    altq on $ext_if cbq bandwidth 20Mb queue {
    internet,throttled,firewall,secure_out,trusted_in}

    # Main Internet services of servers -
    queue internet bandwidth 16Mb {any_eurojava,any_web,any_commons}
    # EuroJava Service for any customers
    queue any_eurojava bandwidth 14Mb priority 2 qlimit 500
    cbq(borrow ecn)
    queue any_web bandwidth 1Mb priority 4 cbq(borrow ecn)
    queue any_commons bandwidth 1Mb priority 4 cbq(default borrow
    ecn)
    queue firewall bandwidth 1Mb priority 1 cbq(borrow ecn) #
    firewall management queue
    # secure servers, behind FW , established out queue
    queue secure_out bandwidth 1Mb priority 5 cbq(borrow ecn)
    # trusted hosts' traffic on un-common services
    queue trusted_in bandwidth 1Mb priority 3 cbq(borrow ecn)
    # throttled access for flooding hosts on throttled queuee
    queue throttled bandwidth 1Mb priority 7 cbq(red)
    ################################################## ######
    ## RELATED RULES TO MY PROBLEM

    # Rule 10
    block in log on $ext_if all


    # Rule 11
    block out log on $ext_if all





    block in log quick on $ext_if from to any


    #

    # my main PASS rule for tcp in to port 443

    pass in on $ext_if proto tcp from any to port
    {$eurojava_ports} flags S/SA \

    tag CHECK_MORE keep state ( max 5000, max-src-states 100, \


    tcp.established 300, tcp.closing 10, tcp.finwait 10, tcp.closed
    10 \

    max-src-conn 100, overload ) \


    queue any_eurojava

    # CHECK rule for flooting customers
    pass in on $ext_if from tagged CHECK_MORE tag THROTTLED
    keep state \

    ( max 500, tcp.established 60, tcp.closing 10, tcp.closed 5,
    tcp.finwait 10 \

    max-src-conn-rate 50/5, overload flush global)
    \

    queue throttled


    ## OUT BOUND TRAFFIC OF SECURED SERVERS
    pass out on $ext_if proto tcp from to any flags S/SA modulate
    state queue secure_out


  2. Re: PF with 20Mbps data streaming

    with
    # tcpdump -n -e -o -vvv -ttt -i pflog0 port 443

    rule 10/(match) [uid 0, pid 1807] block in on fxp0: 85.100.124.74.14464
    > server1.443: [|tcp] (ttl 249, id 65259, len 40, bad cksum 0! differs by f890)


    rule 11/(match) [uid 0, pid 1807] block out on fxp0: server1.443 >
    85.105.113.156.2802: [|tcp] (DF) (ttl 64, id 20448, len 40)

    block in on fxp0: 81.215.12.114.2051 > server1.443: [|tcp] (ttl 250, id
    62897, len 40, bad cksum 0! differs by 7430


  3. Re: PF with 20Mbps data streaming

    thanks Rmkml

    I also started to debug with clasicall method , checking out the rule
    options which i have put with great enthusiasm to do the job as it must
    be
    done

    queue is must, bandwith is also. scrube didnot change anything ,
    modulate
    is not for inbound rules
    But i had to disable S/SA flags

    problem may be solved, i dont distinguish who is comming in any more

    as the nature of PF, S/SA is right thing to do. but in man pages , I
    had
    noticed it may break some connections for specific os plaoform and
    connection types.

    like sne hosts send Tcp RST for begginning of comminication.

    The problem is seemed to be solved now thank yo

    when i enable S/SA, i get blocks with "bad cksum 0! "





    ----- Original Message -----
    From: "rmkml"
    To: "ilker ARABACI"
    Sent: Monday, April 10, 2006 1:42 PM
    Subject: Re: 20 Mbps data streaming behind PF


    > Hi ilker,
    > sorry I don't help,
    > but I have question,
    > If you remove S/SA tcp flags check ?
    > If you remove bandwith ?
    > If you remove scrube* options ?
    > If you remove modulate state ?
    > maybe record traffic with tcpdump (on two intf) ?
    > Regards
    > Rmkml



  4. Re: PF with 20Mbps data streaming

    I do not filtering anything on $int_if
    But if i disable S/SA for internal -> out
    then i would get wrong state tables from in -> out and wrong queue
    will be assinged and my state table will be overwhelmed again. ( esp at
    times FW reset)

    now, I am sure I know the reson of outbound blocks, those are all
    inactive connections, and killed from state table due to time limit,
    not important.it must be, OS netstat may increase but it is also not
    imp. because of established timeout of OS.

    if i can find what TCP flags I should put in place of S/SA as tcp
    option to describe the comminucation beginning, i will be verry happy.


+ Reply to Thread