tc and bandwidth limiting: What am I doing wrong? - Networking

This is a discussion on tc and bandwidth limiting: What am I doing wrong? - Networking ; So I thought I'd limit the bandwidth on one of my network interfaces to run a few tests on a few things. I tried: tc qdisc add dev eth1 root tbf burst 20480 limit 20480 \ mtu 1514 rate 32000bps ...

+ Reply to Thread
Results 1 to 8 of 8

Thread: tc and bandwidth limiting: What am I doing wrong?

  1. tc and bandwidth limiting: What am I doing wrong?

    So I thought I'd limit the bandwidth on one of my network interfaces to
    run a few tests on a few things. I tried:

    tc qdisc add dev eth1 root tbf burst 20480 limit 20480 \
    mtu 1514 rate 32000bps

    ....as was mentioned in the Traffic Control HOWTO. However, sustained
    transfer rates weren't 64K/sec, but were 15K/sec. Doubling and
    quadrupling the numbers for rate, burst, and limit made no difference to
    the sustained transfer rates I observed.

    Kernel 2.6.22.1 (vanilla), iproute2 2.6.22.20070710 . Is the HOWTO
    outdated? Am I doing something stupid? Is there an easier way? The
    machines are using forcedeth and e1000 cards, and sustained transfer
    rates without tc are ~10M/sec. If you need more information, say what
    you need and I'll find it. Thanks in advance.

    (I ended up limiting bandwidth with an old 10bT DEC Tulip card I had
    lying around. Yay for obsolete hardware, I guess.)

    --
    "Depress the button," it said. Gloria looked at the button. "You're
    a very ugly button," she began. --Phil Janes, _The Galaxy Game_
    My blog and resume: http://crow202.dyndns.org:8080/wordpress/
    Matt G|There is no Darkness in Eternity/But only Light too dim for us to see

  2. Re: tc and bandwidth limiting: What am I doing wrong?

    On Jun 14, 10:25*am, Dances With Crows
    wrote:

    > tc qdisc add dev eth1 root tbf burst 20480 limit 20480 \
    > mtu 1514 rate 32000bps


    > ...as was mentioned in the Traffic Control HOWTO. *However, sustained
    > transfer rates weren't 64K/sec, but were 15K/sec. *Doubling and
    > quadrupling the numbers for rate, burst, and limit made no difference to
    > the sustained transfer rates I observed.


    It's hard to figure out what exactly happened. You claim you set a
    traffic limit of 32,000 bits per second, and expected a transfer rate
    of 64,000 somethings per second (bits? bytes?) but got 15,000. Why
    would you expect 64,000 if you asked for 32,000? No conceivable bits/
    bytes conversion can make that work.

    What was the command you used? What was the rate you got (and specify
    if it's in bits or bytes) and what did you expect? And how did you
    measure it?

    The problem may be that you are confusing bits with bytes. It also may
    be that you are expecting *application* cooked data throughput to
    equal raw network throughput.

    It's hard to tell though because your command seems not to match any
    expectation, with or without such errors.

    DS

  3. Re: tc and bandwidth limiting: What am I doing wrong?

    David Schwartz staggered into the Black Sun and said:
    > Dances With Crows wrote:
    >> tc qdisc add dev eth1 root tbf burst 20480 limit 20480 \
    >> mtu 1514 rate 32000bps


    >> ...as was mentioned in the Traffic Control HOWTO. However, sustained
    >> transfer rates weren't 64K/sec, but were 15K/sec.

    > It's hard to figure out what exactly happened. You claim you set a
    > traffic limit of 32,000 bits per second, and expected a transfer rate
    > of 64,000 somethings per second (bits? bytes?) but got 15,000.


    That's 64K/sec (kilobytes) expected, got 15K/sec. (Actually I should
    have expected 32K/sec, but 32 still != 15.)

    > What was the command you used?


    See first paragraph. That was the exact syntax given in the HOWTO for
    limiting an interface to 256 kbit/second. All right, they might have
    gotten their math wrong, but I usually expect HOWTOs to be reasonably
    accurate. Please also note that removing the rule, then replacing it
    with the numbers doubled, quadrupled or *ed by 8 for burst, limit, and
    rate produced the same measured 15K/sec. I said that in the first
    message, but it's worth repeating.

    > And how did you measure it?


    Sustained transfer rate using scp.

    > It also may be that you are expecting *application* cooked data
    > throughput to equal raw network throughput.


    The overhead isn't *that* high with scp, and see above.

    > It's hard to tell though because your command seems not to match any
    > expectation, with or without such errors.


    That's sort of why I posted here, since what I saw is so different from
    what I expected. So: Who's using tc, how are you using it, and how
    do you make it work the way you want it to?

    --
    That which does not kill us makes us stranger.
    --Trevor Goodchild, Aeon Flux
    My blog and resume: http://crow202.dyndns.org:8080/wordpress/
    Matt G|There is no Darkness in Eternity/But only Light too dim for us to see

  4. Re: tc and bandwidth limiting: What am I doing wrong?

    David Schwartz wrote:
    > On Jun 14, 10:25 am, Dances With Crows
    > wrote:
    >
    >> tc qdisc add dev eth1 root tbf burst 20480 limit 20480 \
    >> mtu 1514 rate 32000bps

    >
    > It's hard to figure out what exactly happened. You claim you set a
    > traffic limit of 32,000 bits per second, and expected a transfer rate
    > of 64,000 somethings per second (bits? bytes?)


    Just to make things even more confusing TC reads bps as bytes/sec you
    have to use bit/kbit/mbit for bits/sec.

    When using tc on gig nics you really need to turn off tcp segmentation
    offload with IIRC ethtool -k. With it on you can get strange results.

    When you put qdiscs on the root of eths you need to consider arp - these
    will be delayed/dropped as well - which can mess things up.

    Andy.

  5. Re: tc and bandwidth limiting: What am I doing wrong?


    One last clarification -- changing the configured 'tc' *rate* had no
    effect on the transfer speed?!

    DS

  6. Re: tc and bandwidth limiting: What am I doing wrong?

    David Schwartz staggered into the Black Sun and said:
    > One last clarification -- changing the configured 'tc' *rate* had no
    > effect on the transfer speed?!


    The bits in my notes here say that:

    tc qdisc add dev eth1 root tbf burst 131122 limit 131122 mtu 1514 \
    rate 2012144

    gave the same results as--

    tc qdisc add dev eth1 root tbf burst 262244 limit 262244 mtu 1514 \
    rate 4024288

    which gave the same results as--

    tc qdisc add dev eth1 root tbf burst 524488 limit 524488 mtu 1514 \
    rate 8048576

    ....which is why I said that I was using 2.6.22.1 (vanilla) and iproute2
    2.6.22.20070710 , just in case there was some sort of stupid bug in
    either/both that was fixed in later revisions.

    --
    I realised I was dyslexic after attending a toga party dressed
    as a goat. --Author unknown
    My blog and resume: http://crow202.dyndns.org:8080/wordpress/
    Matt G|There is no Darkness in Eternity/But only Light too dim for us to see

  7. Re: tc and bandwidth limiting: What am I doing wrong?

    Andy Furniss staggered into the Black Sun and said:
    > David Schwartz wrote:
    >> On Jun 14, 10:25 am, Dances With Crows
    >> wrote:
    >>> tc qdisc add dev eth1 root tbf burst 20480 limit 20480 \
    >>> mtu 1514 rate 32000bps

    >> It's hard to figure out what exactly happened. You claim you set a
    >> traffic limit of 32,000 bits per second, and expected a transfer rate
    >> of 64,000 somethings per second (bits? bytes?)

    > Just to make things even more confusing TC reads bps as bytes/sec, you
    > have to use bit/kbit/mbit for bits/sec.
    >
    > When using tc on gig nics you really need to turn off tcp segmentation
    > offload with IIRC ethtool -k. With it on you can get strange results.


    eth1 in this case was a forcedeth CK804 10/100 card. The other
    machine's eth0 is an e1000, but since the tc'ing is being done on the
    machine with the forcedeth, I didn't think that would make a difference.
    Maybe it does. I can check later.

    > When you put qdiscs on the root of eths you need to consider arp -
    > these will be delayed/dropped as well - which can mess things up.


    This could have screwed things up as well. However, I tried this as
    well:

    tc qdisc add dev eth1 handle 1:0 root dsmark indices 1 default_index 0
    tc qdisc add dev eth1 handle 2:0 parent 1:0 tbf burst 20480 limit 20480
    mtu 1514 rate 32000bps

    ....and got exactly the same results. Yeah, I'm confused, which is why I
    posted here, hoping to find someone who knows more about tc than I do.

    --
    Jesus is the best radio producer in the beans. We need some saliva
    and pickles to get mad. --MegaHAL, "The Best of MegaHAL"
    My blog and resume: http://crow202.dyndns.org:8080/wordpress/
    Matt G|There is no Darkness in Eternity/But only Light too dim for us to see

  8. Re: tc and bandwidth limiting: What am I doing wrong?

    Dances With Crows wrote:

    >> When using tc on gig nics you really need to turn off tcp segmentation
    >> offload with IIRC ethtool -k. With it on you can get strange results.

    >
    > eth1 in this case was a forcedeth CK804 10/100 card. The other
    > machine's eth0 is an e1000, but since the tc'ing is being done on the
    > machine with the forcedeth, I didn't think that would make a difference.
    > Maybe it does. I can check later.
    >
    >> When you put qdiscs on the root of eths you need to consider arp -
    >> these will be delayed/dropped as well - which can mess things up.

    >
    > This could have screwed things up as well. However, I tried this as
    > well:
    >
    > tc qdisc add dev eth1 handle 1:0 root dsmark indices 1 default_index 0
    > tc qdisc add dev eth1 handle 2:0 parent 1:0 tbf burst 20480 limit 20480
    > mtu 1514 rate 32000bps
    >
    > ...and got exactly the same results. Yeah, I'm confused, which is why I
    > posted here, hoping to find someone who knows more about tc than I do.
    >


    Strange - I have a forcedeth 100mbit nic and just tried your tbf rule
    from the first post and it works OK.

    Could be unlucky kernel version I suppose, or maybe timers, if you have
    multicore/smp you shouldn't use tsc. I haven't tested noHZ on the box I
    shape on I still use Hz=1000.

    Does htb work eg -

    tc qdisc del dev eth1 root

    Forgetting to do this between tests can sometimes cause weirdness

    tc qdisc add dev eth1 root handle 1:0 htb
    tc class add dev eth1 parent 1:0 classid 1:1 htb rate 256kbit
    tc qdisc add dev eth1 parent 1:1 bfifo limit 20k
    tc filter add dev eth1 protocol ip parent 1:0 u32 match u32 0 0 classid 1:1

    It's not quite the same - ie it doesn't catch arp and I didn't use
    burst/cburst because it doesn't seem to behave the same as burst on tbf.

+ Reply to Thread