Rate limiting with "tc"? - Networking

This is a discussion on Rate limiting with "tc"? - Networking ; Hi, all. My company has a reasonably large pipe to the Internet; however, certain large transfers (e.g., our hourly database replication with our offsite server) spike the usage. I'd like to limit our traffic, both inbound and outbound, such that ...

+ Reply to Thread
Results 1 to 10 of 10

Thread: Rate limiting with "tc"?

  1. Rate limiting with "tc"?

    Hi, all. My company has a reasonably large pipe to the Internet;
    however, certain large transfers (e.g., our hourly database
    replication with our offsite server) spike the usage. I'd like to
    limit our traffic, both inbound and outbound, such that no single user
    can use more than 50% of our bandwidth. I've poked around, and it
    seems that the best way to accomplish this is by way of the "tc"
    command... unfortunately, its documentation is, ummmm... "in depth."
    While I'm willing to wade through it, and play with trial-and-error,
    this is one of those times where I just want that one magic command,
    and then I'll move on. So any help would be greatly appreciated.

    Thanks much!

    -Ken

  2. Re: Rate limiting with "tc"?

    Ravenpi wrote:
    > Hi, all. My company has a reasonably large pipe to the Internet;
    > however, certain large transfers (e.g., our hourly database
    > replication with our offsite server) spike the usage. I'd like to
    > limit our traffic, both inbound and outbound, such that no single
    > user can use more than 50% of our bandwidth.


    No single _user_ as in the sum of her connections, or do you mean no
    single connection/flow?

    > I've poked around, and it seems that the best way to accomplish this
    > is by way of the "tc" command... unfortunately, its documentation
    > is, ummmm... "in depth." While I'm willing to wade through it, and
    > play with trial-and-error, this is one of those times where I just
    > want that one magic command, and then I'll move on. So any help
    > would be greatly appreciated.


    What is used to implement this hourly database replication? Is it a
    transfer via TCP?

    Ostensibly the congestion control and avoidance stuff is supposed to
    do the "reasonably right thing" wrt sharing the bandwidth. Are you
    simply concerned that the utilization gets high, or does it actually
    meaninfully affect other uses of the link?

    One limit to the performance of a TCP connection is:

    Throughput < Window/RTT

    The Window in Linux will be bound by the socket buffer size, which can
    be controlled via some sysctls for defaults (look at sysctl -a | grep
    wmem; sysctl -a | grep rmem; etc) or applications can make explicit
    setsockopt() calls.

    Assuming you have a reasonable estimate for the RTT to the replication
    site you could use that knowledge to cap the window sizes and so keep
    TCP from being able to use more bandwidth than desired without having
    to use tc.

    Not suggesting you _don't_ use tc, just offering another possiblity.

    rick jones
    --
    portable adj, code that compiles under more than one compiler
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  3. Re: Rate limiting with "tc"?

    Ravenpi wrote:

    > Hi, all. My company has a reasonably large pipe to the Internet;
    > however, certain large transfers (e.g., our hourly database
    > replication with our offsite server) spike the usage. I'd like to
    > limit our traffic, both inbound and outbound, such that no single user
    > can use more than 50% of our bandwidth. I've poked around, and it
    > seems that the best way to accomplish this is by way of the "tc"
    > command... unfortunately, its documentation is, ummmm... "in depth."
    > While I'm willing to wade through it, and play with trial-and-error,
    > this is one of those times where I just want that one magic command,
    > and then I'll move on. So any help would be greatly appreciated.


    First:
    Outbound policing is possible, but inbound is not.

    Now onto tc:
    tc works together with something to classify packets. There's a number of
    tools to do so, some with more features, others simpler to maintain. I for
    myself use iptables for this task. To do so, you need the MARK target
    available for iptables, because it's a mark applied to each packet used to
    classify and thus assign a packet to a specific rate-limited queue.
    Also I'm using the HTB (Hierarchical Token Bucket) to classify packets.
    That's based on historic reasons: I've did it with htb years ago and never
    had any reason or will to change it.

    Now onto the usage:
    1. You need a qdisc for your nic.
    tc qdisc add dev root handle :0 htb default
    NIC is the network device to work on, x is the master number of your chains,
    n is the default chain if packets can not be assigned by other means.

    2. Now you need a master chain:
    tc class add dev parent :0 classid :1 htb \
    rate kbit ceil kbit
    NIC and x are always the same as used for 'tc qdisc' above, NORM and MAX are
    the normal and maximum rates (in kbit/s) available for this nic on the
    outbound direction. both might be the same, but NORM should not be greater
    than MAX. This means that normally not more than NORM kbit/s should be
    sent, but if traffic exceeds this rate, there's more bandwith available.

    3.a You need a default chain:
    tc class add dev parent :1 classid : htb \
    rate <(NORM-T)>kbit ceil kbit prio 2
    (NORM-T) is the normal traffic bandwith without traffic handled by specific
    chains. By specifying 'ceil kbit' the hard limit is anyway the maximum
    bandwith you allow. Thus, if default traffic exceeds the normal rate and no
    other traffic is using bandwith, default traffic can use up to MAX kbit/s.
    There's no need to explicitly put all packets into this chain, they are put
    into this chain by default due to 'default ' from step 1.

    3.b Now you need chains with limited bandwith and priorities:
    tc class add dev parent :1 classid : htb \
    rate 10kbit ceil kbit prio 0
    This is a chain for very low traffic but with maximum priority. I use it for
    ACK packets speeding up my downloads even if there's lot of other upstream
    traffic. But it's also limited in bandwith. y is a unique number to name
    the chain.

    3.c To use it you need an iptables rule:
    iptables -t mangle -A OUTPUT -p tcp \
    -m length --length :64 -j MARK --set-mark
    This needs the length match for iptables. You can set the mark (z) to
    whatever you like; I prefer the number of the chain to make it easier to
    see which iptables rule(s) belong to which chain.

    3.d Now you need to tell the system to put packets with a specific mark to a
    chain:
    tc filter add dev parent :0 prio 0 protocol ip \
    handle fw flowid :

    That's it. Now packets up to the size of 64byte are prioritized first but
    may normally not use more than 10kbit/s of bandwith.
    Repeat steps 3.b-d for all types of traffic you may need to prioritize.

    HTH,
    Felix

  4. Re: Rate limiting with "tc"?

    WOW. Thanks, guys. You've given me lots to chew on. As for not
    being able to throttle in-bound connections, I had that realization
    while taking a walk today. But then I asked myself: couldn't you just
    slow down ACKs to the remote host? Or does the fact that TCP can re-
    order out-of-sequence packets make that moot?

    Regardless, again, thanks!

    -Ken

  5. Re: Rate limiting with "tc"?

    Ravenpi wrote:

    > WOW. Thanks, guys. You've given me lots to chew on. As for not
    > being able to throttle in-bound connections, I had that realization
    > while taking a walk today. But then I asked myself: couldn't you just
    > slow down ACKs to the remote host? Or does the fact that TCP can re-
    > order out-of-sequence packets make that moot?


    The fact that prioritizing ACKs on their way to the remote host speeds up
    downloads lets me guess that it is also possible to do it the other way -
    de-prioritizing ACKs should slow down a TCP-connection.
    Then the main reason for me to use bandwith management was in fact that I
    wanted to speed up my downloads while still sending eMail at maximum speed.

    Greetings,
    Felix

  6. Re: Rate limiting with "tc"?

    Ravenpi wrote:
    > WOW. Thanks, guys. You've given me lots to chew on. As for not
    > being able to throttle in-bound connections, I had that realization
    > while taking a walk today. But then I asked myself: couldn't you
    > just slow down ACKs to the remote host? Or does the fact that TCP
    > can re- order out-of-sequence packets make that moot?


    The TCP window size hack won't care about direction so long as you set
    both wmem and rmem values and/or have applications make both SO_SNDBUF
    and SO_RCVBUF settings.

    rick jones
    --
    portable adj, code that compiles under more than one compiler
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  7. Re: Rate limiting with "tc"?

    "Ravenpi" wrote in message
    news:13e6eb39-7467-40af-8b01-884e9e1eb838@m45g2000hsb.googlegroups.com...
    > WOW. Thanks, guys. You've given me lots to chew on. As for not
    > being able to throttle in-bound connections, I had that realization
    > while taking a walk today. But then I asked myself: couldn't you just
    > slow down ACKs to the remote host? Or does the fact that TCP can re-
    > order out-of-sequence packets make that moot?
    >
    > Regardless, again, thanks!


    Don't forget in the tc/iptables interface that iptables can also be
    configured to classify based on other things such as process group or user
    ID. If your backup process is run as its own user ID, then you have a
    unique way of identifying it.



  8. Re: Rate limiting with "tc"?

    Pardon me for breaking in late,

    On Thu, 25 Sep 2008 14:59:04 -0700, "D. Stussy" wrote:

    >"Ravenpi" wrote in message
    >news:13e6eb39-7467-40af-8b01-884e9e1eb838@m45g2000hsb.googlegroups.com...
    >> WOW. Thanks, guys. You've given me lots to chew on. As for not
    >> being able to throttle in-bound connections,


    Both rsync and wget offer working rate-limiting in userspace, why is it
    then that people are still saying inbound throttling cannot be done when
    there are these working examples of it?

    Thanks,
    Grant.
    --
    http://bugsplatter.id.au/

  9. Re: Rate limiting with "tc"?

    Grant wrote:
    > Both rsync and wget offer working rate-limiting in userspace, why is
    > it then that people are still saying inbound throttling cannot be
    > done when there are these working examples of it?


    I cannot speak to can/cannot do inbound throttling in tc, but will
    point-out that when a receiver does rate limiting above the socket
    layer he may induce some behavoiurs in TCP (eg zero window probing)
    which are not _normally_ seen in a transfer, nor in a situation when
    the rate limiting is implemented _below_ TCP.

    It may not matter for rsync and wget in and of themselves, but it can
    be an issue if it is used in say load generators for networked server
    benchmarking.

    rick jones
    --
    No need to believe in either side, or any side. There is no cause.
    There's only yourself. The belief is in your own precision. - Jobert
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  10. Re: Rate limiting with "tc"?

    Grant wrote:
    > Pardon me for breaking in late,
    >
    > On Thu, 25 Sep 2008 14:59:04 -0700, "D. Stussy" wrote:
    >
    >> "Ravenpi" wrote in message
    >> news:13e6eb39-7467-40af-8b01-884e9e1eb838@m45g2000hsb.googlegroups.com...
    >>> WOW. Thanks, guys. You've given me lots to chew on. As for not
    >>> being able to throttle in-bound connections,

    >
    > Both rsync and wget offer working rate-limiting in userspace, why is it
    > then that people are still saying inbound throttling cannot be done when
    > there are these working examples of it?
    >
    > Thanks,
    > Grant.


    I think it would be better to say you can't do it perfectly.
    You can still do a lot better than doing nothing for inbound.
    As the OP has plenty of bandwidth then shaping or policing each user to
    50% should work well.

    Andy.

+ Reply to Thread