Barrage of repeated requests for large files on my server frombots... what to do? - Firewalls

This is a discussion on Barrage of repeated requests for large files on my server frombots... what to do? - Firewalls ; Hello all, first quickly I'm sorry if this is the wrong place, but it seemed the most appropriate usenet group. My server is currently receiving a barrage of requests straight for the largest files (videos) on my web server, which ...

+ Reply to Thread
Results 1 to 5 of 5

Thread: Barrage of repeated requests for large files on my server frombots... what to do?

  1. Barrage of repeated requests for large files on my server frombots... what to do?

    Hello all, first quickly I'm sorry if this is the wrong place, but it
    seemed the most appropriate usenet group.

    My server is currently receiving a barrage of requests straight for
    the largest files (videos) on my web server, which was causing the
    server to become dreadfully slow to respond (at least a minute before
    it hints a response) and sometimes not even at all.

    This is the 3rd day it's been going on.

    I've been keeping track of stats from the bots for the last 16
    hours...
    - Requests were from 845 different IP addresses
    - Nearly all the IP addresses are from China
    - 30,377 requests for video files (not the video pages)
    - 40 distinct video files were requested
    - One host made 13,000 requests within those 40 files.

    I'm running a dedicated server.

    I've had a few similar "attacks" (or algorithm malfunction?) in the
    past, but it was only by 3 or 4 different IP addresses and so it was
    easy enough to just block them using iptables.

    The only way I'm able to block the current "attack" and get these
    stats is that for some reason they all have the same referer:
    http://www.173la.cn/u/egao/ -- which I don't know about you but it
    doesn't even seem like a real website to me.

    If the requests didn't have this referer, I would not have a way to
    block these requests and my server would be beaten to a pulp. Right
    now I'm redirecting any request with that referer to my logger script.



    Any ideas of a better way to solve this once and for all? I'm
    passively blocking them, but they are still trying to jam the same
    requests down my server's throat. And I'm afraid that they'll get rid
    of the referer if I try any sort of active block rather than my
    passive method, and I'll be screwed.

    The IP addresses also seem to be from several different hosts (judging
    from the whois data), so it's not exactly easy to inform them all...


    An example of one of the bots from the access_log:
    211.139.255.10 - - [06/May/2008:22:45:31 -0400] "GET /videos/content/
    439_poppy2606.wmv HTTP/1.1" 403 317 "http://www.173la.cn/u/egao/"
    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; )"
    (Notice it's 403 because I'm blocking them myself)

    Any tips at all?

    Thanks in advance!!
    - Richard

  2. Re: Barrage of repeated requests for large files on my server from bots... what to do?

    richard k zad wrote in news:c9f6d507-a64d-463b-95a7-
    6ccb7d13f778@w1g2000prd.googlegroups.com:

    > An example of one of the bots from the access_log:
    > 211.139.255.10 - - [06/May/2008:22:45:31 -0400] "GET /videos/content/
    > 439_poppy2606.wmv HTTP/1.1" 403 317 "http://www.173la.cn/u/egao/"
    > "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; )"
    > (Notice it's 403 because I'm blocking them myself)
    >


    according to whois -h whois.apnic.net
    [quote]
    Please send abuse e-mail to
    remarks: chenxi@gd.chinamobile.com
    remarks: Please send probe e-mail to
    remarks: chenxi@gd.chinamobile.com
    [unquote]

    Of course you could bloock all china.

    05/07/08 04:24:16 Fast traceroute 211.139.255.10
    Trace 211.139.255.10 ...
    .....
    7 4.69.132.77 61ms 55ms 56ms TTL: 0 (ae-
    3.ebr2.LosAngeles1.Level3.net ok)
    8 4.69.137.22 55ms 53ms 52ms TTL: 0 (ae-72-
    72.csw2.LosAngeles1.Level3.net ok)
    9 4.68.20.69 51ms 57ms 52ms TTL: 0 (ae-23-
    79.car3.LosAngeles1.Level3.net ok)
    10 62.156.138.69 54ms 89ms 55ms TTL: 0 (No rDNS)
    11 62.154.14.105 310ms 354ms 207ms TTL: 0 (lax-
    sa1.LAX.US.net.DTAG.DE ok)
    12 62.154.5.205 210ms 208ms 208ms TTL: 0 (hkg-
    sa1.HKG.HK.net.DTAG.DE ok)
    13 217.239.41.10 208ms 209ms 348ms TTL: 0 (No rDNS)
    14 217.6.24.138 280ms 247ms 265ms TTL: 0 (No rDNS)
    15 218.200.253.237 262ms 250ms 250ms TTL: 0 (No rDNS)
    16 211.136.3.57 243ms 236ms 235ms TTL: 0 (No rDNS)
    17 211.136.6.34 270ms 259ms 262ms TTL: 0 (No rDNS)
    18 211.139.135.70 269ms 256ms 260ms TTL: 0 (No rDNS)
    19 211.139.254.3 278ms 254ms 267ms TTL: 0 (No rDNS)
    20 211.139.255.10 271ms 264ms * TTL:244 (No rDNS)

    The lack of rDNS shows they are rather clueless.
    You might complain to HKG.HK.net.DTAG.DE about the bots
    It looks like HKG may be Hongcong and



    --
    bz

    please pardon my infinite ignorance, the set-of-things-I-do-not-know is an
    infinite set.

    bz+csm@ch100-5.chem.lsu.edu remove ch100-5 to avoid spam trap

  3. Re: Barrage of repeated requests for large files on my server from bots... what to do?

    In article @w1g2000prd.googlegroups.com>, richkzad@gmail.com says...
    > Any ideas of a better way to solve this once and for all? I'm
    > passively blocking them, but they are still trying to jam the same
    > requests down my server's throat. And I'm afraid that they'll get rid
    > of the referer if I try any sort of active block rather than my
    > passive method, and I'll be screwed.


    Enter a subnet block for their providers IP range in your firewall so
    that no address in that range can reach your server.

    You do have a proper firewall, right?

    The following list works for me, and I'm inside the USA and don't deal
    with customers/users outside the USA, the entire 211.0.0.0/8 network is
    blocked in my list below:

    Updated 2008MAR01

    12.144.182.0/24
    12.45.203.0/24
    12.98.139.0/24
    121.0.0.0/8
    124.0.0.0/8
    125.172.237.0/24
    125.213.42.0/24
    134.159.0.0/16
    134.160.0.0/16
    140.109.0.0/16
    140.110.0.0/15
    140.112.0.0/12
    140.128.0.0/13
    140.136.0.0/15
    140.138.0.0/16
    155.48.106.0/24
    162.40.0.0/16
    168.126.0.0/16
    172.184.111.203
    190.3.209.0/24
    193.248.60.0/24
    193.251.0.0/16
    193.252.0.0/16
    193.253.0.0/16
    194.170.0.0/16
    195.174.0.0/16
    195.175.16.0/20
    195.229.0.0/23
    195.58.124.0/24
    200.181.0.0/16
    200.244.0.0/16
    200.30.203.0/24
    201.0.0.0/8
    201.130.192.0/18
    201.230.0.0/16
    201.240.0.0/16
    202.40.148.0-202.40.149.255
    202.84.128.0-202.84.255.255
    202.88.186.0/24
    203.150.101.0/24
    203.152.22.0/24
    203.162.0.0-203.162.255.255
    203.210.128.0-203.210.255.255
    205.251.79.0/24
    210.0.0.0/8
    211.0.0.0/8
    212.150.124.0/24
    212.162.8.0/24
    212.18.57.0/24
    212.202.178.0/24
    212.27.32.0-212.27.63.255
    212.64.0.0/16
    212.9.7.0/24
    213.13.26.0/24
    213.192.0.0-213.192.255.255
    216.184.97.0/24
    216.76.35.0/24
    217.118.224.0-217.118.239.255
    217.160.110.0/24
    218.164.28.0/24
    218.234.0.0-218.239.255.255
    218.252.74.0/24
    218.67.128.0-218.76.255.255
    219.115.214.0/24
    219.212.4.0/24
    219.56.0.0/24
    219.97.93.0/24
    220.0.0.0/8
    222.0.0.0/8
    41.221.19.0/24
    60.0.0.0/8
    61.135.148.0/24
    61.175.239.0/24
    61.181.0.0/16
    61.218.19.0/24
    61.33.206.0/24
    61.48.18.0/24
    62.154.0.0/17
    62.240.161.0-62.240.161.127
    64.230.125.0/24
    66.250.125.0/24
    66.250.32.0/24
    66.28.35.131
    66.57.133.0/24
    71.184.44.154
    78.48.8.16
    80.0.0.0/8
    81.0.0.0/8
    82.0.0.0/8
    83.0.0.0/8
    85.17.255.0-85.255.255.255
    87.0.0.0/8
    88.0.0.0/8
    89.0.0.0/8
    91.76.56.0/24


    --
    - Igitur qui desiderat pacem, praeparet bellum.
    - Calling an illegal alien an "undocumented worker" is like calling a
    drug dealer an "unlicensed pharmacist"
    spam999free@rrohio.com (remove 999 for proper email address)

  4. Re: Barrage of repeated requests for large files on my server from bots... what to do?

    On Wed, 7 May 2008, in the Usenet newsgroup comp.security.firewalls, in article
    , bz wrote:

    >according to whois -h whois.apnic.net


    That's good, but most find that the abuse contacts are ignored.

    >Of course you could bloock all china.


    [compton ~]$ zgrep -c CN APNIC.gz
    1430
    [compton ~]$

    Don't forget Hong Kong and Macau (.hk and .mo, which adds 648 more
    networks) which are now special districts of China, and some people
    consider Taiwan (.tw with 397 more networks) as part of China for a
    total of 2475 address blocks, ranging from 200 /24s (255.255.255.0)
    on up to 4 /10s (255.192.0.0) - but hey, that's only 174,089,728 IPv4
    addresses in total.

    [compton ~]$ zgrep -E (CN|HK|MO|TW) APNIC.gz | cut -d' ' -f2 | cut -d'.'
    -f1 | sort -n | uniq -c | column
    51 58 85 124 2 161 3 204
    42 59 48 125 1 162 2 206
    50 60 3 134 4 163 161 210
    115 61 1 137 1 165 44 211
    63 116 2 139 2 166 96 218
    41 117 21 140 1 167 58 219
    44 118 1 143 4 168 29 220
    86 119 1 144 2 169 68 221
    29 120 1 147 103 192 70 222
    50 121 4 152 4 198
    44 122 2 158 708 202
    58 123 1 159 269 203
    [compton ~]$

    Translation: 51 address blocks in the 58.x.x.x range, 42 in the 59.x.x.x
    range, 50 in the 60.x.x.x range, and so on. And before you use a /8
    rule, you should know that there are other countries using those blocks,

    [compton ~]$ zgrep -h ' 58\.' [ALR]* | cut -d' ' -f1 | sort | uniq -c |
    column
    1 AF 3 ID 2 NZ 4 TW
    24 AU 3 IN 2 PH 1 VN
    5 BD 30 JP 5 PK
    43 CN 15 KR 6 SG
    4 HK 4 MY 8 TH
    [compton ~]$ ^58^59
    zgrep -h ' 59\.' [ALR]* | cut -d' ' -f1 | sort | uniq -c | column
    8 AU 4 HK 6 KR 4 TW
    1 BD 7 IN 1 PK
    34 CN 16 JP 1 SG
    [compton ~]$

    That's just two /8s. Do you know your ISO-3166 country codes?

    >The lack of rDNS shows they are rather clueless.


    and also makes it hard to use rDNS to identify Chinese blocks - but what
    else is new? This happens in a lot of countries.

    >05/07/08 04:24:16 Fast traceroute 211.139.255.10


    And what IF ANYTHING does this have to do with the spam?

    >11 62.154.14.105 310ms 354ms 207ms TTL: 0 (lax-
    >sa1.LAX.US.net.DTAG.DE ok)
    >12 62.154.5.205 210ms 208ms 208ms TTL: 0 (hkg-
    >sa1.HKG.HK.net.DTAG.DE ok)


    >You might complain to HKG.HK.net.DTAG.DE about the bots


    Why - they happen to be an intermediate step on YOUR route to China,
    but this is not as likely for the rest of us.

    >It looks like HKG may be Hongcong and


    HKG is probably Hong Kong - a former British enclave on the coast of
    China around 22.2 degrees North, as LAX is Los Angeles. Your point?

    Old guy

  5. Re: Barrage of repeated requests for large files on my server frombots... what to do?

    On May 6, 10:48 pm, richard k zad wrote:
    > Hello all, first quickly I'm sorry if this is the wrong place, but it
    > seemed the most appropriate usenet group.
    >
    > My server is currently receiving a barrage of requests straight for
    > the largest files (videos) on my web server, which was causing the
    > server to become dreadfully slow to respond (at least a minute before
    > it hints a response) and sometimes not even at all.
    >
    > This is the 3rd day it's been going on.
    >
    > I've been keeping track of stats from the bots for the last 16
    > hours...
    > - Requests were from 845 different IP addresses
    > - Nearly all the IP addresses are from China
    > - 30,377 requests for video files (not the video pages)
    > - 40 distinct video files were requested
    > - One host made 13,000 requests within those 40 files.
    >


    Since the same IP requests the file a bunch of times, maybe
    you could put a CGI interface infront of the file instead of
    serving the file directly then log the IPs in a database with
    a time stamp (or use the webserver logs or whatever) to
    see if multiple requests have been made in a short period of
    time. If the same IP is requesting it over and over again

    while true
    do
    sleep 1
    transmit byte
    done

    Transmit the file really slowly so that the connection
    stays active, but does not use up your bandwidth.

    You could also check the referer in the CGI and block
    anything which does not have a referer page at least
    from the same domain.

    There is probably a way to do a better job with a
    rate limiting firewall rule. Maybe spamd on OpenBSD
    perhaps?

    iptables has a rate limiting option

    limit
    This module matches at a limited rate using a token bucket
    filter. A
    rule using this extension will match until this limit is
    reached
    (unless the ‘!’ flag is used). It can be used in combination
    with the
    LOG target to give limited logging, for example.

    --limit rate
    Maximum average matching rate: specified as a number,
    with an
    optional ‘/second’, ‘/minute’, ‘/hour’, or ‘/day’
    suffix; the
    default is 3/hour.

    --limit-burst number
    Maximum initial number of packets to match: this
    number gets
    recharged by one every time the limit specified above
    is not
    reached, up to this number; the default is 5.

    Maybe you could create a rule limiting the average rate and the burst
    rate for these subnets. That way they can still connect but only
    use a fraction of your bandwidth.

    > I'm running a dedicated server.
    >
    > I've had a few similar "attacks" (or algorithm malfunction?) in the
    > past, but it was only by 3 or 4 different IP addresses and so it was
    > easy enough to just block them using iptables.
    >
    > The only way I'm able to block the current "attack" and get these
    > stats is that for some reason they all have the same referer:http://www.173la.cn/u/egao/-- which I don't know about you but it
    > doesn't even seem like a real website to me.
    >
    > If the requests didn't have this referer, I would not have a way to
    > block these requests and my server would be beaten to a pulp. Right
    > now I'm redirecting any request with that referer to my logger script.
    >
    > Any ideas of a better way to solve this once and for all? I'm
    > passively blocking them, but they are still trying to jam the same
    > requests down my server's throat. And I'm afraid that they'll get rid
    > of the referer if I try any sort of active block rather than my
    > passive method, and I'll be screwed.
    >
    > The IP addresses also seem to be from several different hosts (judging
    > from the whois data), so it's not exactly easy to inform them all...
    >
    > An example of one of the bots from the access_log:
    > 211.139.255.10 - - [06/May/2008:22:45:31 -0400] "GET /videos/content/
    > 439_poppy2606.wmv HTTP/1.1" 403 317 "http://www.173la.cn/u/egao/"
    > "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; )"
    > (Notice it's 403 because I'm blocking them myself)
    >
    > Any tips at all?
    >
    > Thanks in advance!!
    > - Richard


    ----
    http://www.1150riverviewdr.com/

+ Reply to Thread