Yodi Aditya

Block Incapsula Monitoring System flooding your web server by IP and NGINX

I have subscribe Incapsula services about one years and i’m feel great and bad with their services. First thing, I like to see that their system protecting my websites very good. But, “overprotecting” is the only way i complain to them since a years ago and it’s doesn’t fixed till now.

What is it? It’s about “Incapsula Monitoring Bot / Incapsula Uptime Monitor” that flooding my server which make 2-3 request in every SECOND! For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
198.143.34.2 – – [08/Oct/2012:14:37:59 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
198.143.34.2 – – [08/Oct/2012:14:37:59 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
198.143.32.193 – – [08/Oct/2012:14:37:59 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
198.143.32.193 – – [08/Oct/2012:14:37:59 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
198.143.33.129 – – [08/Oct/2012:14:37:59 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
198.143.33.129 – – [08/Oct/2012:14:37:59 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
149.126.76.129 – – [08/Oct/2012:14:37:59 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
149.126.76.129 – – [08/Oct/2012:14:37:59 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
149.126.76.129 – – [08/Oct/2012:14:37:59 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
149.126.76.129 – – [08/Oct/2012:14:37:59 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
198.143.32.129 – – [08/Oct/2012:14:38:01 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
198.143.32.129 – – [08/Oct/2012:14:38:01 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
149.126.76.2 – – [08/Oct/2012:14:38:01 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
149.126.76.2 – – [08/Oct/2012:14:38:01 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
149.126.75.2 – – [08/Oct/2012:14:38:01 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
149.126.75.2 – – [08/Oct/2012:14:38:01 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
149.126.75.2 – – [08/Oct/2012:14:38:01 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
149.126.75.2 – – [08/Oct/2012:14:38:01 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
149.126.75.2 – – [08/Oct/2012:14:38:01 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
149.126.75.2 – – [08/Oct/2012:14:38:01 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
149.126.75.2 – – [08/Oct/2012:14:38:01 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
149.126.75.2 – – [08/Oct/2012:14:38:01 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
199.83.132.2 – – [08/Oct/2012:14:38:02 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
199.83.132.2 – – [08/Oct/2012:14:38:02 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
199.83.132.2 – – [08/Oct/2012:14:38:02 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
199.83.132.2 – – [08/Oct/2012:14:38:02 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
199.83.132.2 – – [08/Oct/2012:14:38:02 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
199.83.132.2 – – [08/Oct/2012:14:38:02 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
198.143.33.161 – – [08/Oct/2012:14:38:03 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
198.143.33.161 – – [08/Oct/2012:14:38:03 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
198.143.33.161 – – [08/Oct/2012:14:38:03 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
198.143.33.161 – – [08/Oct/2012:14:38:03 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
198.143.33.193 – – [08/Oct/2012:14:38:03 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
198.143.33.193 – – [08/Oct/2012:14:38:03 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
149.126.76.193 – – [08/Oct/2012:14:38:05 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
149.126.76.193 – – [08/Oct/2012:14:38:05 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
149.126.76.193 – – [08/Oct/2012:14:38:05 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
149.126.76.193 – – [08/Oct/2012:14:38:05 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
149.126.76.193 – – [08/Oct/2012:14:38:05 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
149.126.76.193 – – [08/Oct/2012:14:38:05 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
149.126.76.193 – – [08/Oct/2012:14:38:05 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
149.126.76.193 – – [08/Oct/2012:14:38:05 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
199.83.128.231 – – [08/Oct/2012:14:38:05 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
199.83.128.231 – – [08/Oct/2012:14:38:05 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"
198.143.33.2 – – [08/Oct/2012:14:38:06 +0000] "GET / HTTP/1.1" 301 184 "-" "Incapsula Uptime Monitor"
198.143.33.2 – – [08/Oct/2012:14:38:06 +0000] "GET / HTTP/1.1" 403 140 "-" "Incapsula Uptime Monitor"

Now I try to calculate Incapsula request:

1
sed -n ‘s!.* "GET.* "([[:alnum:].]+/*[[:digit:].]*)[^"]*"$!1!p’ /var/log/nginx/access.log | sort | uniq -c | sort -rfg

And this is the results:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
  81321 Incapsula
  10523 Mozilla/5.0
   3311 Mozilla/4.0
    573 Googlebot
    209 Mediapartners
    161 Opera/9.80
     99 Java/1.6.0
     94 SocialSearcher/0.1
     87 msnbot
     84 Feedfetcher
     79 Google/2.0.1.10455
     20 facebookexternalhit/1.0
     16 SAMSUNG
     13 sam
      9 Opera/9.30
      9 Apple
      7 DoCoMo/2.0
      7 AndroidDownloadManager
      6 Tiny
      5 Sogou
      5 Apache
      4 TosCrawler/
      4 BlackBerry8520/5.0.0.592
      3 Mozilla/4.61
      3 Google/2.0.0.10163
      3 facebookexternalhit/1.1
      3 Baiduspider
      2 Yeti/1.0
      2 YahooCacheSystem
      2 Wget/1.11.4
      2 Mozilla/4.7
      2 Mozilla/0.6
      2 ia
      2 Googlebot/2.1
      1 SonyEricssonK660i/
      1 Ruby
      1 rogerbot/1.0
      1 OperaMini/7.0.4.44138
      1 Opera/9.64
      1 Nutraspace/
      1 NokiaX2
      1 NokiaC3
      1 micromaxq5/
      1 MAUI
      1 InternetSeer.com
      1 ImageSearcherFreeS/1.0.5
      1 Fun
      1 Dalvik/1.2.0
      1 Clipish/9.29.1
      1 BlackBerry9000/4.6.0.167

I can’t believe it, it make my server overload and over and over. So, then I decide to un-subscribe from Incapsula, but guess what, their crawlers still flooding my servers. Then, I take action by blocking Incapsula IP by processing my NGINX logs:

1
cat /var/log/nginx/access.log | grep "Incapsula Uptime Monitor" | awk -F’"’ ‘{print $1}’ | cut -d’ ‘ -f1 | sort | uniq -c | sort -rn

And to block Incapsula IP is by putting this into “/etc/nginx/blockips.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
deny 149.126.0.0/8;
deny 199.83.132.0/24;
deny 199.83.128.0/24;
deny 199.83.132.0/24;
deny 198.143.34.0/24;
deny 198.143.33.0/24;
deny 149.126.76.0/24;
deny 198.143.32.0/24;
deny 149.126.75.0/24;
deny 103.28.248.0/24;
deny 198.143.32.0/24;
deny 149.126.73.0/24;
deny 149.126.75.0/24;
deny 149.126.76.0/24;
deny 149.126.77.0/24;
deny 149.126.78.0/24;
deny 198.143.32.0/24;
deny 198.143.33.0/24;
deny 198.83.128.0/24;
deny 198.83.132.0/24;
deny 198.83.131.0/24;
deny 149.126.70.0/24;
deny 149.126.71.0/24;
deny 149.126.72.0/24;
deny 198.143.32.0/24;
deny 149.126.70.0/24;
deny 149.126.0.0/8;
deny 149.126.72.0/24;
deny 149.126.0.0/8;
deny 198.143.30.0/24;
deny 198.143.31.0/24;
deny 198.143.32.0/24;
deny 198.143.33.0/24;
deny 103.28.248.0/24;
deny 149.126.70.0/24;
deny 149.126.71.0/24;
deny 149.126.72.0/24;
deny 198.143.32.0/24;
deny 198.143.33.0/24;
deny 198.83.128.0/24;
deny 198.83.129.0/24;
deny 198.83.130.0/24;
deny 198.83.132.0/24;
deny 212.199.180.0/24;
deny 198.143.33.0/24;

And inside “http” in /etc/nginx/nginx.conf, you can include this file by:

1
include /etc/nginx/blockips.conf;

yodi

,

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.