Everyday there are so many (thousands of daily) attacks and bots coming from Microsoft Azure and Google Cloud which steals your bandwidth and slows down your website by hackers. Apart from these it’s easier to block by ASN other attackers (Azure has millions of IPs starting with 20.96.x.x). But Googlebot’s ASN is same as Google Cloud origin and Bingbot’s ASN is same as Microsoft Azure which makes hard to block it includes concern of blocking Googlebot & Bingbot mistakenly. Fortunaly Cloudflare WAF has filters to add rule. There are 2 ways to Block Azure & Google Cloud users without blocking Bing & Googlebot.
First is blocking Google Cloud & Azure ASNs
Google Cloud ASN: 15169 and 396982
Microsoft Azure ASN: Mostly 8075, (8068 8069 8070 8071 8072 8073 8074 8075)
Also i recommend checking ASN of IP by this open-source website IP.ME which given by IANA to ProtonTeam, example: https://ip.me/ip/20.96.29.54
i will show blocking Azure:
Bingbot’s hostname: search.msn.com or msnbot (April 2022 Update: it’s bingbot) https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26 https://blogs.bing.com/webmaster/april-2022/Announcing-user-agent-change-for-Bing-crawler-bingbot
Current user-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36
Cloudflare Rule For Blocking Azure:
Solution 1 (Block Azure but “Cloudflare bot check”) | Trustworthy
(ip.geoip.asnum in {8075} and not cf.client.bot)
A firewall rule like this will block all traffic from Microsoft ASNs, but allow known good bots.
want to use all azure ASNs: (ip.geoip.asnum in {8068 8069 8070 8071 8072 8073 8074 8075} and not cf.client.bot)
Solution 2 (Block Azure But Hostname | Fake bingbots may allowed)
(ip.geoip.asnum eq 8075 and not http.host contains “bingbot”)
Update: Microsoft changed user agent msnbot to bingbot in April 2022: https://blogs.bing.com/webmaster/april-2022/Announcing-user-agent-change-for-Bing-crawler-bingbot
Cloudflare Rule For Blocking GoogleCloud:
Solution 1 (Block Google Cloud but “Cloudflare bot check”) | Trustworthy
(ip.geoip.asnum in {15169} and not cf.client.bot)
or below Azure & Google Cloud & Amazon
(ip.geoip.asnum in {8075 396982 15169 16509} and not cf.client.bot)
A firewall rule like this will block all traffic from GoogleCloud ASNs, but allow known good bots.
Other ASN: AS396982
Solution 2 (Block Google But Hostname | Fake googlebots may allowed)
(ip.geoip.asnum eq 15169 and not http.host contains “googlebot”)
One Rule For Blocking Both and allow legit bots
How Cloudflare Filters Bots
Based on https://cloud.google.com/compute/docs/faq#find_ip_range (https://www.gstatic.com/ipranges/cloud.json), the AS that announces the prefixes is the same for both Google Cloud and their primary services:
Google Cloud listing
Googlebot IP
Although, you might notice that the “host” of the IP is different. For Google Cloud, the IP’s host ends in googleusercontent.com, while Googlebot ends in googlebot.com. This method is precisely how Google expects people to verify that content coming from Google’s network is legitimate traffic from Googlebot.
https://support.google.com/webmasters/answer/80553?hl=en 5
As for how this matters in a Firewall rule, CF does verify Googlebot using the method Google recommends, and will appropriately set “verified bot” to false since it’s a fake Googlebot.
So, to go about making sure fake Bingbot and fake Googlebot are blocked, you should modify it to be:
(ip.geoip.asnum in {8075 15169} and not cf.client.bot)
Your firewall rule above would only trigger if the ASN were both numbers at the same time, which wouldn’t ever trigger.
If you have the Pro plan or above with the WAF and the “Cloudflare Specials” ruleset enabled, the managed rule 100035 is already set up to block fake Googlebot.
IP.ME or ipinfo.io
Also i publish edmalicious bad ASNs to block list here. Until that you may check by this website: https://cleantalk.org/blacklists/asn
Cloudflare Known Bots list: https://radar.cloudflare.com/verified-bots
Additional info: DuckDuckGo bots use Microsoft Azure IPs too and it’s listed in Cloudflare’s known bots list, so DuckDuckBot will be allowed: https://help.duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/