A few days ago, while reviewing htop on a production server, I came across a suspiciously high load for the actual traffic it was supposed to be receiving. A quick look at the nginx logs and there was the culprit: an endless parade of bots, AI scrapers, and SEO crawlers hammering the site around the clock. No human traffic, just robots doing their thing.
The server in question acts as a reverse proxy in front of several websites, giving it a privileged position: all traffic passes through it before reaching the backends. That makes it the ideal place to set up a first line of defense and filter out the junk before it touches anything else.
The idea is simple: many of these bots still (honestly) identify themselves with their User-Agent. GPTBot, ClaudeBot, Bytespider, AhrefsBot, SemrushBot, and friends leave their signature in every request. So why not use exactly that as a filter and hand them back a blunt 403?
Note: this is running on Debian
Before diving in, one important note because it affects where the files go. This setup is on Debian, and on Debian (and derivatives like Ubuntu) the default nginx.conf already includes a line like this inside the http block:
include /etc/nginx/conf.d/*.conf;
That means any .conf file we drop in /etc/nginx/conf.d/ gets loaded automatically. Very convenient for keeping config neatly split rather than stuffing everything in one place. If you are on another distro or FreeBSD, check that include exists (or add it), because otherwise the file will not load by magic.
The filter file
We create the file /etc/nginx/conf.d/bad-bots.conf with the following content. It uses the map directive to create a $blocked_agent variable that will be 1 if the User-Agent matches any of the patterns, and 0 otherwise:
map $http_user_agent $blocked_agent {
default 0;
# --- AI / LLM scrapers ---
"~*openai" 1; # OpenAI
"~*GPTBot" 1; # OpenAI
"~*ChatGPT-User" 1; # OpenAI (plugins)
"~*OAI-SearchBot" 1; # OpenAI
"~*ClaudeBot" 1; # Anthropic
"~*Claude-Web" 1; # Anthropic
"~*anthropic-ai" 1; # Anthropic
"~*Google-Extended" 1; # Google AI (Bard/Gemini training)
"~*Applebot-Extended" 1; # Apple AI
"~*Bytespider" 1; # ByteDance / TikTok (very aggressive)
"~*CCBot" 1; # Common Crawl
"~*PerplexityBot" 1; # Perplexity
"~*Perplexity-User" 1; # Perplexity
"~*Amazonbot" 1; # Amazon
"~*FacebookBot" 1; # Meta (AI training)
"~*meta-externalagent" 1; # Meta external agent
"~*meta-externalfetcher" 1; # Meta external fetcher
"~*Diffbot" 1;
"~*Omgilibot" 1;
"~*Omgili" 1;
"~*Applebot" 1;
"~*ImagesiftBot" 1;
"~*cohere-ai" 1;
"~*cohere-training-data-crawler" 1;
"~*Timpibot" 1;
"~*YouBot" 1;
"~*BacklinksExtendedBot" 1;
"~*YandexBot" 1;
"~*DataForSeoBot" 1;
"~*facebookexternalhit" 1;
# --- Noisy SEO / marketing crawlers ---
"~*AhrefsBot" 1;
"~*SemrushBot" 1;
"~*DotBot" 1; # Moz
"~*MJ12bot" 1; # Majestic
"~*BLEXBot" 1;
"~*MegaIndex" 1;
"~*SeznamBot" 1;
"~*serpstatbot" 1;
"~*PetalBot" 1; # Huawei
"~*ZoominfoBot" 1;
"~*Barkrowler" 1;
"~*IbouBot" 1;
"~*AliyunSecBot" 1;
"~*AwarioBot" 1;
"~*Qwantbot" 1;
# --- Generic download / scraping tools ---
"~*wget" 1;
"~*curl" 1; # warning: blocks legitimate curl, remove if you use it
"~*python-requests" 1;
"~*python-urllib" 1;
"~*Scrapy" 1;
"~*Go-http-client" 1;
"~*libwww-perl" 1;
"~*HTTrack" 1;
"~*WebCopier" 1;
"~*WebReaper" 1;
"~*Nutch" 1;
# --- Empty or suspicious ---
"" 1; # empty User-Agent
}
A couple of details about the syntax so you know what is happening:
The ~* prefix on each pattern means it is a case-insensitive regular expression, so GPTBot, gptbot, or GPTBOT will all match. The default 0; is the key: by default everything is let through, and we only mark as blocked (1) what explicitly matches. It is a blacklist approach, more permissive than the alternative, but easy to maintain.
The last entry, "" 1;, blocks requests that arrive without a User-Agent. A legitimate browser or client almost always sends one, so an empty User-Agent is usually a sign of something automated and poorly written.
Activating the filter on each vhost
The map directive alone blocks nothing: it just defines the variable. To make it take effect you need to use it where you want it. Inside the server { ... } (or the specific location) of each vhost where you want to apply the filter, add:
if ($blocked_agent) {
return 403;
}
And that is it. If the request User-Agent triggered the map, nginx responds with a 403 Forbidden and does not bother passing the request to the backend. The beauty of having it on the reverse proxy is that you protect all the sites behind it at once, without touching each application individually.
Variant: return 444 (close without responding)
The 403 is correct and semantically honest, but it has a small cost: nginx generates and sends an error response to the client. If you want to spend the minimum resources possible on these bots, nginx offers a very handy non-standard code, 444, which immediately closes the connection without sending any response:
if ($blocked_agent) {
return 444;
}
The bot is left with the connection cut dead — no body, no headers, nothing. It uses less bandwidth and less CPU than returning an error page, and gives less information to whoever is on the other end. I prefer this variant for clearly automated bots; I reserve 403 for cases where I want the forbidden to be explicitly recorded.
Before reloading, always verify the config is valid:
# nginx -t
And if everything is fine, reload without dropping the service:
# systemctl reload nginx
Log the blocks to know what you are catching
Blocking is great, but flying blind is uncomfortable: how do you know if you are filtering the right things, or if you are accidentally breaking something legitimate? The trick is to send the blocks to their own log so you can audit them without polluting the general access_log.
nginx lets you conditionally write to an access_log with if=, and this is perfect because we already have the $blocked_agent variable. First we define a custom log format inside the http block (for example, in bad-bots.conf itself or in nginx.conf):
log_format blockedhosts '$remote_addr - [$time_local] "$request" $status "$http_user_agent"';
And then in the vhost, right next to the if, we write to that log only when the request has been flagged as blocked:
if ($blocked_agent) {
return 444;
}
access_log /var/log/nginx/blocked_agents.log blockedhosts if=$blocked_agent;
The key is the if=$blocked_agent at the end: nginx will only write a line to blocked_agents.log when that variable is 1. Legitimate traffic does not touch this file at all. So you get a clean, dedicated record of everything you are dropping.
From there, a quick look tells you a lot. For example, to see which User-Agents are the most persistent:
# awk -F'"' '{print $6}' /var/log/nginx/blocked_agents.log | sort | uniq -c | sort -rn | head
It is a great way to keep refining the list: if you see something falling that should not, remove it from the map; and if you spot a new bot causing trouble, add it. The log becomes your source of truth for iteration.
Tip: do not forget to add
blocked_agents.logto yourlogrotate, or over time it can grow more than you would expect.
Review and adapt the list to your needs
This is the list that works for me, but do not copy it blindly. Go through it carefully and adapt it to your case:
If you use curl for your own healthchecks, monitoring, or deployments, that "~*curl" 1; will give you a nasty surprise by blocking legitimate requests. Same with wget, python-requests, or Go-http-client if you have internal scripts that hit the site. The list is yours: add, remove, and test.
The result: -50% load
The result on that production server? After applying the filter, the server load dropped by more than 50%. Yes, more than half of what it was processing was, plain and simple, automated junk that contributed absolutely nothing. Makes you think about how much of today’s internet traffic is just robots talking to robots.
This is not foolproof (important)
Here comes the obligatory caveat, because it is worth having realistic expectations. Filtering by User-Agent is a cheap, fast, and surprisingly effective first line of defense against bots that identify themselves honestly… but nothing more.
The User-Agent is an HTTP header that the client sends and can be spoofed in a second. Any scraper with even a minimal interest in bypassing the filter just needs to send a normal browser User-Agent and it will pass right through. In other words: this stops polite bots and generic tools, but will not stop someone who really wants in.
For serious protection against malicious traffic, aggressive scraping, or attacks, you need to go a step further and consider a WAF (Web Application Firewall) or more complete mitigation solutions. Some options worth considering:
Cloudflare (with its bot and AI scraper blocking mode), Anubis (a lightweight proof-of-work tool designed precisely against AI crawlers, very popular lately), BitNinja, or Sucuri, to name just a few. Each has its own approach, advantages, and drawbacks.
But in the meantime, this nginx filter is a great first step: free, no external dependencies, easy to maintain, and it eliminates a frightening amount of noise. And sometimes, cutting a server load in half with four lines of config is exactly what you needed.