Who this is for
Docs sites, open-source project sites, forums and wikis, small SaaS and content sites. Anything on usage-billed hosting (Vercel, Netlify, Fly, Render, S3 and CloudFront, serverless functions) where every page fetch costs you money.
AI crawlers re-fetch whole sites, often daily, and many ignore caching hints. On metered hosting that shows up as a bandwidth or invocation spike nobody can explain.
How it works
- Send your latest bill. Add access logs or the botcheck output below if you have them.
- We tell you the crawler share, free. Which bots, how much traffic, what it costs you per month.
- If it’s worth fixing, we fix it on your current host. Blocking, rate limits and caching rules for the crawlers that cost you, with search engines and real visitors left alone. You pay a share of the reduction. If the bill doesn’t go down, you pay nothing.
Check your own logs first
A single Python file, no dependencies. It reads nginx or Apache access logs (plain or .gz) on your machine and sends nothing anywhere.
curl -O https://crawlbill.anas.link/botcheck.py
python3 botcheck.py /var/log/nginx/access.log*
AI crawler operator requests %req bytes %bytes
GPTBot OpenAI 6,033 30.2% 365.6 MB 34.8%
ClaudeBot Anthropic 3,988 19.9% 222.1 MB 21.1%
Bytespider ByteDance 2,996 15.0% 211.0 MB 20.1%
meta-externalagent Meta 1,560 7.8% 77.5 MB 7.4%
AI crawlers total 14,577 72.9% 876.2 MB 83.3%Read the script before you run it. Pass --price-per-gb with your host’s rate for a monthly cost estimate.
Questions
- I’m already behind Cloudflare.
- Turn on its AI crawler blocking. It’s free on every plan, and you probably don’t need us.
- Will this hurt my search ranking?
- No. Googlebot, Bingbot and the other search crawlers aren’t touched. Only crawlers that take content without sending traffic back get blocked or slowed.
- Why not just robots.txt?
- Most well-known crawlers respect it, but plenty don’t, and the user-triggered fetchers often don’t. It also does nothing about caching or crawl rate. It’s step one, not the whole fix.
- What do you charge?
- A share of what your bill goes down by, for a few months. The exact split depends on your size; you see it before anything changes.