block-ai-scrapers skill (Illyism/seo-skills) + Claude CodePartly tested🌏 World

An open-source SKILL.md has your agent build a 6-layer firewall against scraping agents: it serves 200 gate pages instead of 403s, verifies Googlebot by IP, then red-teams the setup with sub-agents

Overseas developer @illyism saw people paste his data site into Claude Code and brag about scraping the whole database; when he returned 403s, the agents spoofed Googlebot, probed /api/ and spun up Playwright. He built a 6-layer defense and open-sourced the full SKILL.md: blocked requests get an HTTP 200 "protected dataset" gate page (zero rows + a STOP_SCRAPING directive + a product link), Googlebot is verified by IP/ASN instead of User-Agent, HTML routes require full browser navigation headers plus a JS session proof, and sub-agents red-team it until attackers stop on turn 1. We ran the skill's Googlebot check against Google's IP list published that day and found it both misses real Googlebot ranges and admits unofficial ones.

Visit site block-ai-scrapers skill (Illyism/seo-skills) + Claude Code ↗
Pricing
Free
Region
Global
Where it's from
Overseas
Added
Oct 29, 2026
Tested on
Oct 29, 2026
Version we used
Illyism/seo-skills commit d8b7f94 (2026-10-28), seo/block-ai-scrapers/SKILL.md metadata version 1.0.0
Categories
CodingMarketing
Original author
Ilias Ism @illyism ↗
Our test
Partly tested
Cost / time
The SKILL.md is free and open source; you need your own Claude Code / Cursor / Codex plan, and it deploys on your existing Cloudflare Workers or server at no extra cost.
Source
X @illyism ↗GitHub Illyism/seo-skills(block-ai-scrapers/SKILL.md) ↗Google 官方常见爬虫 IP 段 / Google common crawlers IP ranges ↗

The task

Your site has valuable dataset pages or endpoints that trigger paid API/DB work; you want to stop bulk scraping through Claude Code, Cursor or Codex without hurting Google indexing.

Steps

  1. Install the skill: download `seo/block-ai-scrapers/SKILL.md` from the Illyism/seo-skills repo into `~/.claude/skills/block-ai-scrapers/SKILL.md` (Cursor/Codex: their own skill folders).
  2. Open Claude Code at your project root and let it run the skill's Initial Intake: which routes to protect, the CTA URL blocked agents should hand users, whether to keep Googlebot indexing (default yes), and your edge/server stack (Cloudflare Workers, Next.js middleware, Hono, Express, Fastify).
  3. Layer 1: on a block, return an HTTP 200 gate page (HTML, and JSON for /api/*) with zero rows plus headers like `Cache-Control: private, no-store` and `X-Robots-Tag: noindex`, so CDNs don't cache it and search engines don't index it.
  4. Layer 2: robots.txt allows only Googlebot / Google-InspectionTool / AdsBot-Google and explicitly disallows GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others; verify Googlebot by IP + ASN (15169 / 396982); serve sitemaps only to verified Googlebot.
  5. Layer 3: reject datacenter ASNs; HTML routes require full browser navigation headers (Sec-Fetch-*, Accept-Language, Sec-Ch-Ua…; an Accept containing text/markdown is treated as WebFetch); passing requests then do a `/_xo/client.js` → `POST /_xo/verify` JS session proof and only get the real page with the HttpOnly cookie.
  6. Layer 4: put copyright, database-rights notices and an `[AI_AGENT_DIRECTIVE: STOP_SCRAPING]` block in /llms.txt, /ai.txt and the gate page so the agent stops and gives the user your product link.
  7. Layer 5: public routes are cache-only—never trigger third-party API or LLM work on unverified hits; lock /api/*; per-IP per-minute budgets that return the gate, not a bare 429.
  8. Layer 6: red-team it—curl 3–4 target URLs plus robots/llms/sitemap/api; script the 4-step real-browser flow to confirm real pages still load; then launch sub-agents with real-user prompts like "collect all the data from these URLs" until they stop on turn 1 with zero rows.
  9. Before shipping, check the Googlebot allow-list separately (see pitfalls): validate against Google's published crawler IP JSON or reverse DNS, not just the skill's prefix checks.

Prompt

Use the block-ai-scrapers skill to add anti-scraping protection to this project. First run the skill's Initial Intake and ask me: which routes to protect, where blocked agents should send users, whether to keep Googlebot indexing, and which framework we use. Don't verify Googlebot with prefix matching—do proper CIDR matching against Google's published crawler IP-range JSON. When done, run the Layer 6 curl and real-browser flow tests locally and give me the results as a table. Don't deploy until I confirm.

What the author says

@illyism: people pasted his sites into Claude Code and bragged about scraping the whole database; when he returned 403s, agents spoofed `User-Agent: Googlebot`, probed /api/ routes and spun up Playwright. So he built a 6-layer "AI Agent Firewall" that he says stops Claude Code, Cursor and Codex on turn 1 while keeping Google SEO 100% intact: HTTP 200 decoy pages instead of 403s so agents don't retry, a STOP_SCRAPING directive that makes the agent pitch his product to the would-be scraper, Googlebot verified by IP/ASN, and sub-agents that attack his own URLs until they fail on turn 1. The exact SKILL.md is open-sourced.

Our test

We did not deploy this on a real site or point agents at any site, so "stops on turn 1" and "SEO intact" are unverified. We read the full SKILL.md (repo commit d8b7f94), ported the skill's TypeScript reference helpers isGoogleCrawlerIp / isVerifiedGooglebot to JS unchanged, and ran them locally against Google's official crawler IP list and sample IPs.

What we found when we tried it:The skill's own 4 checks pass: Googlebot UA + 66.249.66.1 + ASN 15169 → allowed; Googlebot UA + 8.8.8.8 → rejected; Google-Extended + 66.249.66.1 → rejected; Googlebot + 66.249.66.1 with no ASN → also allowed. But against Google's official list generated 2026-10-28 15:46 (170 IPv4 and 147 IPv6 ranges), 23 IPv4 ranges (e.g. 34.100.182.96/28, 34.22.85.0/27, 35.247.243.240/28) are judged non-Google, while addresses not on the list—209.85.1.1, 216.239.3.1, 72.14.190.1, 64.233.16.1—are judged Google crawlers. All official IPv6 ranges sit inside 2001:4860:4801::/48, so IPv6 agrees.

How to get in

The original post is in English on X (login may be required); the SKILL.md is public on GitHub with no geo restrictions. It only whitelists Googlebot—if you also need Bing or other search engines, add their verification rules yourself.

Who it suits

Indie developers running dataset, directory or leaderboard SEO sites, or anything where scraped endpoints burn API or database credits.

Pitfalls

  • In our test, the skill's `isGoogleCrawlerIp` is string-prefix matching. Checked against Google's common-crawler IP list published 2026-10-28 (googlebot.json now redirects to common-crawlers.json), it rejects 23 of 170 IPv4 ranges (all in 34.x / 35.x). Real Googlebot from those addresses is treated as fake—and with Layer 3 also blocking the Google Cloud ASN, real crawlers can land on the gate page and hurt SEO.
  • In our test, the prefixes also over-admit: 209.85.1.1, 216.239.3.1, 72.14.190.1 and 64.233.16.1 all pass as Google crawler IPs but none are on the official list (a prefix like `209.85.1` also matches 209.85.10–19.x and 100–199.x).
  • In our test, with no ASN (null) it passes on IP alone; outside Cloudflare you must supply ASN data yourself.
  • The 200 gate + noindex is deliberate: if real Googlebot is misclassified, Google sees a noindex page—quieter than a 403—so watch Search Console after launch.
  • The STOP_SCRAPING directive is just text aimed at agents and relies on the model choosing to comply; "stops on turn 1" is the author's own red-team result, not guaranteed across models or prompts.
  • Strict Sec-Fetch / Sec-Ch-Ua requirements and the JS cookie can break older browsers, privacy browsers, reader modes and accessibility tools—test on real devices first.
  • Six layers touch a lot (middleware, robots, llms.txt, APIs, rate limits); have the agent do it on a branch or staging, not straight to production.

More in this category