CCBot: Common Crawl’s bot and AI training data

CCBot is the crawler of Common Crawl, a non-profit that builds a free, open repository of web crawl data anyone can download. That archive is a common ingredient in AI training datasets, so blocking CCBot is a way to keep your new pages out of one widely used source of training data. It follows robots.txt, including Crawl-delay.

CCBot/2.0 (https://commoncrawl.org/faq/)

An older version identified itself as CCBot/1.0 (+https://commoncrawl.org/bot.html). Common Crawl warns that some other crawlers falsely claim to be CCBot.

What does CCBot do?

It crawls the web on a regular schedule and Common Crawl publishes the results as open datasets. Researchers, search start-ups and AI developers then use those datasets. Your content’s use depends on who downloads the archive, which is outside Common Crawl’s control and yours.

How do I block CCBot?

User-agent: CCBot
Disallow: /

How do I slow it down instead?

Common Crawl’s FAQ says CCBot honors Crawl-delay, for example one request every two seconds:

User-agent: CCBot
Crawl-delay: 2

How do I verify the real CCBot?

Common Crawl publishes its crawler IP ranges at index.commoncrawl.org/ccbot.json. For IPv4 addresses you can also do a reverse DNS lookup: real CCBot hosts resolve under crawl.commoncrawl.org; confirm with a forward lookup. Common Crawl notes its IPv6 addresses don’t have reverse DNS.

Should you block CCBot?

If you want to limit training use of your content, CCBot is one of the most effective single rules, because its archive is used so widely. If you want AI models to know your brand, allowing it helps. It has no effect on search engines or on AI search crawlers such as OAI-SearchBot. Check your rules with the AI crawler checker.