Common Crawl publishes a free public web archive, and CCBot is the crawler that builds it. A log line naming CCBot can come from that crawler or another program using the same label. Your robots.txt file tells the real CCBot whether to slow down or stay away.
What Common Crawl publishes
Common Crawl is a 501(c)(3) nonprofit providing web crawl data for research and analysis. The FAQ says researchers, companies, and individuals may use that copy. Crawl data is stored on Amazon S3 for bulk download and for direct processing. Each crawl cycle processes about three billion pages, according to the about page.
A 2025 Common Crawl note says datasets from the archive appear in over 10,000 papers. Monthly crawls add several billion new pages, according to that same note. Those pages come from a frontier the note describes as more than one trillion URLs.
Raw captures are stored as WARC files, including the full HTTP responses. WET files hold plaintext extracted from HTML. WAT files hold extracted metadata, including HTTP headers and link graphs.
Why training sets use this archive
The about page describes filtered Common Crawl data as a training source for many models. Its write-up cites a 2024 Mozilla study of 47 large language models from 2019 through 2023. At least 64 percent of those models were trained on filtered versions of the data. Third parties do the filtering, so a raw crawl is not a finished training set.
Named derivatives on that page include C4 for T5 and the CCNet pipeline for LLaMA. BigScience trained BLOOM with OSCAR, a corpus derived from Common Crawl. Falcon training used RefinedWeb, which was also processed from this same archive.
How CCBot fetches a page
CCBot is a Nutch-based crawler that uses Apache Hadoop, which the FAQ describes. Map-Reduce builds the candidate list, sorts it by host, and sends it to crawler servers. Before a fetch, CCBot checks robots.txt and continues only when the page is allowed. Allowed fetches use HTTP GET over HTTP/1.1, or HTTP/2 only when the URL is HTTPS.
Connections are supported on both IPv4 and IPv6. Redirect chains can run up to four hops, or five when robots.txt is fetched. JavaScript is not executed during the fetch, and cookies are not used either. When the server returns 429 or a 5xx status, adaptive backoff slows further requests.
URL order depends on harmonic centrality and on PageRank together. Both scores come from Common Crawl's web graph, as the about page explains.
Which user-agent string to expect
Both the CCBot page and the FAQ publish one current user-agent string. Look for CCBot/2.0 (https://commoncrawl.org/faq/) in the User-Agent header on the request. Older crawls identified themselves with the string CCBot/1.0 (+https://commoncrawl.org/bot.html) instead. Later versions may change that number, which the FAQ says can happen.
For robots.txt, the exclusion token is CCBot, not the full versioned string. Put that token alone on the User-agent line for this crawler. The CCBot reference lists its user-agent string, robots token, and vendor sources.
How to slow CCBot down
By default, CCBot already waits a few seconds between requests to one site. Crawl-delay is non-standard, and this Common Crawl post says CCBot obeys it. In the FAQ, a value of 2 limits CCBot to one request every two seconds. Place these two lines in robots.txt: User-agent: CCBot and Crawl-delay: 2.
What a CCBot block changes
A full block is a CCBot group whose only rule is Disallow: /. After that rule is live, the FAQ says the crawler stops fetching your pages. Checks of robots.txt continue, so a later edit can change the decision. The AI optimization post says missing CCBot rules are treated as permission to crawl. An explicit allow group is optional, and the same post shows User-agent: CCBot with Allow: /.
User-agent: CCBot
Disallow: /Read the 2024 opt-out post: the block also covers search-indexing uses of this archive. Separate bot tokens stay in separate groups on the robots.txt guide.
How to check a real request
Some crawlers falsely present themselves as CCBot, which the CCBot page warns about. Compare the source IP with the published JSON feed before you trust the name. The FAQ lists ranges dated 2026-08-11: one IPv6 prefix and four IPv4 prefixes.
Listed IPv4 ranges are 3.41.188.32/29, 18.97.9.168/29, 18.97.14.80/29, and 18.97.14.88/30. Reverse DNS is not supported yet on the IPv6 range 2600:1f28:365:8000::/56. The JSON feed also recommends forward-confirmed reverse DNS for IPv4. Common Crawl demonstrates this on its CCBot page. Its example maps 18.97.14.84 to 18-97-14-84.crawl.commoncrawl.org, then resolves that hostname back to the same address.
Where Aiola shows CCBot
The AI view shows crawl totals, bots, pages, and activity over time. It includes Answers, Indexing, Training and Other buckets, AI-answer referrals, crawler 404s, and discovery-file health.
CCBot's crawl kind maps to Training. The Analytics docs describe the AI view.
Crawlers do not run JavaScript, so the browser tag cannot see them. One server or edge snippet from the setup guide posts each matching hit. Only known crawler user agents are stored.
A recorded crawl is not proof that the vendor cited the page. An empty period is not proof that the vendor cannot reach the site.
Try it in Aiola
Download Aiola free for macOS 12 or later and Windows 10 or later. Hosted Analytics needs Pro or the 7-day trial, and the pricing page lists both. Open the AI view to inspect recorded CCBot requests.
