Baiduspider collects pages and builds Baidu's search index. You can limit the whole site or selected paths in robots.txt, or reject a matching request at a WAF. Confirm the caller before you trust the User-Agent.
When a block changes Baidu search
Baidu's crawler help describes Baiduspider as the program that collects pages and builds the index. Its webmaster FAQ says those indexed pages are what search can return.
The FAQ lists a user-agent for each Baidu product. PC search and mobile search both use Baiduspider. Image search uses Baiduspider-image. News search uses Baiduspider-news. Video search uses Baiduspider-video. Baidu says that video token does not currently support robots.txt rules.
Baidu says this group blocks crawling from Baidu:
User-agent: Baiduspider
Disallow: /Baiduspider-cpro and Baiduspider-ads do not follow robots.txt. Baidu says those two are not added to the search index. A Baiduspider-image group can still allow a path, such as /image/, beside the broader block.
Baidu says a robots.txt change can take about 48 hours to stop new crawling. Its DNS data also refreshes on a cycle, so crawling can continue until then. Pages already in the index can take several months to disappear. For urgent removal, the FAQ points to Baidu's webmaster feedback form.
Baidu's robots.txt page describes another case. A blocked URL can still appear when another site links to it. Baidu will not crawl, index, or show that page's content. The visible text can be the other site's description.
Block Baiduspider in robots.txt
Serve robots.txt at the site root. Baidu checks that file before it crawls the host. Add the group above to the file you already serve. Leave your other user-agent groups in place.
A missing file, or an empty file, leaves the site open to crawlers. Create one only when you need to exclude something. Each group starts with User-agent, then Allow or Disallow lines. Put every excluded path on its own Disallow line. One line that lists two paths does not apply both.
Disallow is a prefix. Disallow: /help also matches /help.html, /helpabc.html, and /help/index.html. Disallow: /help/ matches /help/index.html and does not match /help.html. Check the paths your site really serves. Baidu reads Allow and Disallow from the top, and the first match wins. Baiduspider also accepts * and $ in a path. * matches any run of characters, and $ matches the end.
Fetch the live robots.txt on the host that received the request. Confirm the Baiduspider group matches the policy you published.
Baidu says the crawler has the same access as an ordinary visitor. It cannot fetch a page a visitor cannot open. Aiola's robots.txt guide treats Disallow as a request to a cooperative crawler. Keep private pages behind authentication.
Check the IP before you trust the name
Start with the source IP on your server or edge log. Keep the path, status, and full User-Agent with it. Baidu's webmaster FAQ says to reverse-resolve that IP. On Linux, run host. On Windows, run nslookup. On macOS, run dig -x. A real hostname ends in .baidu.com or .baidu.jp. Any other name is fake.
Baidu's host example returns baiduspider-123-125-66-120.crawl.baidu.com. Treat that name as an example, not a permanent allowlist. A name that only contains "baidu" fails the suffix test.
Aiola's Baiduspider reference adds a forward lookup. Baidu's pages do not require that second query. A PTR record alone can be controlled by an attacker.
- Reverse-resolve the recorded source IP.
- Require the hostname to end in
.baidu.comor.baidu.jp. - Resolve that full hostname forward.
- Require the forward result to include the original IP.
Aiola's page shows both steps with host. If either lookup fails, leave the request unconfirmed. Do not allow it only because the header says Baiduspider.
Aiola lists this sample User-Agent: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html). A Developer Center article prints the same sample. That article is not Baidu's crawler help. Use it to find log lines, then decide identity with DNS.
Block the header at your WAF
Baidu says Baiduspider follows robots.txt. That file does not reject the connection. Use a WAF when the request itself must fail. Keep the robots.txt group as the stated preference, and add the edge rule separately.
Cloudflare recommends custom rules instead of User Agent Blocking rules. Its example matches http.user_agent with eq, and the action can be Block. String comparison is case-sensitive unless you change case. The lower() function converts a string field to lowercase.
A bot-rule page shows http.user_agent contains with Block. Bot Management on that page requires Enterprise. The expression below does not use a bot score. It joins lower(), contains, and http.user_agent. Cloudflare did not publish it as a Baiduspider rule. Review it in your zone before you deploy it.
lower(http.user_agent) contains "baiduspider"Set the action to Block. The rule matches header text and does not run the DNS checks. For a partial block, add starts_with on http.request.uri.path. Then compare the rule's matches with the paths you meant to stop.
Compare logs with search results
After you publish, read the live robots.txt again. If a WAF rule is active, review the requests it matched. Compare a similar window before and after the change.
A missing search listing is not enough proof. Check links from other sites, the 48-hour crawl stop, and the multi-month index wait separately. Logs show paths and statuses. A search check shows exposure.
If requests continue, test one caller before you widen the block. Repeat the suffix test and the forward lookup. A failed DNS check means the name is still unconfirmed.
Read Baiduspider rows in Analytics
Aiola Analytics keeps browser traffic and crawler hits in one project. Those sets stay separate metrics. Crawlers do not run the browser tag. You record them from the server or the edge. The setup guide sends a known User-Agent to the bot ingest endpoint. Its examples match Baiduspider before they send the hit.
Unknown crawler names are not stored, and the host must match the configured domain. Aiola Analytics names 70 crawlers, including Baiduspider. Baiduspider is in the Indexing bucket. The AI view shows crawl totals, bots, pages, and a timeseries.
A row means that crawler requested the recorded URL. Classification uses the User-Agent, which can be spoofed. Analytics does not run Baidu's reverse-DNS check. No row means no matching beacon arrived in that period. It does not prove Baidu was blocked or never fetched the page.
Hosted Analytics needs Pro or the 7-day trial. The trial is for a first-time subscriber on Pro Monthly with 100,000 events. Checkout collects a card. Without Pro or an active trial, collectors return 402 and do not store the event. GUI Lifetime does not include hosted Analytics.
Crawler totals are not the monthly event allowance. Billable events are pageviews and custom events. You only see a request your capture point sends. A block in front of that point does not create a row, so read the WAF log for those rejects.
Try it in Aiola: Download Aiola for macOS 12+ or Windows 10+. The download is free. Hosted Analytics needs Pro or the 7-day trial.
