Dynamic validation of good bot IP addresses

[jsheard]

AFAICT Anubis currently uses hard-coded lists of IP addresses to detect impostors of good bots, which might fall behind the current IPs those services are using if Anubis's update cadence doesn't line up. Bing in particular specifically advises against hard-coding their IP ranges as they are subject to change without any warning.

https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26

Don’t Use Hardcoded IP Addresses or Address Ranges

Use the reverse/forward DNS lookup method you can easily verify that an IP address is coming from Bing.

Or alternatively, you can identify Bingbot by IP address by matching the crawler's IP address to the list of Bingbot IP addresses. Refresh this list daily as we can change this list any time.

Would it be worth implementing DNS-based validation for Googlebot and Bingbot?

[Axelen123]

I wanted this feature for my homelab and have already managed to implement it. I can open a pull request if the maintainers are interested.

[jsheard]

FWIW Mojeek and Yandex also supports DNS validation, though Anubis doesn't currently include a policy file for the latter

https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot
https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26
https://www.mojeek.com/bot.html
https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots.html

[JasonLovesDoggo]

I wanted this feature for my homelab and have already managed to implement it. I can open a pull request if the maintainers are interested.

Sure! We'd love to see it

[magma1447]

FCrDNS Verification Support

I came here to request this exact feature - verifying legitimate crawlers through reverse DNS lookups - and was pleased to find this issue already exists with PR #682 in progress! This is a critical security improvement.

I have a few questions about the implementation:

Timeline & Status

• What's the current status of PR feat(config): Add FCrDNS checker #682? I see it was accepted ~3 weeks ago but has merge conflicts.
• Can we expect a release with this feature included in 2025?

Configuration

• Will this FCrDNS verification be enabled by default, or will it require configuration changes to opt-in?
• Is there any caching of DNS lookup results, or does it rely entirely on DNS server caching? This could impact performance for high-traffic sites.

Scope

• Which crawlers will this DNS verification apply to? Just the ones in _allow-good.yaml (Google, Bing, DuckDuckGo, etc.), or configurable per bot policy?
• Will there be documentation on which crawlers support proper DNS verification vs. which only publish IP ranges?

Security Model

• Once this is implemented, would it be advisable to create a more restrictive policy that only allows verified crawlers (those passing FCrDNS) and blocks everything else claiming to be a bot? This would prevent the current situation where spoofing a bot User-Agent is actually easier than pretending to be a browser.

---

The current vulnerability where curl -A "Googlebot/2.1" bypasses protection entirely is a significant issue, especially with aggressive scrapers from ByteDance, Alibaba, and others that commonly spoof legitimate crawler User-Agents. Really looking forward to this landing!

--

Many thanks regardless. Both for the product and this improvement.

[Axelen123]

The current implementation in the PR caches positive DNS lookups. The default rules use the feature and will add weight to anyone claiming to be a bot with incorrect rDNS.