meta-externalagent: Meta’s AI crawler explained
meta-externalagent crawls the web for Meta’s AI training and content indexing. Its user agent, robots.txt rules, and Meta’s other fetchers.
Updated
meta-externalagent is Meta’s web crawler for use cases such as training AI models and improving products by indexing content directly. You control it with a User-agent: meta-externalagent group in robots.txt. It is separate from the crawler that builds link previews, so blocking it does not break how your links look when shared on Facebook.
| Owner | Meta |
|---|---|
| Purpose | Training |
| robots.txt token | meta-externalagent |
| Follows robots.txt | Yes (controlled by robots.txt) |
| Published IP ranges | None published |
| If you block it | Your pages are not crawled for Meta’s AI training or its direct content indexing. |
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
meta-externalagent/1.1Meta’s documentation shows both strings; the shorter one is also seen in logs. Match on the token.
Which Meta crawlers are there?
| Agent | What Meta says it does | robots.txt |
|---|---|---|
meta-externalagent | Crawls for AI training and for improving products by indexing content directly | Controlled by robots.txt |
meta-externalfetcher | Fetches individual links at a user’s request, including for AI agents | May bypass robots.txt |
facebookexternalhit | Builds link previews for content shared on Meta’s apps | May bypass it for security and integrity checks |
How do I block meta-externalagent?
User-agent: meta-externalagent
Disallow: /
User-agent: facebookexternalhit
Allow: /Can I block meta-externalfetcher?
Meta says this fetcher may bypass robots.txt because a user asked for the page, the same position OpenAI and Perplexity take for their user-triggered agents. To stop it, block the user agent at your server or CDN.
How do I verify Meta’s crawlers?
Meta’s page recommends allowlisting by user-agent string or, more securely, by IP address. We did not find a published IP list for meta-externalagent there, so check the page for current details before relying on IP rules.
Should you block meta-externalagent?
It is a training-oriented crawler with no documented role in showing or linking your site in search answers, so blocking it costs little visibility. Allow it if you want Meta’s AI to know your content. Either way, leave facebookexternalhit alone. Check your rules with the AI crawler checker.
Sources
Common questions
Will blocking meta-externalagent break link previews on Facebook?
No. Link previews when someone shares your URL on Facebook, Instagram or Messenger come from a different crawler, facebookexternalhit. Keep that one allowed if you want previews.
Does Meta publish IP ranges for its crawlers?
Meta's crawler page suggests allowlisting by user agent or IP address, but we did not find a published IP list for meta-externalagent on that page.
How quickly does Meta see robots.txt changes?
Meta says its crawlers may cache robots.txt, so allow up to 24 hours for changes to take effect.