Bytespider: ByteDance’s crawler and how to block it

Bytespider is a web crawler widely attributed to ByteDance, the company behind TikTok, and commonly listed among AI training crawlers. Unlike the other bots on this site, it has no official documentation we could find: no published user-agent string, purpose, IP ranges or robots.txt policy. That makes it the one crawler where you should not rely on robots.txt alone.

What is known, and what isn’t?

• Known: requests identify themselves with the token Bytespider in the user-agent string.
• Not officially documented: the full user-agent string, what the data is used for, whether it honors robots.txt, and which IP addresses it uses. Third-party sources make claims about all of these; we have not been able to confirm them with ByteDance, so we don’t repeat them.

How do I block Bytespider?

Add a robots.txt rule to record your wishes:

User-agent: Bytespider
Disallow: /

Then enforce it at the server, since compliance is undocumented. For example, on nginx:

if ($http_user_agent ~* "Bytespider") {
    return 403;
}

Most CDNs and web application firewalls, such as Cloudflare, can do the same with a rule that matches the user agent, and many offer a managed setting for blocking AI crawlers.

How do I verify it?

There is no official IP list or reverse DNS pattern to verify against. You can only identify it by the user agent, which anyone can fake. That is fine for blocking (blocking a fake costs you nothing) but means you can’t confirm a request really came from ByteDance.

Should you block Bytespider?

If you are blocking AI training crawlers, include it: it offers no documented search or citation benefit to weigh against the block. Make sure your rule doesn’t also catch crawlers you want, such as OAI-SearchBot or PerplexityBot. Check your file with the AI crawler checker.