Firecrawl . tier 2
FirecrawlAgent
Retrieval API used by agent frameworks.
206 of the 3,633 sites we measured deny FirecrawlAgent access to their root, which is 5.7% of the index.
How to allow or block it
A group naming the token wins outright over the wildcard group. It does not inherit from it, so rules set under User-agent: * do not carry over.
# Allow FirecrawlAgent everywhere
User-agent: FirecrawlAgent
Disallow:
# Block FirecrawlAgent entirely
User-agent: FirecrawlAgent
Disallow: /Sites blocking FirecrawlAgent
Most-visited first. Showing up to 100 of 206.
| Rank | Domain | Score | Answer-surface crawlers | Stack |
|---|---|---|---|---|
| 5 | facebook.com | Score 38 out of 100, grade F, partial assessment | 6 blocked | unidentified |
| 11 | instagram.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 12 | mail.ru | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 16 | twitter.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 18 | linkedin.com | Score 18 out of 100, grade F, partial assessment | 9 blocked | unidentified |
| 40 | netflix.com | Score 65 out of 100, grade C, partial assessment | 3 blocked | unidentified |
| 52 | x.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 53 | pinterest.com | Score 20 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 72 | chatgpt.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 74 | myfritz.net | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 86 | baidu.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 102 | reddit.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 161 | netflix.net | Score 65 out of 100, grade C, partial assessment | 3 blocked | unidentified |
| 176 | flickr.com | Score 47 out of 100, grade D, partial assessment | 6 blocked | unidentified |
| 207 | bbc.com | Score 38 out of 100, grade F | 9 blocked | Next.js . Fastly |
| 227 | bbc.co.uk | Score 41 out of 100, grade D | 9 blocked | Fastly |
| 252 | imdb.com | Score 8 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 258 | linktr.ee | Score 84 out of 100, grade B, partial assessment | 1 blocked | unidentified |
| 263 | meraki.com | Score 8 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 264 | launchpad.netrefused GPTBot | Score 19 out of 100, grade F | 9 blocked | unidentified |
| 277 | reuters.com | Score 29 out of 100, grade F, partial assessment | 9 blocked | unidentified |
| 292 | weibo.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 344 | naver.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 347 | trustpilot.com | Score 65 out of 100, grade C, partial assessment | 3 blocked | unidentified |
| 348 | wsj.com | Score 35 out of 100, grade F, partial assessment | 8 blocked | unidentified |
| 368 | vkontakte.ru | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 430 | nflxvideo.net | Score 65 out of 100, grade C, partial assessment | 3 blocked | unidentified |
| 455 | sohu.com | Score 8 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 464 | gcdn.co | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 471 | edgecdn.ru | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 476 | quora.com | Score 14 out of 100, grade F, partial assessment | 10 blocked | unidentified |
| 481 | gosuslugi.ru | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 531 | usatoday.com | Score 27 out of 100, grade F | 11 blocked | unidentified |
| 568 | imgsmail.ru | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 601 | threads.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 630 | sharethrough.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 641 | yelp.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 657 | cnet.com | Score 45 out of 100, grade D | 10 blocked | WordPress |
| 673 | fb.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 679 | lemonde.fr | Score 53 out of 100, grade D | 5 blocked | Fastly |
| 681 | pinimg.com | Score 20 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 683 | hihonorcloud.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 684 | quantserve.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 710 | creativecdn.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 766 | adform.net | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 767 | bild.de | Score 63 out of 100, grade C | 3 blocked | unidentified |
| 777 | blackhub.team | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 795 | people.com | Score 41 out of 100, grade D, partial assessment | 8 blocked | Cloudflare |
| 798 | welt.de | Score 55 out of 100, grade D | 3 blocked | Varnish |
| 800 | disneyplus.com | Score 84 out of 100, grade B, partial assessment | 1 blocked | unidentified |
| 807 | theverge.com | Score 46 out of 100, grade D | 9 blocked | WordPress . Fastly |
| 810 | daum.net | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 815 | academia.edu | Score 53 out of 100, grade D, partial assessment | 6 blocked | unidentified |
| 824 | lefigaro.fr | Score 58 out of 100, grade D | 7 blocked | unidentified |
| 881 | hstgr.net | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 908 | investopedia.com | Score 41 out of 100, grade D, partial assessment | 8 blocked | Cloudflare |
| 938 | mailchi.mp | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 978 | btloader.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 982 | stbid.ru | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1002 | theconversation.com | Score 29 out of 100, grade F | 11 blocked | Next.js . Fastly |
| 1050 | ouest-france.fr | Score 84 out of 100, grade B, partial assessment | 1 blocked | unidentified |
| 1079 | bfmtv.com | Score 36 out of 100, grade F | 11 blocked | Amazon CloudFront |
| 1110 | netcraze.io | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1147 | namu.wiki | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1170 | rocketmoney.dev | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1178 | namecheaphosting.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1181 | xiaohongshu.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1190 | lequipe.fr | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1228 | furaffinity.netrefused GPTBot | Score 7 out of 100, grade F, partial assessment | 11 blocked | Cloudflare |
| 1245 | kakao.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1269 | merkur.de | Score 53 out of 100, grade D, partial assessment | 5 blocked | unidentified |
| 1292 | fmkorea.com | Score 36 out of 100, grade F, partial assessment | 6 blocked | unidentified |
| 1314 | f95zone.torefused GPTBot | Score 27 out of 100, grade F | 9 blocked | Cloudflare |
| 1318 | amap.com | Score 60 out of 100, grade C, partial assessment | 5 blocked | unidentified |
| 1450 | aka.ms | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1517 | exhentai.org | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1667 | keirin.jp | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1685 | avito.st | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1738 | xxxvideosind.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1784 | threads.net | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1807 | attn.tv | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1813 | secureserver.net | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1817 | zdnet.comrefused GPTBot | Score 34 out of 100, grade F | 10 blocked | unidentified |
| 1822 | qcloud.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1824 | wikihow.com | Score 33 out of 100, grade F | 9 blocked | unidentified |
| 1832 | marketwatch.com | Score 35 out of 100, grade F, partial assessment | 8 blocked | unidentified |
| 1917 | jamanetwork.comrefused GPTBot | Score 63 out of 100, grade C, partial assessment | 5 blocked | Cloudflare |
| 1925 | mega.nz | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1946 | adrta.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1961 | vtb.ru | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1978 | hulu.com | Score 84 out of 100, grade B, partial assessment | 1 blocked | unidentified |
| 2081 | sogou.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 2093 | mashable.comrefused GPTBot | Score 38 out of 100, grade F | 10 blocked | Cloudflare |
| 2094 | themoviedb.org | Score 27 out of 100, grade F | 11 blocked | Amazon CloudFront |
| 2098 | ign.comrefused GPTBot | Score 31 out of 100, grade F | 10 blocked | WordPress . Fastly |
| 2111 | clevertap-prod.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 2119 | salesforceliveagent.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 2132 | thesun.co.uk | Score 35 out of 100, grade F, partial assessment | 8 blocked | unidentified |
| 2137 | juicyscore.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 2244 | vox.com | Score 46 out of 100, grade D | 9 blocked | WordPress . Fastly |
Using these figures
Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.
CrawlIndex by Fidget Labs BV. "Sites blocking FirecrawlAgent." https://crawlindex.org (measured 2026-08-09). Licensed CC BY 4.0.