OpenAI . tier 1
OAI-SearchBot
Builds the index behind ChatGPT search results.
260 of the 3,633 sites we measured deny OAI-SearchBot access to their root, which is 7.2% of the index.
How to allow or block it
A group naming the token wins outright over the wildcard group. It does not inherit from it, so rules set under User-agent: * do not carry over.
# Allow OAI-SearchBot everywhere
User-agent: OAI-SearchBot
Disallow:
# Block OAI-SearchBot entirely
User-agent: OAI-SearchBot
Disallow: /Sites blocking OAI-SearchBot
Most-visited first. Showing up to 100 of 260.
| Rank | Domain | Score | Answer-surface crawlers | Stack |
|---|---|---|---|---|
| 5 | facebook.com | Score 38 out of 100, grade F, partial assessment | 6 blocked | unidentified |
| 11 | instagram.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 12 | mail.ru | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 16 | twitter.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 25 | amazon.comrefused GPTBot | Score 8 out of 100, grade F | 10 blocked | Akamai |
| 52 | x.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 53 | pinterest.com | Score 20 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 56 | tiktok.com | Score 15 out of 100, grade F | 11 blocked | unidentified |
| 72 | chatgpt.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 74 | myfritz.net | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 86 | baidu.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 102 | reddit.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 154 | nytimes.comrefused GPTBot | Score 38 out of 100, grade F | 11 blocked | WordPress . Fastly |
| 177 | soundcloud.com | Score 47 out of 100, grade D, partial assessment | 8 blocked | Amazon CloudFront |
| 188 | cnn.com | Score 32 out of 100, grade F | 10 blocked | Fastly |
| 207 | bbc.com | Score 38 out of 100, grade F | 9 blocked | Next.js . Fastly |
| 227 | bbc.co.uk | Score 41 out of 100, grade D | 9 blocked | Fastly |
| 252 | imdb.com | Score 8 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 263 | meraki.com | Score 8 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 264 | launchpad.netrefused GPTBot | Score 19 out of 100, grade F | 9 blocked | unidentified |
| 292 | weibo.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 317 | amazon.co.uk | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 344 | naver.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 359 | bloomberg.comrefused GPTBot | Score 30 out of 100, grade F, partial assessment | 10 blocked | Fastly |
| 368 | vkontakte.ru | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 382 | amazon.de | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 453 | cnbc.comrefused GPTBot | Score 30 out of 100, grade F | 9 blocked | Akamai |
| 455 | sohu.com | Score 8 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 458 | claude.airefused GPTBot | Score 47 out of 100, grade D | 4 blocked | Cloudflare |
| 464 | gcdn.co | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 471 | edgecdn.ru | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 476 | quora.com | Score 14 out of 100, grade F, partial assessment | 10 blocked | unidentified |
| 481 | gosuslugi.ru | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 501 | amazon.co.jprefused GPTBot | Score 8 out of 100, grade F | 10 blocked | Akamai |
| 531 | usatoday.com | Score 27 out of 100, grade F | 11 blocked | unidentified |
| 567 | telegraph.co.uk | Score 39 out of 100, grade F | 10 blocked | Adobe Experience Manager . Fastly |
| 568 | imgsmail.ru | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 601 | threads.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 602 | dailymail.co.ukrefused GPTBot | Score 30 out of 100, grade F | 11 blocked | Akamai |
| 612 | amazon.fr | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 630 | sharethrough.com | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 641 | yelp.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 642 | amazon.ca | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 657 | cnet.com | Score 45 out of 100, grade D | 10 blocked | WordPress |
| 666 | amazon.in | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 673 | fb.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 681 | pinimg.com | Score 20 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 683 | hihonorcloud.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 684 | quantserve.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 693 | dailymail.comrefused GPTBot | Score 30 out of 100, grade F | 11 blocked | Akamai |
| 707 | amazon.co.za | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 710 | creativecdn.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 728 | amazon.it | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 734 | amazon.es | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 766 | adform.net | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 777 | blackhub.team | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 791 | amazon.com.au | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 810 | daum.net | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 816 | nbcnews.com | Score 34 out of 100, grade F | 11 blocked | Next.js . Akamai |
| 824 | lefigaro.fr | Score 58 out of 100, grade D | 7 blocked | unidentified |
| 834 | amazon.com.brrefused GPTBot | Score 8 out of 100, grade F | 10 blocked | Akamai |
| 867 | figma.com | Score 52 out of 100, grade D | 8 blocked | Sanity . Amazon CloudFront |
| 881 | hstgr.net | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 938 | mailchi.mp | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 977 | elmundo.es | Score 56 out of 100, grade D | 5 blocked | Fastly |
| 978 | btloader.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 982 | stbid.ru | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1002 | theconversation.com | Score 29 out of 100, grade F | 11 blocked | Next.js . Fastly |
| 1006 | huffpost.com | Score 49 out of 100, grade D | 11 blocked | Fastly |
| 1035 | amazon.com.mx | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 1065 | marca.com | Score 60 out of 100, grade C | 5 blocked | Fastly |
| 1079 | bfmtv.com | Score 36 out of 100, grade F | 11 blocked | Amazon CloudFront |
| 1091 | skroutz.grrefused GPTBot | Score 60 out of 100, grade C, partial assessment | 6 blocked | Cloudflare |
| 1110 | netcraze.io | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1124 | gsmarena.com | Score 45 out of 100, grade D | 6 blocked | unidentified |
| 1147 | namu.wiki | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1158 | economist.comrefused GPTBot | Score 38 out of 100, grade F, partial assessment | 10 blocked | Cloudflare |
| 1170 | rocketmoney.dev | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1178 | namecheaphosting.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1181 | xiaohongshu.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1187 | amazon.eg | Score 18 out of 100, grade F | 10 blocked | Amazon CloudFront |
| 1190 | lequipe.fr | Score 17 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1195 | nu.nlrefused GPTBot | Score 42 out of 100, grade D, partial assessment | 8 blocked | Akamai |
| 1202 | ad.nlrefused GPTBot | Score 42 out of 100, grade D, partial assessment | 8 blocked | Akamai |
| 1223 | pimpbunny.comrefused GPTBot | Score 34 out of 100, grade F | 8 blocked | Cloudflare |
| 1228 | furaffinity.netrefused GPTBot | Score 7 out of 100, grade F, partial assessment | 11 blocked | Cloudflare |
| 1242 | huffingtonpost.com | Score 49 out of 100, grade D | 11 blocked | Fastly |
| 1245 | kakao.com | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1265 | hln.berefused GPTBot | Score 42 out of 100, grade D, partial assessment | 8 blocked | Akamai |
| 1274 | franceinfo.fr | Score 35 out of 100, grade F | 11 blocked | unidentified |
| 1279 | dw.com | Score 25 out of 100, grade F | 9 blocked | Akamai |
| 1306 | actu.fr | Score 46 out of 100, grade D | 8 blocked | Cloudflare |
| 1314 | f95zone.torefused GPTBot | Score 27 out of 100, grade F | 9 blocked | Cloudflare |
| 1432 | iltalehti.fi | Score 29 out of 100, grade F | 11 blocked | Amazon CloudFront |
| 1441 | spiegel.derefused GPTBot | Score 41 out of 100, grade D | 6 blocked | Google Cloud / GFE |
| 1450 | aka.ms | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1472 | ekstrabladet.dk | Score 31 out of 100, grade F | 11 blocked | Fastly |
| 1473 | eenadu.netrefused GPTBot | Score 48 out of 100, grade D | 9 blocked | Amazon CloudFront |
| 1517 | exhentai.org | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
| 1667 | keirin.jp | Score 7 out of 100, grade F, partial assessment | 11 blocked | unidentified |
Using these figures
Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.
CrawlIndex by Fidget Labs BV. "Sites blocking OAI-SearchBot." https://crawlindex.org (measured 2026-08-09). Licensed CC BY 4.0.