Kagi . tier 2

kagi-fetcher

Fetches pages for Kagi assistant features.

167 of the 3,633 sites we measured deny kagi-fetcher access to their root, which is 4.6% of the index.

How to allow or block it

A group naming the token wins outright over the wildcard group. It does not inherit from it, so rules set under User-agent: * do not carry over.

# Allow kagi-fetcher everywhere
User-agent: kagi-fetcher
Disallow:

# Block kagi-fetcher entirely
User-agent: kagi-fetcher
Disallow: /

Sites blocking kagi-fetcher

Most-visited first. Showing up to 100 of 167.

Domains that block kagi-fetcher
Rank	Domain	Score	Answer-surface crawlers	Stack
5	facebook.com	Score 38 out of 100, grade F, partial assessment	6 blocked	unidentified
11	instagram.com	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
12	mail.ru	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
16	twitter.com	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
18	linkedin.com	Score 18 out of 100, grade F, partial assessment	9 blocked	unidentified
40	netflix.com	Score 65 out of 100, grade C, partial assessment	3 blocked	unidentified
52	x.com	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
53	pinterest.com	Score 20 out of 100, grade F, partial assessment	11 blocked	unidentified
72	chatgpt.com	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
74	myfritz.net	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
86	baidu.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
102	reddit.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
161	netflix.net	Score 65 out of 100, grade C, partial assessment	3 blocked	unidentified
176	flickr.com	Score 47 out of 100, grade D, partial assessment	6 blocked	unidentified
252	imdb.com	Score 8 out of 100, grade F, partial assessment	11 blocked	unidentified
258	linktr.ee	Score 84 out of 100, grade B, partial assessment	1 blocked	unidentified
263	meraki.com	Score 8 out of 100, grade F, partial assessment	11 blocked	unidentified
277	reuters.com	Score 29 out of 100, grade F, partial assessment	9 blocked	unidentified
292	weibo.com	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
344	naver.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
347	trustpilot.com	Score 65 out of 100, grade C, partial assessment	3 blocked	unidentified
348	wsj.com	Score 35 out of 100, grade F, partial assessment	8 blocked	unidentified
368	vkontakte.ru	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
430	nflxvideo.net	Score 65 out of 100, grade C, partial assessment	3 blocked	unidentified
455	sohu.com	Score 8 out of 100, grade F, partial assessment	11 blocked	unidentified
464	gcdn.co	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
471	edgecdn.ru	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
476	quora.com	Score 14 out of 100, grade F, partial assessment	10 blocked	unidentified
481	gosuslugi.ru	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
568	imgsmail.ru	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
601	threads.com	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
630	sharethrough.com	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
641	yelp.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
657	cnet.com	Score 45 out of 100, grade D	10 blocked	WordPress
673	fb.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
679	lemonde.fr	Score 53 out of 100, grade D	5 blocked	Fastly
681	pinimg.com	Score 20 out of 100, grade F, partial assessment	11 blocked	unidentified
683	hihonorcloud.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
684	quantserve.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
710	creativecdn.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
766	adform.net	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
777	blackhub.team	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
800	disneyplus.com	Score 84 out of 100, grade B, partial assessment	1 blocked	unidentified
810	daum.net	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
815	academia.edu	Score 53 out of 100, grade D, partial assessment	6 blocked	unidentified
881	hstgr.net	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
938	mailchi.mp	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
978	btloader.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
982	stbid.ru	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
1002	theconversation.com	Score 29 out of 100, grade F	11 blocked	Next.js . Fastly
1050	ouest-france.fr	Score 84 out of 100, grade B, partial assessment	1 blocked	unidentified
1110	netcraze.io	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
1147	namu.wiki	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
1170	rocketmoney.dev	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
1178	namecheaphosting.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
1181	xiaohongshu.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
1190	lequipe.fr	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
1228	furaffinity.netrefused GPTBot	Score 7 out of 100, grade F, partial assessment	11 blocked	Cloudflare
1245	kakao.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
1269	merkur.de	Score 53 out of 100, grade D, partial assessment	5 blocked	unidentified
1292	fmkorea.com	Score 36 out of 100, grade F, partial assessment	6 blocked	unidentified
1318	amap.com	Score 60 out of 100, grade C, partial assessment	5 blocked	unidentified
1450	aka.ms	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
1517	exhentai.org	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
1667	keirin.jp	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
1685	avito.st	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
1738	xxxvideosind.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
1784	threads.net	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
1807	attn.tv	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
1813	secureserver.net	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
1817	zdnet.comrefused GPTBot	Score 34 out of 100, grade F	10 blocked	unidentified
1822	qcloud.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
1832	marketwatch.com	Score 35 out of 100, grade F, partial assessment	8 blocked	unidentified
1925	mega.nz	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
1946	adrta.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
1961	vtb.ru	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
1978	hulu.com	Score 84 out of 100, grade B, partial assessment	1 blocked	unidentified
2081	sogou.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
2093	mashable.comrefused GPTBot	Score 38 out of 100, grade F	10 blocked	Cloudflare
2111	clevertap-prod.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
2119	salesforceliveagent.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
2132	thesun.co.uk	Score 35 out of 100, grade F, partial assessment	8 blocked	unidentified
2137	juicyscore.com	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
2302	ad.gt	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
2360	openssl.org	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
2426	smi2.ru	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
2428	pcmag.comrefused GPTBot	Score 41 out of 100, grade D	10 blocked	Cloudflare
2433	sureserver.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
2458	redditmedia.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
2533	tappx.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
2536	nextdoor.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
2542	news.com.au	Score 17 out of 100, grade F, partial assessment	11 blocked	unidentified
2560	urldefense.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
2571	weibo.cn	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
2645	pxf.io	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
2698	kuaishou.com	Score 47 out of 100, grade D, partial assessment	6 blocked	unidentified
2700	ksapisrv.com	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
2734	kahoot.it	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
2756	huntress.io	Score 7 out of 100, grade F, partial assessment	11 blocked	unidentified
2795	instacart.com	Score 18 out of 100, grade F, partial assessment	9 blocked	unidentified

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "Sites blocking kagi-fetcher." https://crawlindex.org (measured 2026-08-09). Licensed CC BY 4.0.