Platform

Astro

Name: CrawlIndex dataset
License: https://creativecommons.org/licenses/by/4.0/

21 measured sites. 3 of them, 14.3%, block at least one answer-surface AI crawler. Mean readiness score 73.3.

Sites on Astro
Rank	Domain	Score	Answer-surface crawlers	Stack
2	cloudflare.com	Score 92 out of 100, grade A	all allowed	Astro . Cloudflare
104	workers.dev	Score 72 out of 100, grade C	all allowed	Astro . Cloudflare
259	roku.com	Score 81 out of 100, grade B	all allowed	Astro . Cloudflare
617	ikea.com	Score 75 out of 100, grade B	all allowed	Astro . Cloudflare
769	bugsnag.com	Score 62 out of 100, grade C	all allowed	Astro . Cloudflare
922	kleinanzeigen.de	Score 71 out of 100, grade C	all allowed	Astro . Akamai
962	netlify.app	Score 95 out of 100, grade A	all allowed	Astro . Netlify
1057	jimdo.com	Score 87 out of 100, grade B	all allowed	Astro . Cloudflare
1184	aftonbladet.se	Score 48 out of 100, grade D	6 blocked	Astro . Amazon CloudFront
1192	vg.no	Score 56 out of 100, grade D	6 blocked	Astro . Amazon CloudFront
1353	asurascans.comrefused GPTBot	Score 76 out of 100, grade B	all allowed	Astro . Cloudflare
1549	rustdesk.comrefused GPTBot	Score 80 out of 100, grade B	all allowed	Astro . Cloudflare
2060	getbootstrap.com	Score 67 out of 100, grade C	all allowed	Astro . Cloudflare
2949	proton.me	Score 83 out of 100, grade B	all allowed	Astro
3184	adingo.jp	Score 56 out of 100, grade D	all allowed	Astro . Cloudflare
3273	wireshark.org	Score 62 out of 100, grade C	all allowed	Astro . Cloudflare
4251	netlify.com	Score 95 out of 100, grade A	all allowed	Astro . Netlify
4662	nerdwallet.com	Score 76 out of 100, grade B	1 blocked	Astro . Cloudflare
4712	bip.com	Score 61 out of 100, grade C	all allowed	Astro
4718	zomato.com	Score 53 out of 100, grade D	all allowed	Astro
5047	clara.net	Score 91 out of 100, grade A	all allowed	Astro

Using these figures

Free to reuse in research, journalism or a product under CC BY 4.0, with credit to Fidget Labs BV. Quote the measurement date so the claim stays checkable as the index moves.

CrawlIndex by Fidget Labs BV. "Agent readiness on Astro." https://crawlindex.org (measured 2026-08-09). Licensed CC BY 4.0.