What CrawlIndex is

CrawlIndex is a public measurement project. It reads publicly served files from the most-visited sites on the web and publishes what it finds about how those sites treat AI crawlers and agents.

Who runs it

Research and data by Fidget Labs BV, an architecture and AI consultancy in Breda, Netherlands. The index is operated as a standalone public resource, not as a marketing surface for client work. Nobody can pay to change a score, and there is nothing here to buy.

How it is funded, and why that matters

It costs essentially nothing to run, which is the point. The crawler is a scheduled job on free CI minutes, the dataset is files in a public git repository rather than a hosted database, and the site is static. There is no server to keep alive and no bill that grows with traffic.

That is deliberate. A free public resource that depends on someone continuing to pay for it is a resource with an expiry date. This one can sit here indefinitely without anyone deciding to keep funding it.

What data it holds

Only what a web server hands to any anonymous visitor: robots.txt, a homepage, and two well-known paths. No personal data is collected, no accounts exist, no cookies are set, and there is no advertising or third-party tracking on this site.

Being removed from the index

Self-service, and it needs no request. Disallow CrawlIndexBot in your robots.txt and the domain drops out on the next crawl. The crawler checks that rule before it requests anything else, so opting out costs your server a single request:

User-agent: CrawlIndexBot
Disallow: /

Note that a blanket User-agent: * / Disallow: / is treated differently. We honour it by fetching no pages, but robots.txt is public and the access policy it states is still reported, because dropping the most restrictive operators from an index about restrictiveness would quietly bias every figure on the site.

Corrections

Measurement projects get things wrong. If a result looks incorrect, the methodology explains how it was produced and the underlying observation is downloadable, so you can usually see the cause yourself.

Corrections can be raised as an issue on the repository. To be straight with you about the service level: this project is deliberately unstaffed. Issues are not monitored on any schedule and there is no support address. The robots.txt opt-out above is the mechanism that is guaranteed to work, immediately and without anyone reading anything.

Reuse

The dataset is licensed CC BY 4.0. Use it in research, journalism or a commercial product. The only condition is credit to Fidget Labs BV. The data page has the files and a ready-made citation.