How do I connect Crawl Insights to my website?

Crawl Insights shows which AI bots visit your website. It reads your server or CDN logs, so you need to connect a data source first.

To connect a data source:

1. Navigate to Crawl Insights in the side panel.
2. Choose your provider: AWS CloudFront, Google Cloud CDN, Cloudflare, Vercel, WordPress, Akamai, a generic webhook or a file upload.
3. Follow the setup steps for that provider in Connecting your data.

Only requests from known AI crawlers and agents are stored. All other traffic is discarded.

You can manage or disconnect your data source at any time from Settings.