Page Insights for AI agents: how to audit your website for AI traffic

To audit your website for AI traffic, use an ai agent tool for website traffic monitoring. It combines readiness checks, infrastructure logs, referrers, analytics, and request verification.

It shows whether AI systems can reach, cite, and influence your pages while separating training crawlers from search and user-triggered fetches in infrastructure data.

In this guide you will learn:

• How to decide which question your data must answer
• How to run an Is Agentic readiness audit
• How to find AI-agent requests in your infrastructure
• How to connect analytics with referrers and assisted conversions
• How to verify requests before reporting them
• How to turn findings into a repeatable operating loop

Key insights

• A readiness score covers whether your site can be accessed and understood; revenue evidence comes from logs and analytics.
• Server and CDN logs reveal requests that browser analytics may exclude or fail to capture.
• Training, search, user-triggered, and unknown requests need separate reporting paths.
• Bot identity requires corroboration because a user agent string alone is weak evidence.
• The audit becomes useful when each finding maps to a testable engineering task and a business question.

Start with the question your data must answer

AI traffic covers several reporting categories. A training crawler, a search crawler, and a person arriving from an AI answer create different evidence and different decisions.

Start by naming the decision your report must support. The dashboard comes later.

Discoverability, access, crawling, referral, influence, or conversion

Your first question determines which data deserves attention. A readiness scan can show whether a page is reachable, while logs show whether a crawler requested it.

These questions should stay separate in your report. A large crawl count cannot prove referral traffic, citation frequency, or conversion influence.

According to Cloudflare’s August 29, 2025 crawl analysis, training accounted for 79% of AI crawling in July 2025, search 17%, and user action 3.2%.

The same analysis found 38,000 Anthropic crawls per referred visitor in July 2025. Crawl activity can signal attention without producing a visit.

Define the evidence threshold before opening the dashboard

Write the evidence threshold beside each question. This stops a plausible chart from becoming a reportable finding.

For discoverability, require a page-level crawl result and a reproducible URL.

Access needs the response behavior recorded alongside the rule or rendering condition that shaped it.

For referral, require a source signal connected to a session.

A practical threshold looks like this:

1. State the question in one sentence.
2. Name the primary evidence source.
3. Add one corroborating signal.
4. Record what would disprove the finding.
5. Assign the next engineering or analytics action.

For example, “AI traffic reached the pricing page” needs more than a crawler user agent. It needs a request record, a referral or session connection, and an event after arrival.

Cloudflare’s July 1, 2025 analysis measured a 70,900:1 crawl-to-refer ratio for Anthropic and 0.1:1 for Mistral in its provider comparison. The Cloudflare crawl-to-referral analysis shows why provider behavior needs separate baselines.

Report crawling as crawling until a referral or session record supports the traffic claim.

Run an Is Agentic readiness audit

A readiness audit answers a narrower question: can your public site be accessed and interpreted by agentic systems?

It belongs at the start of the investigation because it reveals site conditions before you interpret traffic. It does not replace infrastructure or conversion analysis.

Record score, timestamp, checks, and raw evidence

The Is Agentic methodology describes its output as an estimate of public-site readiness; traffic and conversion sit outside its scope. Is Agentic’s methodology also notes that automated checks can produce false positives and false negatives.

Record the score with its timestamp and the exact checks behind it. A score without raw evidence cannot explain what changed or whether the result applies to the pages that matter.

Capture:

• The audited URL and page type
• The score and scan timestamp
• Each passed, failed, or unavailable check
• The response evidence behind every result
• The page version or release identifier
• Any freshness or caching note shown by the report

The methodology describes essential checks using an 80-point pool, recommended checks using a 20-point pool, and an emerging bonus capped at 5 points. Reports may also use a six-hour freshness cache, so a displayed score may not reflect the latest deployment.

Treat the scan as a diagnosis of public access conditions. For code-level causes, a tech audit can trace failed checks into architecture, rendering, and deployment decisions.

Turn failed checks into testable engineering tasks

A failed check needs a root cause before it becomes a backlog item. Convert it into a task with a root cause, a test, and a clear pass condition.

Use this format:

1. Describe the failed behavior at a specific URL.
2. Identify the rule, component, or rendering decision causing it.
3. State the change your team will make.
4. Define the request or page test that proves the change.
5. Attach the business area affected by the page.

“Improve agent readiness” is too vague for a sprint. “Return the product description in the server-rendered HTML for these product routes” gives an engineer something to test.

The same finding may require different work across rendering strategies. A blocked page needs access changes, while an empty client-rendered shell needs output changes.

Keep those causes separate, or the ticket will attract the wrong fix.

A failed audit does not tell you which work to prioritize. Your page inventory and business data do that.

Find AI-agent requests in your infrastructure

Browser analytics sees activity after a page loads in a browser context. AI crawlers often request documents directly, so CDN and server logs provide the first reliable view of machine access.

Build the request dataset before drawing conclusions from any analytics tool.

Filter CDN and server logs by user agent and source

A crawler that receives repeated redirects is a different finding from one receiving complete HTML with a successful response.

Start with raw request records before you trust a dashboard category. Filter on user agent strings, then inspect the surrounding request attributes.

• Request timestamp and timezone
• Host, path, query string, and HTTP method
• Response status and content type
• User agent and source IP
• Referrer when present
• Cache status, response size, and latency
• Region, autonomous system, or provider metadata when available

Group requests by agent, page type, and response outcome.

Treat user agents as discovery filters; they carry no proof of identity. Preserve the raw value, because normalization can remove details needed during verification.

Several tools to monitor AI agent traffic can summarize these records, but your reporting model should retain request-level evidence. Aggregated totals hide which pages were fetched and what the server returned.

Separate training, search, user-triggered, and unknown traffic

Intent changes the meaning of a request. Training activity may support broad content discovery, while search activity may relate to answer generation.

User-triggered fetches sit closer to an individual interaction.

As of August 31, 2026, OpenAI’s crawler documentation distinguishes GPTBot for training, OAI-SearchBot for ChatGPT Search, and ChatGPT-User for certain user actions.

Use separate categories in your log model:

Do not assign unknown requests to a named provider because the user agent looks familiar. Keep the category visible until DNS, provider documentation, or additional infrastructure evidence supports a stronger classification.

Cloudflare’s August 2025 data gives the categories practical context, though its proportions describe one measured population.

Can analytics confirm AI-agent traffic?

Analytics can confirm some outcomes after an AI-influenced visitor reaches your site. It cannot provide a complete inventory of machine requests.

That distinction matters when leadership asks for a single AI traffic number. The answer may combine incompatible evidence.

What browser analytics can miss or exclude

As of August 31, 2026, Google Analytics documentation says known bots and spiders are automatically excluded, without showing how much traffic was removed.

A crawler request may never execute browser JavaScript. It may also stop before an analytics event fires, receive a cached response, or arrive without a recognizable referrer.

Keep two datasets:

• Request-level machine activity from CDN and server logs
• Browser-level visitor activity from analytics and product events

Logs show access, analytics shows what happened after a page became an interactive visit, and neither replaces the other.

A missing analytics session only means the request never became visible through that analytics path.

How to connect referrers, sessions, events, and assisted conversions

Create a joinable record for each page visit or request. Preserve the original source value even when it maps to no named channel.

A useful sequence is:

1. Store the request or landing-page timestamp.
2. Capture the referrer and landing URL when available.
3. Join the visit to a session identifier.
4. Attach product events and funnel milestones.
5. Record direct and assisted conversion paths separately.
6. Mark missing links as unknown instead of inferred.

Choose the attribution window before comparing periods. Otherwise, a late conversion can move between reports and change the apparent contribution of AI referrals.

For an AI referral, the strongest record connects a source, a landing page, a session, and a downstream event. If only the source exists, report referral evidence without claiming conversion influence.

Use page-level reporting. A homepage referral may indicate brand discovery, while a documentation referral may support product research or implementation.

Verify before you report

An AI-agent traffic detection report needs confidence labels. A user agent string is easy to copy, and a familiar hostname does not automatically prove that the request belongs to the claimed provider.

Run verification after you filter the logs and before anything reaches a leadership deck, because that is where an unchecked user agent turns into a reported number.

Reverse and forward DNS validation

As of August 31, 2026, Google’s crawler verification workflow uses reverse DNS followed by forward DNS confirmation.

Apply that sequence to requests claiming a provider identity:

1. Resolve the source IP to a hostname.
2. Check that the hostname belongs to the claimed provider.
3. Resolve the hostname back to an IP address.
4. Confirm that the original IP appears in the returned addresses.
5. Record the lookup result beside the request sample.

Keep the raw IP, hostname, timestamp, and user agent together.

A failed check lowers confidence and moves the request into the unknown category until you have stronger evidence.

Why experimental bot authentication does not replace current checks

Experimental authentication can add a signal, but it should not replace request validation already supported by provider documentation.

Keep multiple indicators visible in the report:

• Claimed user agent
• Source IP and DNS result
• Request pattern across pages
• Response behavior and retry pattern
• Provider documentation supporting the identity

If a new authentication method is still experimental, label it accordingly. Treat a proposed header or token as one signal among several, and only for providers that document it.

Your report should distinguish verified, probable, and unknown traffic. That language gives leadership a usable view without turning uncertain requests into false precision. DNS results can change, so keep the verification inside the evidence record itself.

Turn findings into a repeatable operating loop

An audit becomes useful when it runs alongside releases, content changes, and measurement reviews. A single scan gives you a snapshot.

A repeatable loop shows whether the site is changing in the direction your business needs.

The operating model should be small enough to run regularly and precise enough to survive a challenge from engineering or finance.

Prioritize fixes by business impact

Start with pages that matter to acquisition, activation, support cost, or revenue. Then connect the AI finding to the page’s role.

A page with no AI requests may need no work. A high-revenue page with failed access checks deserves attention before a low-value page with more crawl volume.

Rank findings by page importance, evidence confidence, and the cost of leaving the root cause in place.

Re-scan after releases and compare evidence over time

Keep a run ledger for every audit cycle. Store the release identifier, scan timestamp, URL set, failed checks, raw evidence, request totals, verified categories, and analytics results.

Compare the same pages across matching time windows. A new score may reflect a changed page, a changed crawler, or a report refreshed from cached results.

The Is Agentic methodology says reports may use a six-hour freshness cache. Record that condition before comparing two scans, especially after a deployment.

A repeatable cycle looks like this:

1. Select pages tied to a business journey.
2. Run the readiness scan and save raw evidence.
3. Review CDN and server requests for the same period.
4. Verify provider claims and separate unknown traffic.
5. Join referrals with sessions and downstream events.
6. Create engineering tasks from confirmed failures.
7. Re-scan after release and compare the evidence.

Keep the findings even when nothing changed. Stable evidence prevents your team from reopening the same debate every quarter.

Where to start with AI-agent traffic auditing

An audit is finished when every reported number names its request record, its verification result, and the session it connects to. The decision is straightforward: fix the observability gap first, then prioritize page and infrastructure work using verified findings.

Talk to Blazity about AI Agents Development when your team needs help turning agent traffic evidence into production engineering work.

FAQ on ai agent tool for website traffic monitoring

Can a readiness score prove that AI traffic reaches my website?

No, a readiness score shows site conditions rather than traffic or conversion activity. Use CDN and server logs to confirm requests, then use analytics to study visitor outcomes.

Which AI requests deserve separate reporting?

Split reporting by training, search, user-triggered, and unknown before anyone reads a total. Their purposes differ, so combining them hides whether activity represents content collection, retrieval, referral, or an unverified source.

Why might Google Analytics show fewer AI visits than my server logs?

Google Analytics can exclude known bots and spiders, while many machine requests never execute browser analytics. Compare browser sessions with request-level logs instead of treating either source as complete.

How should I handle a suspicious crawler identity?

Keep the request in an unknown category until you verify its identity. Use reverse and forward DNS checks, preserve the raw request details, and record the confidence level beside the finding.

Sources

• The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (August 29, 2025)
• The crawl before the fall of referrals: understanding AI’s impact on content providers (July 1, 2025)
• Methodology, Is Agentic
• Overview of OpenAI Crawlers
• Verify requests from Google crawlers and fetchers
• Known bot-traffic exclusion