Measure AEO with Cloudflare Workers

TL;DR

Run a machine-friendly “shadow” version of your site at the edge with Cloudflare Workers. Only AI crawlers see it; humans get your normal site. Log each bot visit so you can measure:

• Which pages models actually read
• How quickly they re-crawl after updates
• When and where citations appear (explicit or inferred)

This follows our Salespeak “bot-only, zero-CMS-changes” approach.

Why Cloudflare Workers

Workers run at the CDN edge, which means you can:

• Target bots only without touching templates or risking human UX
• Instrument every request with structured events (page, bot label, status, latency, cache state)
• Observe and export logs to your analytics stack for dashboards and experiments
• Adapt quickly as crawlers change behavior—update rules at the edge, not in your CMS

The Setup (no CMS changes)

1. Put Cloudflare in front of your origin.
2. Identify AI crawlers. Start with declared bots (e.g., GPTBot, PerplexityBot, Anthropic) and add simple behavioral checks over time.
3. Serve a “shadow” version of each page to bots only. Humans always see the standard site.
4. Log every bot request with page path, bot label, status, timing, cache state, updated-at, and optional “ref” hints.
5. Dashboard your KPIs to track usage, freshness pickup, citations, and errors by model and by page.

What Goes in the Shadow Version

• Canonical URL and a stable page ID
• Last updated timestamp (so you can measure re-crawl speed)
• Answer-first summary (50–120 words) that states the core facts
• FAQs and key facts in concise lists or small tables
• Internal links to related pages for better crawl paths
• Optional ref parameters to help correlate bot visits with downstream citations

AEO Metrics You’ll Unlock

• Coverage: % of priority pages fetched by at least one model (last 30 days)
• Depth: distinct pages per model (which models explore the widest surface)
• Freshness pickup: time from page update → first bot re-fetch (by model)
• Citations: explicit (when platforms expose them) and inferred (via bot-visit patterns and ref hints)
• Reliability: bot-specific 4xx/5xx and a never-fetched list (your opportunity backlog)

Rollout in Two Weeks

Week 1 — Blueprint & Staging

• Pick your top 25–50 pages.
• Define shadow fields (canonical, updated-at, summary, FAQs, key facts).
• Configure bot identification and edge logging in a staging environment.

Week 2 — Launch & Measure

• Go live with the first set of pages.
• Validate that human traffic is untouched and bot traffic is captured.
• Build dashboards for coverage, freshness pickup, top pages by model, errors, and never-fetched pages.
• Run one micro-experiment (e.g., add 3–5 FAQs to a subset) and compare week-over-week deltas.

Operating Guidelines

• Layer signals: Use declared bots plus basic behavior/network signals; review weekly for drift.
• Keep shadow content factual and compact: Answer-first, tight lists, clear definitions.
• Version & freshness: Keep “last updated” correct so you can measure re-crawl speed.
• Governance: Only expose content in the shadow version that you’re comfortable being quoted.

FAQ

Does this affect SEO or human UX?
No. Humans see the normal site. The shadow version is bot-only. Keep canonicals consistent.

What if a platform doesn’t expose citations?
Use explicit citations when available; otherwise infer from edge telemetry (visit patterns, ref hints, and timing).

Why not rely on standard web analytics?
You need per-request, high-cardinality data to segment by page × bot × time. Edge logging makes that practical.

Where Salespeak Fits

Salespeak’s LLM Optimizer implements this bot-only delivery + observability pattern on Cloudflare. We’ll help you ship a two-week baseline—coverage, freshness, citations, and errors—without changing your CMS.

‍