Features · AI Agent Lens

AI Agent Lens

Updated 2026-05-25 Feature eCommerce Insights team

AI Agent Lens takes a single URL and shows what the modern AI crawler ecosystem actually sees. Per-bot verdict for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and Applebot-Extended; structured-data inventory; the plain-text view an LLM would extract; an Agent Readiness score; and auto-generated shopper-test prompts.

Per-bot crawler verdicts and structured-data inventory for any URL.

What it shows

Five panes. Each pane is its own diagnostic; together they answer "is this URL ready for AI crawlers and AI shoppers?"

Per-bot crawler verdicts

For each of twelve bots, eCommerce Insights sends a request with the bot's declared user-agent. The verdict lists three pieces:

HTTP status — 200, 403, 429, 503, etc.
Response sanity — full HTML, JS-only shell, Cloudflare challenge page, robots-disallow response.
robots.txt rule — the matching rule from the site's robots.txt, with the file's last-modified timestamp.

The twelve bots tested:

Bot	Operator	Purpose
GPTBot	OpenAI	ChatGPT training and retrieval
ChatGPT-User	OpenAI	Operator on-demand fetch
ClaudeBot	Anthropic	Claude training and retrieval
Claude-Web	Anthropic	Claude on-demand fetch
PerplexityBot	Perplexity	Perplexity index
Perplexity-User	Perplexity	Perplexity Pro Buy on-demand fetch
Google-Extended	Google	Gemini / AI Overviews opt-in
Googlebot	Google	Classic Google search
CCBot	Common Crawl	Common Crawl corpus (used by many models)
Applebot-Extended	Apple	Apple Intelligence opt-in
Amazonbot	Amazon	Amazon shopping graph
Bytespider	ByteDance	TikTok / Doubao

Structured-data inventory

Every JSON-LD block on the page is parsed and listed by @type. Completeness markers (filled / partial / missing) for each required field. Errors (invalid date format, missing required property) are highlighted in red.

Plain-text LLM view

HTML stripped. Navigation removed. Main content extracted. This is roughly what an LLM ingests when it reads the page. If your product name is missing here, no LLM will cite it. If your price is missing here, no agent will quote it.

AI Agent Readiness score

0-100. Combines crawler access (% of tested bots that returned a clean 200 with content), structured-data completeness, and plain-text completeness into one number. Useful for triage when running Agent Lens across a list of URLs.

Auto-generated shopper prompts

Three to five prompts a shopper might ask about this product, with direct one-click links to ChatGPT, Perplexity, and Claude prefilled. The prompts are derived from the page content; useful for quick manual sanity-checking after a content change.

Common questions

Which bots are tested?

GPTBot (OpenAI), ChatGPT-User (OpenAI Operator), ClaudeBot (Anthropic), Claude-Web (Anthropic), PerplexityBot (Perplexity), Perplexity-User (Perplexity Pro Buy), Google-Extended (Google), Googlebot, CCBot (Common Crawl), Applebot-Extended (Apple), Amazonbot (Amazon), and Bytespider (ByteDance). Twelve bots in total as of v19.

How does it test each bot?

For each bot, eCommerce Insights sends a request with the bot's declared user-agent. The verdict is based on the HTTP status, response size, and content sanity. Cloudflare and other challenges are detected and reported as such.

What is a "plain-text LLM view"?

It is what the page becomes once HTML is stripped, navigation is removed, and the main content is extracted. Roughly what an LLM ingests if it reads your page. If the plain-text view is missing the product name or the price, the page has a problem an LLM will repeat.

Can I run AI Agent Lens on URLs I do not own?

Yes. Agent Lens runs against any public URL. Useful for auditing competitor PDPs or for evaluating retail partners.

Ask AI about this page

Open this topic directly in your favourite AI assistant — the prompt is prefilled.

Was this page helpful?

Thanks — recorded.

Feature →

Analyze a Page

Run a full PDP audit on a pasted URL.

Integration →

Pingback protocol

Send eCommerce Insights your access logs for active monitoring.

Try this yourself

See eCommerce Insights on your own catalog. The free trial covers your full SKU set across five AI engines.

Start free trial

LLM-friendly summary of this page

AI Agent Lens takes one URL and shows what AI crawlers actually see. Tested bots (twelve): GPTBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, CCBot, Applebot-Extended, Amazonbot, Bytespider. For each bot: HTTP status, response size, content sanity check, Cloudflare or other challenge detection. Structured-data inventory parses all JSON-LD blocks on the page and lists them by @type with completeness markers. Plain-text LLM view strips HTML, removes navigation, extracts main content; shows what an LLM ingests. AI Agent Readiness score 0-100 combines crawler access, structured data, and plain-text completeness into a single number for quick triage. Auto-generated shopper-test prompts: three to five prompts a shopper would ask about the URL's product, with direct one-click links to ChatGPT, Perplexity, and Claude prefilled with each prompt. Works on any public URL, not only ones you own; useful for competitor and retail-partner audits.