Pingback protocol

The Pingback protocol is eCommerce Insights's open spec for receiving AI-crawler access logs from your edge. Stream GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and Applebot-Extended hits into eCommerce Insights and the crawler-access data shows up in Agent Lens and PDP Score in near real time.

Why send pingbacks

eCommerce Insights's Agent Lens runs scheduled checks against the bots-vs-your-URL surface. Pingback gives eCommerce Insights the live, by-request truth: which crawler hit which URL when, and what response it got. This is useful when crawler-access regressions happen between scheduled checks.

Endpoint

POST https://app.ecommerceinsights.ai/pingback/v1
Content-Type: application/x-ndjson
X-eCommerce Insights-Signature: sha256=<hex digest>
X-eCommerce Insights-Workspace: <workspace id>

Payload schema

One JSON object per line. Required fields:

ts

ISO 8601 timestamp.

url

Path of the request (no host).

method

HTTP method.

status

HTTP status code.

user_agent

The full user-agent string. Must match one of the twelve recognised AI crawler patterns.

Optional fields:

request_id

Your own request ID for cross-referencing.

ip_country

ISO 3166 country code from the source IP.

ip_asn

ASN of the source IP.

response_bytes

Size of the response body.

response_time_ms

Time spent serving the request, in milliseconds.

Signing

Compute an HMAC-SHA256 over the raw request body using your workspace secret (from Settings → Integrations → Pingback). Send the hex digest in the X-eCommerce Insights-Signature header prefixed with sha256=.

import hmac, hashlib
secret = b"YOUR_WORKSPACE_SECRET"
body = open("events.ndjson", "rb").read()
sig = hmac.new(secret, body, hashlib.sha256).hexdigest()
# Send header: X-eCommerce Insights-Signature: sha256=<sig>

Batching

For workspaces with high crawler volume, batch up to 10,000 events per request. The endpoint accepts up to 5 MB body size. Above 10,000 events/minute, split across multiple requests.

Example with curl

cat events.ndjson | curl -X POST \
  -H "Content-Type: application/x-ndjson" \
  -H "X-eCommerce Insights-Workspace: ws_abc123" \
  -H "X-eCommerce Insights-Signature: sha256=$(openssl dgst -sha256 -hmac \"$SECRET\" events.ndjson | awk '{print $2}')" \
  --data-binary @events.ndjson \
  https://app.ecommerceinsights.ai/pingback/v1

Edge platform examples

• Cloudflare Worker — template available in the repo. Filters by user-agent and batches every 30 seconds.
• Vercel Edge Middleware — template available. Runs at the edge before your origin.
• Fastly VCL — template available. Logs to a log endpoint that batches to eCommerce Insights.
• AWS Lambda@Edge — template available. Useful for CloudFront-backed sites.

Common questions