How AI Bot Tracking Works
Understand how Hardal measures AI crawler requests on the server and why those visits differ from browser analytics.
AI bot tracking measures requests made by automated systems to your website. Hardal's AI Visibility report uses request metadata collected at your server or CDN to show identified agents, their platforms, and the pages they reach.
Why browser analytics can miss crawler activity
A website tracker runs after the page loads in a browser. An automated client can request a document without running that script. The server can observe the request even when no browser page-view event follows.
This makes the collection point important. For AI Visibility, forward metadata from the layer receiving the original request rather than trying to reconstruct it from a later browser visit. See AI Visibility Setup for supported integration examples.
Follow one request
Receive the page request
Your server or CDN receives a request for a path such as /pricing.
Forward the original metadata
Send the original path, IP address, user agent, request time, and Source ID to the agent-events endpoint. Use the source's generated collection URL and Tenant ID.
Review identified activity
Hardal classifies the submitted request for AI Visibility. Open Agents for traffic context or Pages for the recorded path.
The endpoint filters requests that do not qualify for storage. Changing a browser's user agent alone is not a reliable verification method; preserve the actual request context. The agent-events reference explains response handling.
Separate three measurements
| Measurement | What it answers |
|---|---|
| Agent request | Did an identified automated system request this path? |
| Browser visit | Did the installed web tracker record page activity? |
| AI referral | Was a website visit attributed to an AI-related source? |
These counts are not interchangeable. One agent request does not establish that a person saw your page in an answer. Use AI referral analysis for attributed website activity and AI page coverage for important content paths.