Skip to content

docs: Update bot detection and ClickHouse functions for IP-based detection #18195

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
inkeep wants to merge 2 commits into
base: master
Choose a base branch
from
Open

docs: Update bot detection and ClickHouse functions for IP-based detection #18195

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
89d927b
docs: update bot detection docs for IP-based classification
inkeep[bot] Jul 6, 2026
2e5e3ad
docs: add IP address functions to supported ClickHouse functions list
inkeep[bot] Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions contents/docs/sql/clickhouse-functions.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -525,6 +525,32 @@ If the function's name includes a leading underscore, then it's not safe and sim
- `cutQueryStringAndFragment`
- `cutURLParameter`

## IP addresses

- `IPv4NumToString`
- `IPv4StringToNum`
- `IPv4StringToNumOrDefault`
- `IPv4StringToNumOrNull`
- `IPv6NumToString`
- `IPv6StringToNum`
- `IPv6StringToNumOrDefault`
- `IPv6StringToNumOrNull`
- `IPv4ToIPv6`
- `toIPv4`
- `toIPv4OrDefault`
- `toIPv4OrNull`
- `toIPv4OrZero`
- `toIPv6`
- `toIPv6OrDefault`
- `toIPv6OrNull`
- `toIPv6OrZero`
- `isIPv4String`
- `isIPv6String`
- `isIPAddressInRange`
- `IPv4CIDRToRange`
- `IPv6CIDRToRange`
- `cutIPv6`

## JSON

- `isValidJSON`
Expand Down
101 changes: 52 additions & 49 deletions contents/docs/web-analytics/bot-detection.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Bot and traffic detection
---

PostHog classifies traffic by user agent so you can tell humans apart from bots, crawlers, and automation directly in your queries. It categorizes each request – so you can welcome, measure, or exclude AI agents, search crawlers, and automation independently, rather than treating every bot the same. The classification runs in [SQL](/docs/sql), so it works anywhere HogQL does – the SQL editor, insights, trends, and Web Analytics breakdowns.
PostHog classifies traffic by user agent and IP address so you can tell humans apart from bots, crawlers, and automation directly in your queries. It categorizes each request – so you can welcome, measure, or exclude AI agents, search crawlers, and automation independently, rather than treating every bot the same. The classification runs in [SQL](/docs/sql), so it works anywhere HogQL does – the SQL editor, insights, trends, and Web Analytics breakdowns.

<CalloutBox icon="IconInfo" title="New feature" type="fyi">

Expand All @@ -24,16 +24,16 @@ Set `$raw_user_agent` on each event, plus `$host`, `$current_url`, `$pathname`,

```json
{
"api_key": "<ph_project_token>",
"event": "$http_log",
"distinct_id": "server-log",
"properties": {
"$raw_user_agent": "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)",
"$current_url": "https://yoursite.com/blog/hello-world",
"$pathname": "/blog/hello-world",
"status_code": 200,
"method": "GET"
}
"api_key": "<ph_project_token>",
"event": "$http_log",
"distinct_id": "server-log",
"properties": {
"$raw_user_agent": "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)",
"$current_url": "https://yoursite.com/blog/hello-world",
"$pathname": "/blog/hello-world",
"status_code": 200,
"method": "GET"
}
}
```

Expand All @@ -50,46 +50,46 @@ How you set these depends on how you ingest. With the [capture API](/docs/api/ca

## Classification functions

Each function takes a user agent string and returns its classification. PostHog stores the user agent as `$raw_user_agent` (server-side capture and `$http_log` events) or `$user_agent` (the JavaScript SDK), so pass `coalesce(properties.$raw_user_agent, properties.$user_agent)` to cover both. To skip the fallback, use the [virtual properties](#virtual-properties) below, which read the user agent for you.
Each function takes a user agent string and returns its classification. They also accept an optional second `ip` argument – when the user agent doesn't match a known pattern, the IP is checked against published bot IP ranges. PostHog stores the user agent as `$raw_user_agent` (server-side capture and `$http_log` events) or `$user_agent` (the JavaScript SDK), so pass `coalesce(properties.$raw_user_agent, properties.$user_agent)` to cover both. To skip the fallback, use the [virtual properties](#virtual-properties) below, which read the user agent and IP for you.

| Function | Returns | Example |
| --- | --- | --- |
| `isLikelyBot(user_agent)` | `true` if the user agent matches a bot or automation pattern, otherwise `false`. An empty user agent counts as a bot. | `true` |
| `getTrafficType(user_agent)` | One of `AI Agent`, `Bot`, `Automation`, or `Regular`. | `Bot` |
| `getTrafficCategory(user_agent)` | A subcategory such as `ai_crawler`, `ai_search`, `search_crawler`, `seo_crawler`, `social_crawler`, `monitoring`, `http_client`, or `headless_browser`. Returns `regular` for human traffic. | `search_crawler` |
| `getBotType(user_agent)` | The same subcategory as `getTrafficCategory`, but returns an empty string for human traffic – handy for filtering. | `search_crawler` |
| `getBotName(user_agent)` | The bot's name, such as `Googlebot`, `ChatGPT`, or `curl`. Empty for human traffic. | `Googlebot` |
| `getBotOperator(user_agent)` | The company or operator behind the bot, such as `Google`, `OpenAI`, or `Anthropic`. Empty for human traffic. | `Google` |
| Function | Returns | Example |
| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- |
| `isLikelyBot(user_agent [, ip])` | `true` if the user agent matches a bot or automation pattern, or the IP falls in a known bot range. An empty user agent counts as a bot. | `true` |
| `getTrafficType(user_agent [, ip])` | One of `AI Agent`, `Bot`, `Automation`, or `Regular`. | `Bot` |
| `getTrafficCategory(user_agent [, ip])` | A subcategory such as `ai_crawler`, `ai_search`, `search_crawler`, `seo_crawler`, `social_crawler`, `monitoring`, `http_client`, or `headless_browser`. Returns `regular` for human traffic. | `search_crawler` |
| `getBotType(user_agent [, ip])` | The same subcategory as `getTrafficCategory`, but returns an empty string for human traffic – handy for filtering. | `search_crawler` |
| `getBotName(user_agent [, ip])` | The bot's name, such as `Googlebot`, `ChatGPT`, or `curl`. Empty for human traffic. | `Googlebot` |
| `getBotOperator(user_agent [, ip])` | The company or operator behind the bot, such as `Google`, `OpenAI`, or `Anthropic`. Empty for human traffic. | `Google` |

`isLikelyBot` is named "likely" because detection is based on user agent heuristics – it can't confirm with certainty that a request is automated.
`isLikelyBot` is named "likely" because detection is based on heuristics – user agent pattern matching and IP range lookups – so it can't confirm with certainty that a request is automated.

## Traffic types

`getTrafficType` sorts every request into one of four traffic types:

| Traffic type | Description | Examples |
| --- | --- | --- |
| `Regular` | Normal human visitors | Chrome, Safari, Firefox |
| `AI Agent` | AI crawlers, search bots, and assistants | GPTBot, ClaudeBot, PerplexityBot |
| `Bot` | Traditional crawlers and monitoring | Googlebot, Bingbot, AhrefsBot |
| `Automation` | HTTP clients, headless browsers, and requests with no user agent | curl, Puppeteer, HeadlessChrome |
| Traffic type | Description | Examples |
| ------------ | ---------------------------------------------------------------- | -------------------------------- |
| `Regular` | Normal human visitors | Chrome, Safari, Firefox |
| `AI Agent` | AI crawlers, search bots, and assistants | GPTBot, ClaudeBot, PerplexityBot |
| `Bot` | Traditional crawlers and monitoring | Googlebot, Bingbot, AhrefsBot |
| `Automation` | HTTP clients, headless browsers, and requests with no user agent | curl, Puppeteer, HeadlessChrome |

## Bot categories

`getTrafficCategory` and `getBotType` return a finer-grained category within each traffic type:

| Category | Traffic type | Description | Examples |
| --- | --- | --- | --- |
| `ai_crawler` | AI Agent | Training data collection | GPTBot, ClaudeBot, Google-Extended |
| `ai_search` | AI Agent | AI-powered search results | OAI-SearchBot, Claude-SearchBot, Applebot |
| `ai_assistant` | AI Agent | Real-time user-facing AI | ChatGPT-User, Claude-User, Perplexity-User |
| `search_crawler` | Bot | Traditional search engines | Googlebot, Bingbot, Baidu |
| `seo_crawler` | Bot | SEO analysis tools | AhrefsBot, SemrushBot, Majestic |
| `social_crawler` | Bot | Social media preview crawlers | Facebook, Twitter, LinkedIn, Slack |
| `monitoring` | Bot | Uptime and health monitoring | Pingdom, UptimeRobot, Datadog |
| `http_client` | Automation | HTTP client libraries | curl, Wget, Python requests, axios |
| `headless_browser` | Automation | Automated browsers | HeadlessChrome, Puppeteer, Playwright |
| `no_user_agent` | Automation | Empty or missing user agent | – |
| Category | Traffic type | Description | Examples |
| ------------------ | ------------ | ----------------------------- | ------------------------------------------ |
| `ai_crawler` | AI Agent | Training data collection | GPTBot, ClaudeBot, Google-Extended |
| `ai_search` | AI Agent | AI-powered search results | OAI-SearchBot, Claude-SearchBot, Applebot |
| `ai_assistant` | AI Agent | Real-time user-facing AI | ChatGPT-User, Claude-User, Perplexity-User |
| `search_crawler` | Bot | Traditional search engines | Googlebot, Bingbot, Baidu |
| `seo_crawler` | Bot | SEO analysis tools | AhrefsBot, SemrushBot, Majestic |
| `social_crawler` | Bot | Social media preview crawlers | Facebook, Twitter, LinkedIn, Slack |
| `monitoring` | Bot | Uptime and health monitoring | Pingdom, UptimeRobot, Datadog |
| `http_client` | Automation | HTTP client libraries | curl, Wget, Python requests, axios |
| `headless_browser` | Automation | Automated browsers | HeadlessChrome, Puppeteer, Playwright |
| `no_user_agent` | Automation | Empty or missing user agent | – |

## Example queries

Expand Down Expand Up @@ -130,15 +130,15 @@ ORDER BY hits DESC

## Virtual properties

For convenience, PostHog exposes the classification as virtual event properties. They read the user agent for you (falling back from `$raw_user_agent` to `$user_agent`), so you don't have to pass it in. They're available wherever you select event properties, including breakdowns:
For convenience, PostHog exposes the classification as virtual event properties. They read the user agent (falling back from `$raw_user_agent` to `$user_agent`) and the client IP (`$ip`) for you, so you don't have to pass them in. They're available wherever you select event properties, including breakdowns:

| Property | Equivalent to |
| --- | --- |
| `$virt_is_bot` | `isLikelyBot(...)` |
| `$virt_traffic_type` | `getTrafficType(...)` |
| Property | Equivalent to |
| ------------------------ | ------------------------- |
| `$virt_is_bot` | `isLikelyBot(...)` |
| `$virt_traffic_type` | `getTrafficType(...)` |
| `$virt_traffic_category` | `getTrafficCategory(...)` |
| `$virt_bot_name` | `getBotName(...)` |
| `$virt_bot_operator` | `getBotOperator(...)` |
| `$virt_bot_name` | `getBotName(...)` |
| `$virt_bot_operator` | `getBotOperator(...)` |

For example, to break down traffic without writing out the function:

Expand Down Expand Up @@ -166,11 +166,14 @@ Because the classification reads the raw user agent, this works for any event th

## How classification works

Classification matches the user agent against a maintained list of known bot, crawler, and automation patterns. The list is open source and pull requests are welcome – you can find it in the [PostHog repository](https://github.com/PostHog/posthog/blob/master/products/web_analytics/backend/hogql_queries/bot_definitions.py).
Classification first matches the user agent against a maintained list of known bot, crawler, and automation patterns. When no user agent pattern matches, the client IP is checked against published bot IP ranges – currently [Google's verified crawler and fetcher ranges](https://developers.google.com/crawling/docs/verifying-google-crawlers). This catches bots like Google's mobile rendering service that send real browser user agents with no bot token. User agent classification takes precedence, so the IP lookup only runs for rows the UA check didn't already match.

Both lists are open source and pull requests are welcome – you can find the [user agent patterns](https://github.com/PostHog/posthog/blob/master/products/web_analytics/backend/hogql_queries/bot_definitions.py) and [IP range definitions](https://github.com/PostHog/posthog/blob/master/products/web_analytics/backend/hogql_queries/bot_ip_definitions.py) in the PostHog repository.

## Limitations

Detection relies entirely on the user agent string, so two cases are worth keeping in mind:
Detection uses user agent patterns and IP range lookups, so a few cases are worth keeping in mind:

- **No user agent** – requests with an empty or missing user agent (server-to-server calls, misconfigured SDKs) can't be classified from the user agent alone. They're treated as `Automation` with the `no_user_agent` category, so `isLikelyBot` returns `true` for them.
- **Spoofing** – some bots disguise themselves with regular browser user agents, and some legitimate tools use bot-like ones. User agent detection is a best-effort heuristic, not a guarantee, which is why the boolean function is named `isLikelyBot`.
- **Spoofing** – some bots disguise themselves with regular browser user agents, and some legitimate tools use bot-like ones. IP-based detection helps catch bots that use real browser user agents but come from known bot IP ranges (like Google's mobile rendering service). However, detection is still a best-effort heuristic, not a guarantee, which is why the boolean function is named `isLikelyBot`.
- **IP coverage** – IP range detection currently covers Google's published crawler ranges. Bots from operators that don't publish their IP ranges won't be caught by the IP check.
Loading