Token Counter

A lightweight Cloudflare Worker that counts tokens in text using the GPT-4o encoding (cl200k_base). Includes a web UI and a REST API.

Features

• Real-time token counting — counts update as you type (debounced)
• REST API — POST /api/tokenize for programmatic access
• Character & word counts — displayed alongside token count
• Edge-deployed — runs on Cloudflare's global network with minimal latency

Getting Started

Prerequisites

• Node.js (v20+)
• A Cloudflare account (Workers Paid plan for deployment)

Local Development

npm install
npm run dev

Open http://localhost:8787 in your browser.

Deploy

npx wrangler login   # authenticate with Cloudflare (first time only)
npm run deploy

Your worker will be live at https://tokenizer.<your-subdomain>.workers.dev.

API

POST /api/tokenize

Request:

{ "text": "Hello, how many tokens is this?" }

Response:

{ "tokenCount": 8 }

Limits: max 100,000 characters per request.

Example

curl -X POST https://tokenizer.example.workers.dev/api/tokenize \
  -H "Content-Type: application/json" \
  -d '{"text": "Hello, world!"}'

Tech Stack

• Hono — lightweight web framework for edge runtimes
• gpt-tokenizer — pure JS BPE tokenizer (GPT-4o / cl200k_base)
• Cloudflare Workers — serverless edge compute