A token is the smallest unit large language models use to process text — AI doesn’t read by “characters” or “words” but splits text into tokens. One Chinese character is roughly 1-2 tokens; one English word is roughly 1-3 tokens. Tokens are AI’s starting point for “understanding” your content and the base unit for GEO information density calculations.
Plain-Language Analogy
Humans read sentences word by word. AI doesn’t.
AI first chops all text into smaller fragments — these fragments are tokens. Some tokens are complete words (“the”), some are word parts (“un” + “believe” + “able”), and a Chinese character might be one token or split into two.
Think of tokens as LEGO bricks. Humans see a complete castle (sentence). AI sees a pile of bricks (tokens) — it first understands each brick, then figures out how they fit together.
Why Tokens Matter for GEO
The base unit for information density
Get AI to Speak for You: The Definitive Guide to GEO, Chapter 2, Section 2.6 states: “Make every token carry useful information.”
A 200-token passage where 100 tokens are filler (“with the rapid development of the industry,” “as is well known”) and only 100 tokens carry actual information (data, conclusions, facts) has just 50% information density. A competitor’s 200 tokens at 80% density will outperform in vector retrieval and re-ranking.
The measuring unit for context windows
AI’s context window is measured in tokens. RAG-injected content is typically only a few hundred to a few thousand tokens. Every wasted token is a wasted citation opportunity.
The basic scale for chunking
RAG systems chunk by token count or semantic boundaries. Understanding tokens explains why paragraph length matters — too many tokens per paragraph creates diffuse chunks; too few creates fragments lacking standalone value.
Practical Advice
- Check information density of key paragraphs — count how many tokens carry real information vs. filler. If filler exceeds one-third, trim it
- Evaluate content length in token terms — not “is it long enough” but “is every token delivering value”
- Know your Answer Block’s approximate token count — a 150-300 word English Answer Block is roughly 200-400 tokens, fitting within one chunk
Further Reading
- Get AI to Speak for You: The Definitive Guide to GEO, Chapter 2, Section 2.2 — “Token: AI’s Reading Unit”
- Get AI to Speak for You: The Definitive Guide to GEO, Chapter 2, Section 2.6 — “Make Every Token Carry Useful Information”
- Free GEOBOK tools: Token Calculator, Token Density Detector
English ▾
Chinese (Simplified)