What Exactly Is a Token? (And How AI Actually Reads Text)

If you've ever paid for an AI API, wondered why your carefully crafted prompt got cut off, or noticed an AI completely stumble over a weirdly spelled word, the answer usually comes down to one thing: tokens.

Tokens are the atomic units of the AI universe. If you can wrap your head around how they work, it completely changes how you write prompts, estimate your costs, and troubleshoot weird model behaviour.

Here's the short version: a token isn't a word, and it isn't a letter. It's a strange, learned chunk of text that sits somewhere in between. And that distinction matters a lot more than most people realise.

So, What Is a Token?

When you type a message into a language model, it doesn't actually read your words. The very first thing that happens is a process called tokenization. This slices your text up into tiny, digestible pieces that the model can process as numbers. Each of those little pieces is a token.

These pieces are created by a tokenizer — a completely separate program that runs right before the main AI model. The most popular method is called Byte Pair Encoding (BPE). Think of the tokenizer as an algorithm that reads through millions of books and learns which letters usually hang out together. Really common words get to be a single token. Rare, long, or made-up words get chopped up into multiple pieces.

graph TD
    A["Raw text input"] --> B["Tokenizer\n(BPE algorithm)"]
    B --> C["Token IDs\n[15496, 11, 995, 0]"]
    C --> D["AI Model\nprocesses numbers"]
    D --> E["Output token IDs"]
    E --> F["Detokenizer"]
    F --> G["Human-readable text"]

Notice how the AI never actually sees your text? It only ever deals with a list of numbers (token IDs). Each ID corresponds to a specific chunk of text in the tokenizer's dictionary. The model spends its training learning what each of those IDs means, but under the hood, it's all just numbers.

Why Not Just Use Letters or Words?

You might be wondering: if we just want to process text, why not just feed the AI individual letters? Or better yet, whole words? Well, both extremes actually fail pretty hard. Tokens hit the Goldilocks zone. Instead of random chunks, they act like syllables, word stems, and root words that actually carry real meaning.

Letters are way too small

Imagine if the AI only knew the 26 letters of the alphabet. Every single letter would be its own token. If you typed Unbelievable, the model would have to juggle 12 separate pieces of information just to read one word. Plus, a random letter like b doesn't really mean anything on its own. The AI would burn all its computing power just sounding out words before it could even start thinking about the sentence.

Whole words are way too rigid

Okay, what if it just memorised a giant dictionary of whole words? That causes the dreaded Out of Vocabulary problem. Let's say the AI learns the word friend. Tomorrow, you ask it about someone being unfriendly. If the dictionary only has whole words, the AI looks up unfriendly, finds nothing, and completely panics. Any new slang, combined word, or slight typo becomes a brick wall.

Subwords are "just right"

By splitting language up into subword chunks, the AI gets the best of both worlds. It learns the core building blocks—roots, prefixes, and suffixes—and snaps them together like Lego bricks whenever it encounters something new.

graph LR
    INPUT["unfriendly"] --> UN["'un'\n(means: not)"]
    INPUT --> FRIEND["'friend'\n(root word)"]
    INPUT --> LY["'ly'\n(turns word into adverb)"]

    UN & FRIEND & LY --> MEANING["Meaning assembled:\nnot in a friendly way"]

The AI might never have seen the exact string unfriendly during training. But it doesn't care. It neatly chops the word into un + friend + ly, pieces the three familiar chunks together, and understands exactly what you mean.

This is the true brilliance of tokenization. By focusing on chunks instead of full words, the AI effectively learns how to build language from scratch, rather than trying to memorise every possible Lego set in existence.

Why the Subword Approach is So Powerful

• Handling Unknown Words: If a model encounters a brand-new word, a typo, or internet slang, it doesn't fail. It simply breaks the unknown word down into smaller, familiar subword tokens (e.g., splitting unpredictable into un, predict, and able).
• Controlling Vocabulary Size: Keeping a dictionary of every possible word in every human language would require millions of entries. Grouping common letter pairs keeps the vocabulary size manageable (usually around 32,000 to 100,000 tokens) while remaining flexible enough to construct virtually anything you can type.
• Computational Efficiency: Common words, roots, and phrases are packed into single tokens. This drastically reduces the total number of chunks in a sentence, allowing the model's memory and attention mechanism to process much longer text much faster.
• Learning Grammar and Patterns: Frequent suffixes and prefixes (like -ing, -ed, or re-) become standard tokens. This helps the AI easily recognize tense, plurality, and grammatical structures across completely different root words.

In essence: Tokenization identifies intuitive groupings of characters that appear frequently together. This statistical approach solves a massive scaling problem—it allows the AI to understand and generate an infinite variety of concepts without needing to memorise an impossibly large dictionary of every possible whole word.

What a Token Actually Looks Like

A lot of guides will tell you "a token is roughly four characters" and leave it at that. But to really get it, you need to see what actual tokenization looks like in practice.

Take the phrase: "Tokenization is fascinating."

graph LR
    INPUT["Tokenization is fascinating."]

    INPUT --> T1["Token 1\n'Token'"]
    INPUT --> T2["Token 2\n'ization'"]
    INPUT --> T3["Token 3\n' is'"]
    INPUT --> T4["Token 4\n' fasc'"]
    INPUT --> T5["Token 5\n'inating'"]
    INPUT --> T6["Token 6\n'.'"]

That's six tokens for a three-word sentence! Notice how it split things up: "Token" and "ization" got separated because "Tokenization" is a pretty uncommon word, so it didn't get its own VIP slot in the dictionary. The word "is" actually has a space attached to the front of it (' is'), because the tokenizer learned that words almost always have a space before them. And "fascinating" was chopped into "fasc" and "inating".

Now look at a super common phrase like Hello, world!:

graph LR
    INPUT2["Hello, world!"]
    INPUT2 --> T1["Token 1\n'Hello'"]
    INPUT2 --> T2["Token 2\n','"]
    INPUT2 --> T3["Token 3\n' world'"]
    INPUT2 --> T4["Token 4\n'!'"]

Four tokens. Very clean splits. These words are used so often that they easily earned their own single tokens. Notice that the punctuation gets isolated too? The model has to learn grammar, and grammar lives in the punctuation marks.

Why Tokens Trip Us Up

Once you see how tokenization works, a few weird AI quirks start to make a lot of sense.

The exact same word can tokenize differently

To the AI, the word dog and the word dog (with a space in front) are usually two completely different tokens. It literally sees them as distinct concepts. This is why AI can act a bit strange if you mess with the spacing in your prompts—you're feeding it a different sequence of numbers.

Numbers burn through your token budget

A big number like 1234567 doesn't usually get to be one token. It gets shredded into pieces like 1, 234, and 567. This is a huge reason why early AI struggled so much with basic math—it was trying to do arithmetic on scattered fragments of numbers that it had to hold in its memory.

Code is way more expensive than prose

When you're writing code, things like indentations, brackets, and operators all eat up tokens. A single line of Python might generate way more tokens than an English sentence of the same length, simply because coding symbols rarely get grouped together with the text around them.

for 1 in rang(10):
  print (i)

graph TD
    CMP["Token count comparison\nfor ~same character length"]

    CMP --> P["Prose sentence\n'The cat sat on the mat.'\n→ ~7 tokens"]
    CMP --> C["Code snippet\nfor i in range(10):\n    print(i)\n→ ~14 tokens"]
    CMP --> N["Number\n123456789\n→ ~4 tokens"]

    P --> NOTE1["Prose is cheapest\nto tokenize"]
    C --> NOTE2["Code burns tokens fast\ndue to symbols"]
    N --> NOTE3["Large numbers split\ninto digit groups"]

How Tokens Are Used to Train AI

When an AI is being built, the training process is split into two entirely different phases: building the dictionary, and training the brain.

Phase 1: Building the Dictionary

Before the AI does any learning, developers run the tokenizer over a massive mountain of text just to figure out what the optimal "chunks" should be. The tokenizer finds the most common patterns—deciding if life gets to be a single token, or if it should be split into li and fe based on statistical frequency.

Once it reaches its target size (often around 100,000 unique chunks), the dictionary is permanently locked. No more edits. Every chunk gets a permanent ID number.

Phase 2: Training the Brain

Only after the dictionary is frozen solid does the real AI training start. The AI reads through the internet using only those locked puzzle pieces, doing the heavy mathematical lifting to figure out the probability of which pieces tend to go together.

Why the Dictionary Can Never Change

Imagine if the dictionary could change its mind halfway through training. Let's say the AI spends months learning the math for li (Token ID: 40) and fe (Token ID: 50). It calculates millions of equations linking those specific numbers to other concepts.

If you suddenly stepped in and said, "Hey, let's just combine those into a single token called 'life' (Token ID: 100,001)", you would completely break the AI's brain. All the math it did for 40 and 50 is instantly useless, and it has absolutely zero data on 100,001. This is why you can't just "add new words" to an AI model without retraining it entirely.

The Tokenizer is the Bouncer

Because of this rigid rule, every AI model permanently stores its own specific list of tokens inside its Tokenizer. You can think of the Tokenizer as a bouncer, or a translator, standing at the door to the AI's brain.

graph LR
    YOU["Your Prompt"] --> BOUNCER["The Tokenizer\n(The Bouncer)"]
    BOUNCER -->|"Checks frozen dictionary\nConverts to ID numbers"| AI["AI Brain\n(Does the math)"]

When you send a prompt, you aren't talking directly to the AI. You're talking to the Tokenizer. It takes your words, pulls out its frozen dictionary, chops everything into the exact ID numbers the AI expects, and hands those numbers over to the brain to do the math.

This locked dictionary is also why different AI models speak entirely different "dialects." ChatGPT's tokenizer has a different set of locked tokens than Claude's or Google's. If you tried to use ChatGPT's tokenizer to feed numbers into Claude's brain, Claude would just spit out absolute gibberish!

How the Tokenizer Works So Fast

If the tokenizer has a massive dictionary of 100,000 chunks, doesn't it take forever to scan through that list every single time you type a word? Actually, no. Tokenization is lightning-fast—it can chew through an entire book in a fraction of a second. It pulls this off by ditching the standard alphabetical list for a very specific data structure and a clever "greedy" rule.

The dictionary is actually a tree

Instead of a flat list, the tokenizer organises its dictionary into a branching map called a Trie (often pronounced "try", short for prefix tree).

Imagine you want to tokenize the word hamburger. The tokenizer doesn't start at 'A' and read down a list. It just looks at your first letter, h, and instantly leaps to the h branch of the tree, completely ignoring the other 25 letters of the alphabet. From there, it jumps down the a branch, then the m branch. Because of this structure, the time it takes to find a chunk has absolutely nothing to do with the size of the dictionary. The speed depends entirely on the length of the word you're typing.

The "Greedy" Rule

As the tokenizer walks down this tree, it follows one strict directive: always take the longest possible path. In computer science, this is known as a "greedy" algorithm.

Let's look at our unfriendly example again. The tokenizer starts walking down the tree: u → n → f → r → i → e → n → d. It sees that unfriend is a perfectly valid chunk in its dictionary.

But it's greedy! It immediately checks to see if there is an l branch to try and make unfriendly. If the tree says, "Nope, no branch here", the tokenizer stops, snips off unfriend as the first token, and jumps right back to the top of the tree to figure out what to do with the leftover ly.

Hash Maps for instant ID lookups

Once it snips off a chunk (like unfriend), it needs to find its ID number to send to the AI. It uses a Hash Map—essentially a direct lookup system. It doesn't waste time searching for the ID; it just asks, "What is the exact ID for 'unfriend'?" and the system instantly fires back 4509 in a single step.

Because it uses prefix trees to chop the words and hash maps to grab the IDs, the tokenizer doesn't really do any "searching" at all. It just sprints down a predetermined path, snags the numbers, and hands them off to the AI's brain.

The Context Window (aka Your Token Budget)

Every language model has what's called a context window. This is a hard limit on how many tokens it can juggle at one time in a single conversation. It includes your initial system prompt, the entire chat history so far, and the model's new response. Once you hit that ceiling, the model can no longer see the beginning of the conversation. The oldest messages simply fall off the edge.

graph LR
    subgraph Window ["Context Window (e.g. 128k tokens)"]
        SP["System Prompt\n~500 tokens"]
        CH["Chat History\n~60,000 tokens"]
        UP["Your new message\n~200 tokens"]
        RE["Model response\n~800 tokens"]
    end

    OLD["Older messages\n→ pushed out, invisible to model"] -.-|"falls off"| CH

This is exactly why your AI assistant sometimes seems to "forget" things during really long chats. It doesn't actually have amnesia—those older tokens just couldn't fit inside the window anymore. The newest stuff stays, and the oldest stuff gets quietly dropped.

If you know this, you can be way smarter about how you write long prompts. If you have critical rules you want the AI to follow, put them at the very start and the very end of your prompt. Never bury them in the middle where they might get lost in the shuffle!

Wrapping Up

A token is the smallest unit of meaning that an AI can actually process. It's not quite a word, and it's not quite a character. It's a learned chunk of text that strikes a perfect balance.

The tokenizer sets the boundaries, the AI does the math on the resulting numbers, and everything else—every prediction, every context limit—is ultimately measured in these tiny pieces.

So, the next time your AI forgets a detail from an hour ago, or struggles with a brand new proper noun, you'll know exactly what's happening behind the scenes. And that little bit of knowledge is exactly what makes you a significantly better AI engineer.