Common AI Terms Explained: Tokenization to Attention

When software engineers start exploring Large Language Models (LLMs) and Generative AI, they encounter a wall of specialized machine learning jargon: Tokens, Embeddings, Softmax, Greedy Decoding, Self-Attention.

While these terms sound intimidating, they actually represent logical steps in a clean data pipeline. An LLM is simply a mathematical function that takes input text, converts it into numbers, routes context through vector spaces, and calculates probabilities to predict the next word.

In this engineer's guide, we break down the 7 most fundamental terms in AI with plain-English definitions, concrete code/mathematical examples, visual flowcharts, and links to deep-dive guides.

End-to-End LLM Processing Pipeline

Before diving into individual definitions, let's look at how these 7 terms connect together inside a single forward pass of a Large Language Model:

sequenceDiagram
    autonumber
    actor User
    participant Tok as Tokenizer
    participant Enc as Embedding Layer
    participant PosEnc as Positional Encoder
    participant Attn as Attention Engine
    participant Softmax as Softmax Layer
    participant Dec as Greedy Decoder
    participant Detok as Detokenizer

    User->>Tok: Raw Input Text ("The dog bit the man")
    Tok->>Enc: Integer Token IDs ([464, 3290, 6491, 262, 587])
    Enc->>PosEnc: Semantic Embedding Vectors (X_token)
    Note over PosEnc: Inject Word Order (RoPE / Position Vectors)
    PosEnc->>Attn: Position-Aware Vectors (X = X_token + P_pos)
    Note over Attn: Q, K, V Projections & Context Shift
    Attn->>Softmax: Raw Output Logits ([2.0, 1.0, 0.1])
    Softmax->>Dec: Probability Distribution (65.9%, 24.2%, 9.9%)
    Dec->>Detok: Selected Top Token ID (Argmax)
    Detok-->>User: Human Text String ("dog")

Tokenization

Tokenization is the process of chopping raw human text strings into small, discrete numerical chunks called tokens.

Computers cannot directly read letters or words; they only operate on numbers. Tokenization acts as the entry gateway to any AI pipeline. Modern LLMs do not tokenize by whole words or individual characters; instead, they use subword algorithms like Byte-Pair Encoding (BPE).

Example

Consider the word "unbelievable". A subword tokenizer breaks it into three frequent prefix/root/suffix chunks:

"unbelievable"  ──>  ["un", "believ", "able"]  ──>  [743, 29104, 612]

Common words like "the" or "dog" receive a single token ID, while rare technical terms, code snippets, or foreign characters are split into multiple subword tokens. On average, 1 token corresponds to roughly 4 characters or 0.75 words in English.

Encoding

Encoding is the process of converting discrete integer token IDs into dense, continuous numerical vectors called embeddings.

While integer token IDs (e.g., dog = 3290) give words a numerical ID, they carry a fatal flaw: numbers carry arbitrary distance (e.g., 3290 is adjacent to 3291, making a machine learning model mistakenly think unrelated words are identical). Encoding fixes this by mapping every token index to a list of hundreds or thousands of floating-point numbers.

Example

In a 1,536-dimensional model (like OpenAI's text-embedding-ada-002), the token ID 3290 ("dog") is encoded into a high-dimensional coordinate array:

Token ID: 3290 ("dog")  ──>  Dense Embedding: [0.021, -0.015, 0.891, ..., -0.043]  (1,536 dimensions)

In this geometric hyper-space, semantically similar words sit close to one another: "dog" and "puppy" point in almost identical directions, while "toaster" points far away.

Positional Encoding

Positional Encoding is the process of generating a unique numerical vector for every position in a sequence (position 1, position 2, position 3) and adding those position signals directly to the word's embedding vector before attention takes place.

Unlike Recurrent Neural Networks (RNNs) that read text sequentially word-by-word, modern Transformer attention engines process all tokens in a sentence simultaneously in parallel. This makes Self-Attention permutation-invariant: without Positional Encoding, the attention engine cannot tell the difference between "The dog bit the man" and "The man bit the dog"!

How It Works: If the word "Dog" appears at the beginning of a sentence versus the middle, its base semantic meaning stays the same, but its total input vector changes slightly because a unique position signal is added to it:

Final Input Vector = Word Embedding + Positional Encoding

Semantic Context

Semantic Context refers to how the meaning of a word changes dynamically based on the surrounding tokens in a sentence.

In human language, many words suffer from polysemy—having multiple meanings depending on how they are used. Static dictionary embeddings output the exact same vector for a word regardless of context. Semantic context mechanisms allow an AI model to update a word's coordinates dynamically on the fly.

Example

Consider the word "bank" in two different sentences:

• Sentence A: "I sat by the river bank."
• Sentence B: "I deposited money in the bank."

Without semantic context, "bank" receives a static vector averaging nature and money coordinates. With dynamic semantic processing, "bank" in Sentence A absorbs contextual signals from "river" (shifting its coordinates toward nature/water), while in Sentence B it absorbs signals from "deposited" (shifting toward finance).

Attention (Self-Attention)

Attention is the central neural engine of the Transformer architecture. It is the mathematical mechanism that computes how much every token in a sentence should focus on every other token to build dynamic semantic context.

Attention projects input token vectors into three distinct roles using trainable weight matrices:

1. Query (Q): What the current token is looking for in surrounding words.
2. Key (K): What kind of information the token advertises about itself.
3. Value (V): The actual content payload passed along when a match is found.

Formula & Example

Attention(Q, K, V) = softmax( (Q × Kᵀ) / √d_k ) × V

In the sentence "The dog bit the man", the verb query Qbit scans the keys of all surrounding words. It finds strong compatibility matches with Kdog (the subject actor) and Kman (the target object), and pulls their Value payloads (Vdog, Vman) into its own embedding to construct an understanding of a canine attack.

Softmax Function

The Softmax Function is a mathematical function that takes an unnormalized array of raw numerical scores (called logits) and transforms them into a valid probability distribution where every value is between 0.0 and 1.0, and all values sum up to exactly 1.0 (100%).

Softmax serves two crucial roles in AI:

1. Inside Self-Attention: Turns dot-product similarity scores (Q × KT) into attention percentages.
2. At Model Output: Turns raw output vocabulary logits into next-token prediction probabilities.

Mathematical Formula

Softmax(zᵢ) = e^(zᵢ) / Σ(e^(zⱼ))

Step-by-Step Mathematical Example

Suppose an LLM is predicting the next word after "The sky is..." and outputs raw logits for three candidate tokens: "blue" (logit = 2.0), "cloudy" (logit = 1.0), and "green" (logit = 0.1).

1. Step 1: Compute Exponentials (e^z)

   e^(2.0) ≈ 7.389
   e^(1.0) ≈ 2.718
   e^(0.1) ≈ 1.105

2. Step 2: Sum All Exponentials

   Sum = 7.389 + 2.718 + 1.105 = 11.212

3. Step 3: Divide Each Exponential by the Sum

   Probability("blue")   = 7.389 / 11.212 ≈ 0.659  (65.9%)
   Probability("cloudy") = 2.718 / 11.212 ≈ 0.242  (24.2%)
   Probability("green")  = 1.105 / 11.212 ≈ 0.099  ( 9.9%)

Result: Raw logits [2.0, 1.0, 0.1] become clean, usable probabilities [65.9%, 24.2%, 9.9%] that sum to 100%.

Greedy Algorithm (Greedy Search)

A Greedy Algorithm (or Greedy Search Decoding) is a text generation strategy where the AI model simply selects the single token with the highest probability score (the argmax) at every single step, without considering future paths.

In LLMs, setting the sampling parameter Temperature to 0.0 enables pure Greedy Search.

Example

If Softmax outputs probabilities for the next token:

"blue": 65.9%  |  "cloudy": 24.2%  |  "clear": 9.9%

A Greedy Search algorithm will always pick "blue" (100% deterministically), because 65.9% is the highest number.

Pros & Cons of Greedy Search:

• Pros: Fast, deterministic, reproducible (ideal for code generation, math, and factual Q&A).
• Cons: Can get stuck in repetitive loops or miss creative, high-quality responses that require picking slightly lower-probability tokens.

Detokenization

Detokenization is the final step in an LLM pipeline that converts predicted numerical token IDs back into human-readable text strings for display in the UI.

Because modern tokenizers operate on subwords, detokenization involves stitching subword fragments together, handling whitespace prefixes (like Ġ in GPT tokenizers or in SentencePiece), and resolving special control characters.

Example

Predicted Token IDs: [743, 29104, 612]  ──>  Subwords: ["un", "believ", "able"]  ──>  Text Output: "unbelievable"

When you stream responses from ChatGPT or Claude, the API generates one token ID at a time. The client detokenizes each incoming token ID on the fly, rendering text word-by-word into your browser screen in real time.

LLM Parameters (Model Size)

In a Large Language Model, parameters are the adjustable "knobs and dials" inside the neural network. They are the trainable numerical weight matrices (such as WQ, WK, WV, WO, and Feed-Forward Network weights) that the model learned during pre-training.

When engineers talk about an "8B" (8 billion), "70B" (70 billion), or "405B" model, they are referring to the total count of these individual floating-point weight numbers stored inside the model's architecture:

• More Parameters (e.g., 70B - 405B): Provide higher reasoning capacity, deeper knowledge retrieval, and better nuance, but require massive GPU memory (VRAM).
• Fewer Parameters (e.g., 1B - 8B): Offer faster inference and lower operational costs, suitable for edge devices and lightweight applications.

How Are Parameters Calculated?

Parameters are calculated mathematically from the model's architectural blueprint based on linear algebra matrix dimensions (Rows × Columns):

1. Matrix Multiplication (Rows × Columns): Inside a Transformer layer, a linear layer multiplies an input vector of dimension D by a weight matrix of size D × D. The parameter count for a single square matrix is D2.
2. Attention & FFN Weights per Layer: For a hidden dimension of D = 768, each attention matrix (WQ, WK, WV, WO) contains 768 × 768 = 589,824 parameters (~2.35M total for attention). Adding the Feed-Forward Network (FFN) expansion layers (768 × 3072) adds another ~4.71M parameters.
3. Scaling Across Stacked Layers: Multiplying the total parameter count of one layer by the number of stacked layers (e.g., 32, 80, or 96 layers) yields the total model parameter count (e.g., 7B, 70B, or 405B).