Cracking Self-Attention: Understanding Q, K, and V in Large Language Models

When diving into how Large Language Models (LLMs) work, you inevitably hit the concept of self-attention. It is the central engine powering the Transformer architecture—the magic mechanism that converts static dictionary embeddings into rich, context-aware representations.

If you've ever looked at the math and wondered, "Where do Q, K, and V actually come from, and why do we only have Queries and Keys instead of a separate context matrix?"—you are not alone.

In this guide, we break down the entire journey of self-attention step-by-step: why static embeddings fail, where weight matrices come from, intuitive real-world analogies, and a complete mathematical calculation from scratch.

Why Only Embedding Is Not Enough

When researchers introduced Word Embeddings (like Word2Vec and GloVe), it felt like a breakthrough. Computers could map words to vector coordinates where "king" - "man" + "woman" = "queen".

However, software engineers quickly realized that embeddings alone hit three severe structural walls:

The Polysemy Trap (Static Vectors for Dynamic Words)

In linguistics, polysemy refers to words with multiple meanings depending on context. Consider these two sentences:

• Sentence A: "I sat down by the river bank to watch the sunset."
• Sentence B: "I walked into the bank to deposit a check."

In a standard word embedding lookup table, the token "bank" is assigned a single, fixed coordinate vector (e.g., [0.42, -0.18, 0.89, ...]). Regardless of whether you are describing nature or financial institutions, the embedding layer outputs the exact same vector.

Order and Relationship Blindness

Word embeddings carry no inherent mechanism to determine who is doing what to whom. If you feed a bag of embeddings into a basic network, these two sentences look virtually identical mathematically:

• "The dog bit the man"
• "The man bit the dog"

Both sentences contain identical sets of embedding vectors. Without dynamic routing between words, static embeddings cannot capture true grammatical relationships.

The Sequential Memory Bottleneck (RNNs & LSTMs)

Before Transformers, engineers tried using Recurrent Neural Networks (RNNs). RNNs processed sentences sequentially, passing a hidden memory vector step-by-step.

This created a bottleneck: by word 50, information from word 1 faded away (the vanishing gradient problem). Furthermore, sequential processing could not be parallelized across GPUs.

What Problems Dynamic Embeddings & Attention Solve

Introduced in the landmark 2017 paper "Attention Is All You Need" by Vaswani et al., the Attention mechanism solved the fundamental limitations of static embeddings by introducing Dynamic Contextual Weighting.

Instead of treating words as isolated points in space, Attention allows every token to examine every other token in the sentence simultaneously, calculate mutual relevance, and dynamically adjust its own coordinate representation.

Dynamic Vector Transformation

When text enters a Transformer model, tokens initially lookup static embeddings. However, as they move through attention layers:

• The vector for "bank" in "river bank" absorbs signals from "river" and shifts its coordinates toward nature/water space.
• The vector for "bank" in "bank account" absorbs signals from "account" and shifts its coordinates toward finance space.

By the deep layers of the LLM, token vectors are no longer static dictionary definitions—they become dynamic, context-aware representations of that word's exact meaning in that specific sentence.

Direct 1-Step Long-Range Connections

In an RNN or LSTM, information from token 1 had to step sequentially through 50 intermediate hidden states to reach token 50. With Self-Attention, the mathematical path length between token 1 and token 50 is exactly 1 step. Token 50 can attend directly to token 1 with zero signal decay or memory loss.

Massive GPU Parallelization

Because Attention calculates token relationships using matrix multiplication (Q × K^T) rather than sequential loops, the entire operation can be computed concurrently on modern GPU tensor cores. This parallelization enabled training modern LLMs on trillions of tokens across massive GPU clusters.

Where Do Q, K, and V Come From?

To understand WQ, WK, and WV (the weight matrices that generate Queries, Keys, and Values), we have to look at the lifecycle of a Large Language Model:

• Random Initialization: Before training begins, these three weight matrices are simply initialized with random numbers (drawn from a standard normal distribution).
• Learned Parameters via Backpropagation: As the LLM processes massive amounts of text and attempts to predict missing or next tokens, it computes prediction errors. During backpropagation, gradients flow backward through the network, updating the numbers inside W_Q, W_K, and W_V just like every other weight layer in the neural network.
• Frozen at Inference: By the time the model finishes training, these matrices are locked. When you feed a new sentence into the trained LLM, it uses these fixed projection weights to calculate how words focus on one another in real-time.

What Are They Actually Doing?

Think of WQ, WK, and WV as projection lenses applied to your initial token embeddings (X). When a sentence enters an attention layer, every token embedding vector is multiplied by these matrices to produce three distinct vectors:

1. Query (Q = X · W_Q): What the current token is looking for in other words.
2. Key (K = X · W_K): What kind of information the token contains (its label or index identifier).
3. Value (V = X · W_V): The actual content payload passed along once a match is established.

The Dating App Analogy

A helpful way to visualize Q, K, and V is through a matchmaker or dating app comparison:

• Q and K are the matchmakers: They don't carry the heavy cargo of information; their sole job is to calculate compatibility (who should talk to whom). Think of Query (Q) as your search filters and preferences, and Key (K) as user profile tags or attributes.
• V is the actual personality/content: Once the system determines who matches with whom via Q and K, it pulls the Value (V) from the matched profile to combine and shape the final user experience.

The Database Search Engine Analogy (Exact vs. Probabilistic)

For software engineers, the clearest way to understand Q, K, and V is to look at how database search engines query data:

1. Query (Q): You submit a search request specifying what you are looking for (e.g., WHERE category = 'fruit').
2. Key (K): The database checks your Query against indexed column labels (Keys) attached to every stored entry.
3. Value (V): When a match is found, the database hands you the actual stored data content payload (Value).

The Filing Cabinet Analogy

Imagine standing in a giant archival library:

• You hold a research query written on a notepad (Query).
• Every folder in the cabinet has a label tab printed on its edge (Key).
• You compare your query notepad against every label tab to measure relevance.
• Once you find matching tabs, you open those specific folders and extract the document contents inside (Value).

Conceptual Examples: Grammar, Semantics, and Action Routing

To truly grasp how Attention operates in natural language, let's trace three distinct scenarios handled by Q, K, and V:

1. Grammatical Modifiers: Connecting syntactic roles (e.g., adjectives modifying nouns).
2. Semantic Disambiguation: Resolving multi-meaning words (polysemy) using world knowledge.
3. Action & Relationship Routing: Resolving subject-action-object dynamics across full sentences.

Grammatical Relationship ("Cute Dog")

Consider the two-word phrase: "Cute dog". When transformed by the attention layer, the tokens generate the following roles:

• For "Cute":
  • QCute: "I am an adjective looking for a noun to modify."
  • KCute: "I am a descriptive adjective."
  • VCute: The conceptual traits of cuteness (small, adorable, soft).
• For "Dog":
  • QDog: "I am a noun looking for descriptors or actions."
  • KDog: "I am a physical animal noun."
  • VDog: The conceptual meaning of a dog (canine, fur, barking, four legs).

When the model evaluates compatibility, QCute matches strongly with KDog because adjectives naturally pair with nouns. Once this high compatibility score is computed, the representation for "Cute" pulls a heavy portion of VDog into its own vector, evolving its isolated dictionary meaning into an understanding specifically bound to a dog.

Semantic & Disambiguation Relationship ("Apple Pie")

Now consider a factual, non-grammatical example: "Apple pie".

In isolation, the word "Apple" is ambiguous. It comes in with a static embedding that contains multiple overlapping meanings (fruit, tech giant, record label). When paired with "pie", the LLM resolves this ambiguity step-by-step through the attention mechanism:

1. The Ambiguity (X): The word "Apple" enters with an initial static embedding vector (X) that holds a mixture of all potential meanings (fruit, computer manufacturer, record label).
2. The Search (Q): QApple asks: "I am a multi-meaning entity. I need surrounding context to lock down my exact definition."
3. The Possibilities (K): KPie advertises what it represents: "I am a baked dessert food item."
4. The Match (Q × K^T): The model computes QApple × KPieT. Realizing that an apple is a classic baking fruit, the network's trained weights assign a very high attention percentage between "Apple" and "Pie".
5. The Content Payload (V): Because they matched, the model extracts the Value vector (VPie)—the actual conceptual substance of the pastry—and blends it into "Apple". This resolves the ambiguity, filtering out the "tech company" coordinate space and leaving a context-aware food representation.

Action & Relationship Routing ("The Dog Bit the Man")

Let's trace a full sentence containing a subject, an action verb, and an object: "The dog bit the man".

1. The Starting Ambiguity & Representation (X): Before self-attention, words are isolated dictionary tokens. "Dog" could be barking, sleeping, or chasing. "Bit" is an action, but who is performing it and who is receiving it? Without context, static embeddings cannot determine whether the dog bit the man or the man bit the dog.
2. The Three Roles (Q, K, and V):
   • Queries (Q): Qbit searches: "I am an action verb. I desperately need to know who performed the action (subject) and who received it (object)!" Qdog searches for its action.
   • Keys (K): Kdog advertises: "I am an animate creature capable of performing actions." Kman advertises: "I am a living target capable of receiving actions." Kbit advertises: "I am a physical biting action."
   • Values (V): Vdog, Vbit, and Vman hold the raw conceptual payloads of a dog, a bite, and a man.
3. The Matchmaking (Q × K^T): The verb query Qbit scans all Keys in the sentence. It matches strongly with Kdog (logical actor) and Kman (logical recipient). The model's trained weights evaluate sequence structure, determining that Dog is performing the action and Man is receiving it.
4. Swapping Content Payload (V):
   • Updating "Bit": The verb "bit" pulls in Vdog and Vman. It is no longer an abstract dictionary verb; its embedding transforms into a dog biting a man.
   • Updating "Dog": The noun "dog" pulls in Vbit, updating its representation so the model knows this specific dog is currently acting aggressively.

The Mathematical Calculation (Toy Example)

To see how the math ties everything together, let's step through a concrete numerical calculation using 2-dimensional vectors for two words: "Cute" (Word 1) and "Dog" (Word 2).

The core self-attention formula defined by Vaswani et al. is:

Attention(Q, K, V) = softmax( (Q × Kᵀ) / √d_k ) × V

Step 1: Define the Input Vectors

Suppose our linear projection matrices produce the following 2D Queries, Keys, and Values:

• Queries (Q): Q1 = [1, 0] (Cute),   Q2 = [0, 1] (Dog)
• Keys (K): K1 = [0, 1] (Cute),   K2 = [1, 0] (Dog)
• Values (V): V1 = [2, 5] (Cute),   V2 = [3, 1] (Dog)

Step 2: Calculate Compatibility Scores (Q × K^T)

We multiply the Queries matrix by the transposed Keys matrix (K^T) to measure dot-product similarity between all word pairs (omitting scaling by √dk for simplicity):

Scores = Q × Kᵀ = [ [1, 0], [0, 1] ] × [ [0, 1], [1, 0] ] = [ [0, 1], [1, 0] ]

Interpreting Row 1 ("Cute"):

• Match between QCute and KCute = (1×0) + (0×1) = 0
• Match between QCute and KDog = (1×1) + (0×0) = 1 (High similarity match!)

Step 3: Apply Softmax (Turn Scores into Percentages)

Passing the raw scores through the Softmax function normalizes each row into probabilities that sum to 1.0 (100%):

Weights = softmax( [ [0, 1], [1, 0] ] ) ≈ [ [0.27, 0.73], [0.73, 0.27] ]

Interpretation for "Cute": The model pays 27% attention to itself and 73% attention to "Dog".

Step 4: Multiply by Values (V)

Finally, we take the weighted sum of the Value vectors to construct the updated context vector for "Cute":

New Output (Cute) = (0.27 × V₁) + (0.73 × V₂)
                 = 0.27 × [2, 5]ᵀ + 0.73 × [3, 1]ᵀ
                 = [0.54, 1.35]ᵀ + [2.19, 0.73]ᵀ
                 = [2.73, 2.08]ᵀ

Mathematical Conclusion

The initial Query vector for "Cute" was [1, 0]. Through the projection lenses of Q, K, and V, it outputted [2.73, 2.08]—successfully absorbing the semantic context of "Dog" directly into its mathematical identity.

Transformer Flow & Single-Head Limitations

Below is the complete computational diagram showing how input embeddings pass through QKV matrix projections, dot-product similarity, scaling, softmax, and value aggregation:

graph TD
    Input["Input Token Embedding (X)"] --> LinearQ["Linear Projection W_Q"]
    Input --> LinearK["Linear Projection W_K"]
    Input --> LinearV["Linear Projection W_V"]

    LinearQ --> Q["Query Vector (Q = X · W_Q)"]
    LinearK --> K["Key Vector (K = X · W_K)"]
    LinearV --> V["Value Vector (V = X · W_V)"]

    Q & K --> DotProduct["Dot Product (Q × Kᵀ)\nRaw Similarity Matrix"]
    DotProduct --> Scale["Scale (/ √d_k)\nGradient Stability"]
    Scale --> Softmax["Softmax\nAttention Percentages (0% to 100%)"]
    Softmax & V --> MatMul["Multiply by V\nWeighted Sum of Values"]
    MatMul --> Output["Contextual Output Vector (Z)"]

Beyond Tunnel Vision: How Multi-Head Attention Gives LLMs 96 Perspectives at Once

Standard self-attention has a major blind spot: It suffers from tunnel vision.

If an AI model relies on only a single set of WQ, WK, and WV matrices, it is forced to mash every single relationship—grammar, pronoun coreference, spatial semantics, and action dynamics—into one single, averaged-out output vector.

The Problem: Sentences Have Multiple Layers of Meaning

Consider this classic, complex sentence:

To truly understand what is happening here, a model cannot rely on just one single analytical lens. It needs to track several distinct relationships at the exact same time:

1. Pronoun Resolution: What does "it" refer to? (Connects "it" to "bank").
2. Grammar & Modifiers: What was muddy? (Connects "muddy" to "bank").
3. Geography & Semantics: What kind of bank is near a river? (Connects "bank" to "river").

If a model had only one pair of eyes (single-head attention), it would struggle to balance grammar rules, geographic semantics, and pronoun resolution all at once.

The Solution: Multi-Head Attention (The Team of Experts)

Instead of relying on a single set of WQ, WK, and WV matrices, Multi-Head Attention splits projection weights into multiple independent sets (called "heads").

Think of it like placing a team of specialized detectives on a single sentence:

• Head 1 (Grammar Expert): Trained to focus purely on parts of speech (linking adjectives to nouns).
• Head 2 (Pronoun Expert): Focuses on coreference resolution (linking "it" back to "bank").
• Head 3 (Action Expert): Focuses on subject-action-object relationships (who is doing what to whom).
• Heads 4 through 8+ (Syntax & Flow Experts): Focus on clause boundaries, punctuation, or long-range context.

Each head runs its own independent Q, K, and V calculations simultaneously in parallel.

Scaling Up: 96 Heads Across Deep Layers

In massive models like GPT-3 and GPT-4, the scale is staggering. They do not just use 8 heads; they multiply them across deep neural architectures:

• Multiple Heads Per Layer: A single Transformer layer often contains 96 individual attention heads running parallel matchmaking operations.
• Stacked Deep Layers: That entire multi-head process is repeated 96 layers deep.

This creates a hierarchy of understanding across layers:

• Early Layers: Focus on surface-level details (token spelling, local word pairs).
• Middle Layers: Track clause syntax and phrase structures.
• Deep Layers: Handle abstract logic, long-range context, and paragraph-level coreferences.

How Do We Blend All Those Perspectives Together?

Having 8 or 96 separate expert heads running in parallel is powerful, but eventually, the LLM needs a single, unified vector representation for each word. The model merges them in a two-step mechanical process:

Step 1: Concatenation (Gluing Insights Side-by-Side)

When all individual attention heads finish their calculations, each head outputs its own context vector (Z1, Z2, ..., Zh) for every word. The model glues these separate vectors together end-to-end:

Concatenated Vector = Concat(Z₁, Z₂, Z₃, ..., Zₕ)

At this moment, the word vector is a long array of separate expert opinions stacked side-by-side.

Step 2: The Final Linear Projection (W_O)

To make sure the grammar expert, pronoun expert, and semantic expert cross-pollinate their findings, the model passes that massive concatenated vector through one final trained weight matrix called WO (the Output Projection matrix):

Final Output (Z) = Concatenated Vector × W_O

The WO matrix acts like a boardroom meeting where all expert insights debate and merge. It compresses the vector back down to the model's standard embedding dimension, thoroughly weaving grammar, action, and semantics into a single, multi-dimensional representation.

flowchart TD
    X["Input Token Vector (X)"] --> H1["Head 1: Grammar\n(W_Q1, W_K1, W_V1)"]
    X --> H2["Head 2: Pronoun Coreference\n(W_Q2, W_K2, W_V2)"]
    X --> H3["Head 3: Action & Object\n(W_Q3, W_K3, W_V3)"]
    X --> Hn["Head H (up to 96 Heads)\n(W_Qh, W_Kh, W_Vh)"]

    H1 --> Z1["Output Vector Z1"]
    H2 --> Z2["Output Vector Z2"]
    H3 --> Z3["Output Vector Z3"]
    Hn --> Zn["Output Vector Zh"]

    Z1 & Z2 & Z3 & Zn --> Concat["Concatenation\nConcat(Z1, Z2, ..., Zh)"]
    Concat --> Wo["Linear Output Projection W_O\n(Cross-Pollinate & Compress)"]
    Wo --> FinalOutput["Final Context-Aware Vector (Z)"]

Summary: Word Embeddings vs. Self-Attention