The Math Behind LLM Embeddings: Vectors and Cosine Similarity

If you've ever wondered how an AI "knows" that a dog and a puppy are similar, but a dog and a toaster are not, the secret lies in a concept called Embeddings. It sounds like a complex machine learning term, but at its core, it's just geometry. Let's break down the math that makes large language models (LLMs) understand the semantic meaning of words.

Words as Coordinates in Space

Computers don't understand words. They only understand numbers. To feed text into an AI, we first have to convert words into numbers. But we can't just assign random numbers (like apple = 1, dog = 2, toaster = 3) because those numbers don't capture any relationship. 1 and 2 are close together numerically, but an apple and a dog have nothing to do with each other.

Instead, embeddings represent words as lists of numbers, which act as coordinates in a massive, multi-dimensional space. These lists are called vectors.

Imagine a simple 2D map where the X-axis is "How fuzzy is it?" (0 = not fuzzy, 1 = very fuzzy) and the Y-axis is "Is it an animal?" (0 = object, 1 = animal).

• Dog: [0.9, 1.0] (Very fuzzy, definitely an animal)
• Puppy: [0.95, 1.0] (Extremely fuzzy, definitely an animal)
• Toaster: [0.0, 0.0] (Not fuzzy, not an animal)

quadrantChart
  title Simple 2D Embedding Space
  x-axis Not Fuzzy --> Very Fuzzy
  y-axis Object --> Animal
  quadrant-1 Fuzzy Animals
  quadrant-2 Not Fuzzy Animals
  quadrant-3 Not Fuzzy Objects
  quadrant-4 Fuzzy Objects
  Toaster: [0.2, 0.2]
  Dog: [0.8, 0.8]
  Puppy: [0.9, 0.9]

If you plot these coordinates on a graph, Dog and Puppy will be right next to each other, while Toaster will be far away in the bottom left corner. This is the fundamental magic of embeddings: semantic similarity becomes spatial proximity.

Moving to High Dimensions

A 2D space isn't enough to capture all the nuance in human language. How do you map "sarcasm" or "democracy" on a 2D graph? You can't.

Real LLM embeddings (like those used in OpenAI's models) don't just use 2 dimensions. They use hundreds or thousands of dimensions. For example, the popular text-embedding-ada-002 model uses 1,536 dimensions.

Each dimension represents some abstract feature of language learned during training. We humans can't even visualize a 1,536-dimensional space, but the math works exactly the same as in 2D. A word is just a point in that hyper-dimensional space:

[0.021, -0.015, 0.099, ... and 1,533 more numbers]

To help conceptualize this without breaking your brain trying to imagine 4D geometry, we can visualize a 4-dimensional space as a set of four distinct property dials that give every word a unique coordinate signature:

graph LR
    subgraph "Word: King"
      direction TB
      K1["D1: Fuzziness (0.0)"]:::fuzzy
      K2["D2: Animal (0.0)"]:::animal
      K3["D3: Royalty (0.99)"]:::royalty
      K4["D4: Masculinity (0.99)"]:::gender
    end

    subgraph "Word: Lion"
      direction TB
      L1["D1: Fuzziness (0.85)"]:::fuzzy
      L2["D2: Animal (0.99)"]:::animal
      L3["D3: Royalty (0.60)"]:::royalty
      L4["D4: Masculinity (0.90)"]:::gender
    end

    classDef fuzzy fill:#ffcc80,stroke:#e65100,stroke-width:2px,color:#000;
    classDef animal fill:#a5d6a7,stroke:#1b5e20,stroke-width:2px,color:#000;
    classDef royalty fill:#ce93d8,stroke:#4a148c,stroke-width:2px,color:#000;
    classDef gender fill:#90caf9,stroke:#0d47a1,stroke-width:2px,color:#000;

Dimensions and Structure (Clearing Up Common Myths)

When talking about the sheer size of these models, it's easy to get terms confused. Here is how the structure actually breaks down:

• Vocabulary Size: This is the total dictionary of unique words or subwords (tokens) the model knows—typically between 32,000 and 250,000+. Every single one gets a permanent starting coordinate in the model.
• Embedding Dimensions: This is the length of the number list assigned to each item (commonly 768, 1,536, or 4,096 numbers).
• The Scale: Even with a massive vocabulary, models don't scale dimensions infinitely. They cap them around a few thousand because computer memory and mathematical efficiency hit a wall past that point.

Measuring Similarity: Cosine Similarity

Once words (or entire sentences) are mapped as vectors in space, how does the AI determine how similar they are? It measures the distance or the angle between them. The most common mathematical formula used for this is called Cosine Similarity.

Instead of measuring the straight-line distance between two points (Euclidean distance), Cosine Similarity measures the angle between the two vectors drawn from the origin (0,0) to those points.

graph LR
  O["Origin (0,0)"] -->|Vector A| A["Dog [0.9, 1.0]"]
  O -->|Vector B| B["Puppy [0.95, 1.0]"]
  O -->|Vector C| C["Toaster [0.0, 0.0]"]

The math calculates the cosine of the angle (θ) between the vectors. The result is always a number between -1 and 1:

• 1.0 (Angle is 0°): The vectors point in the exact same direction. They are perfectly similar.
• 0.0 (Angle is 90°): The vectors are perpendicular. They have absolutely no relation to each other.
• -1.0 (Angle is 180°): The vectors point in completely opposite directions. They are exact opposites.

Vector Arithmetic: Word Math

Because embeddings map meaning to numbers, you can actually perform algebraic equations on concepts. The most famous example in natural language processing (NLP) research is:

King - Man + Woman = ?

If you take the 1,536-dimensional vector for King, subtract the vector for Man (removing the concept of "maleness"), and add the vector for Woman (adding the concept of "femaleness"), the resulting coordinate lands almost exactly on the vector for Queen.

The AI doesn't possess a dictionary. It simply learned the spatial relationships between these concepts by reading billions of texts.

A Concrete Example: Building a Customer Support AI

Let's walk through a concrete example to see this in action. Imagine you are building a customer support AI, and you need it to figure out which pre-written help article best matches a user's question.

Step 1: The Setup

You have an incoming user question and two potential help articles in your database:

• User Question: "How do I give back an item I bought?"
• Article A: "Our return policy allows refunds within 30 days."
• Article B: "To reset your online account password, click here."

Step 2: Converting Sentences to Embeddings

The AI passes each of these sentences through its model to turn them into a list of numbers (vectors). To keep it simple, let's pretend our AI uses just 3 dimensions instead of thousands:

• User Question vector: [0.85, 0.10, 0.05]
• Article A (Return policy) vector: [0.80, 0.15, 0.02]
• Article B (Password reset) vector: [0.01, 0.95, 0.88]

Look closely at the numbers. Notice how the coordinates for the User Question and Article A are very close to each other (0.85 vs 0.80). Article B, on the other hand, has completely different numbers because it's about a totally different topic.

Here's what that looks like as vectors in 3D space — all three arrows drawn from the same origin:

Step 3: Measuring Similarity (The Math)

To choose which sentence is closer, the computer uses our old friend, Cosine Similarity. Without getting bogged down in complex algebra, cosine similarity measures the angle between two arrows pointing to these coordinates on a map:

• A score of 1.0 means they point in the exact same direction (identical meaning).
• A score of 0.0 means they are completely unrelated.

The computer calculates the similarity score for both options:

1. User Question compared to Article A: Score = 0.96 (Extremely high! They point in nearly the same direction).
2. User Question compared to Article B: Score = 0.12 (Very low! They point in totally different directions).

Step 4: The Result

Because Article A has the highest score (0.96), the AI instantly chooses it as the correct match and shows the return policy to the user.

Even though the user typed "give back an item I bought" and the article used "return policy allows refunds," the AI's embeddings recognized that they live in the exact same neighborhood on the semantic map!

How AI Actually Calculates and Stores Embeddings

Understanding that embeddings are coordinates is one thing, but how does an AI actually generate and store millions of these numbers? It's not magic; it's just a clever combination of massive training and simple lookup tables. Let's bring this concept down to actual computer science.

Phase 1: Learning the Meaning (Training)

Nobody sits down and manually types out the coordinates for every word in the English language. Instead, the AI learns them through trial and error during its training phase. It starts completely clueless, assigning random numbers to every word.

• As the model reads billions of sentences, it notices patterns. For example, it sees "The cat sat on the mat" and "The feline sat on the mat."
• Because "cat" and "feline" show up in nearly identical contexts, the model's internal math slowly nudges their coordinates closer together on our high-dimensional map.
• Simultaneously, it pushes completely unrelated words (like "bulldozer") far away.
• By the time training is finished, these semantic relationships are mathematically locked into place.

Phase 2: The Instant Lookup

When you chat with an LLM today, it isn't doing intense calculus to figure out the meaning of your words from scratch. It uses a massive, pre-calculated lookup table:

1. Tokenization: First, your text is chopped into small pieces (tokens). For instance, the word "unbelievable" might be split into [un, believ, able].
2. The Matrix: Every token is assigned a simple ID number (like Token #4892). The AI goes to its giant internal spreadsheet, looks at row #4892, and simply copies down that row's 1,536 coordinates.
3. Contextual Twisting: Modern models (using the Transformer architecture) then pass these baseline coordinates through Attention layers. This allows the surrounding words to slightly shift the exact meaning—meaning the coordinates for "bank" will shift depending on whether you're talking about a river bank or a financial bank.
4. Pooling (For Sentences): To turn a whole sentence into a single vector, the AI uses a mathematical shortcut (like averaging) to blend all the final word vectors into one single list of numbers. This is how entire documents can be compressed into a single coordinate!

Where Do These Numbers Actually Live?

Because embeddings are just massive arrays of floating-point numbers, storing them is straightforward but doing it efficiently for millions of items requires some specialized engineering.

Inside the LLM: The baseline embeddings live directly inside the model's weight files (like the massive .safetensors or .gguf files you might download). They act as the foundational dictionary the AI carries around in its brain.

In Vector Databases: When developers build apps that need an AI to search through external data (like a company's internal PDFs or your personal notes), they use a Vector Database (like Pinecone, Qdrant, Milvus, or pgvector). It essentially looks like this:

How Static & Dynamic Embeddings Map Conceptual Meaning for Search & Retrieval

Traditional search engines rely on Lexical Search (like BM25 or SQL LIKE queries), which matches exact keywords. If a user searches for "How do I give back an item?", lexical search fails if your document only contains "Our return policy allows refunds within 30 days" because zero words overlap.

Embeddings solve this by converting text from string tokens into points in a high-dimensional vector space where spatial proximity equals conceptual similarity. However, the way embeddings map conceptual meaning has evolved dramatically from Static Embeddings to Dynamic Contextual Embeddings.

1. Static vs. Dynamic Embeddings: Mapping Conceptual Meaning

Understanding the difference between static and dynamic embeddings is fundamental to designing effective search and retrieval pipelines:

2. How Embeddings Lay the Groundwork for Search & Retrieval

Modern AI search engines and Retrieval-Augmented Generation (RAG) systems leverage dynamic embeddings through a 4-step vector retrieval pipeline:

graph TD
    A["Raw Document Corpus\n(PDFs, Webpages, Code)"] --> B["Text Chunking\n(e.g., 512 Tokens)"]
    B --> C["Dynamic Embedding Model\n(e.g., OpenAI / BGE)"]
    C --> D["Vector Database Index\n(Pinecone / Qdrant / pgvector)"]
    
    E["User Query\n'How do I cancel my order?'"] --> F["Embed Query Vector\n[0.82, -0.14, 0.91...]"]
    F --> G["Cosine Similarity Search\nFind Nearest Vectors"]
    D <--> G
    G --> H["Top-K Relevant Chunks Returned\nPassed to LLM for Answer Generation"]

Here is how conceptual mapping powers each stage of search and retrieval:

1. Conceptual Vectorization (Dense Indexing): When documents are uploaded, a dynamic embedding model maps every text chunk to a high-dimensional vector. Concepts like "canceling an order", "refunding a purchase", and "terminating a subscription" are automatically mapped to neighboring coordinates in vector space.
2. Intent-Based Query Mapping: When a user enters a query, it is embedded using the exact same coordinate system. The query doesn't look for keyword matches; it looks for the closest geometric neighbor.
3. Sub-Millisecond Cosine Nearest Neighbor Search: Using vector indexes like HNSW (Hierarchical Navigable Small World), the Vector DB performs Cosine Similarity comparisons (cos(θ) = (A · B) / (||A|| × ||B||)) across millions of document vectors in under 10ms.
4. Hybrid Search (Combining Sparse & Dense): Modern search pipelines combine Dense Dynamic Embeddings (semantic meaning) with Sparse Lexical Search (BM25 for exact part numbers, SKUs, or error codes) using Reciprocal Rank Fusion (RRF) for maximum retrieval accuracy.

Conclusion

Embeddings translate the messy, nuanced world of human language into cold, hard geometry. By representing words as coordinates in a high-dimensional space and using trigonometry like Cosine Similarity to measure the distance between them, AI can "understand" context, semantics, and relationships with stunning accuracy.