If you’ve built a RAG pipeline, a semantic search box, or an AI agent with vector-backed memory, you’ve used embeddings. You probably called a Mistral or Gemini embedding endpoint, took the array of floats that came back, dumped it into pgvector or Qdrant, and moved on with your day. I’ve done this a lot in the last few months.
\nBut what are those numbers? And why do we need a whole separate model to make them? Why can’t I just ask GPT-4 or Claude, which can generate remarkably fluent language, to hand me a vector?
\nPart of the answer comes down to a fork in the road inside the Transformer architecture. The rest comes down to how a model is trained and how its output gets turned into one vector. Let me walk you through it.
\nStrip away the jargon and an embedding is a set of coordinates. Picture a giant map of meaning. Similar ideas sit close together, unrelated ideas sit far apart.
\n[0.82, 0.19, -0.45].[0.79, 0.22, -0.41].When we turn text into these long lists of numbers, we let a computer measure meaning with plain geometry. Depending on the model, a vector may have hundreds or thousands of dimensions. OpenAI models, for example, have used 1,536 or 3,072, while Mistral Embed uses 1,024. A common comparison tool is cosine similarity. The machine has no idea what “React” means. It just notices that the angle between the React vector and the Vue vector is tiny, so they must be related. That’s the whole trick.
\nThe original 2017 Transformer had two halves, and modern models tend to pick one and run with it. This comparison is simplified, but it captures the two families that matter here.
\n THE ORIGINAL TRANSFORMER (Vaswani et al., 2017)\n |\n +-----------------------+-----------------------+\n v v\n ENCODER-ONLY (BERT, DeBERTa) DECODER-ONLY (GPT-style, Llama)\n - Bidirectional attention - Causal (masked) attention\n - Every token sees surrounding text - Each position sees current/earlier tokens\n - Often used for embeddings - Predicts the next token\n - Strong at classification - Strong at generation, reasoning, chat\nThat architecture helps explain why a general-purpose chat model is not automatically a good embedding model.
\nGPT-style and Llama models use decoder-only architectures. Their job is to predict the next token based on the tokens so far. Decoders use causal masking to prevent a position from attending to later tokens. The hidden state at token #3 can use tokens #1 through #3 to predict token #4, but it cannot use token #4 itself.
\nThat’s ideal for generating text one token at a time, but the raw hidden states are not automatically good whole-passage embeddings. Early token states cannot incorporate later context, although the final token state can represent the entire preceding input. With the right pooling and retrieval training, decoder-based models can still produce strong embeddings.
\nMany dedicated embedding models use BERT-style bidirectional attention. They do not use a causal mask between ordinary content tokens during embedding inference, so each token can incorporate context from both before and after it. The system then turns those contextual token states into one vector using mean pooling, a special token, a learned pooling layer, or another strategy.
\nEncoder architecture alone is not enough, though. A useful embedding space also depends on the pooling method and training objective. One model may start out optimized to predict the next token, while another is trained so whole passages with related meanings end up near each other.
\nYou can ask GPT-4 or Claude to print a list of numbers, but the response is still generated text. A number like 0.82 is emitted as tokens representing that number. The API is not handing you one of the model’s internal vectors.
That distinction matters because useful embeddings must share a stable coordinate system. If two passages mean similar things, the model needs to place them near each other, request after request. The vectors need a consistent number of dimensions, a defined pooling strategy, and training that makes distance meaningful.
\nA chat model’s normal output gives you none of those guarantees. It predicts the next token and sends the generated tokens back. An embedding endpoint returns a fixed-length numerical representation designed for similarity, retrieval, and related tasks.
\nThe chat model is still using vectors internally. Those vectors simply are not what you receive from its chat API. Anthropic does not offer its own embedding model at all, and its documentation points developers to a separate embedding provider.
\nYou can take an open-weight decoder model, extract its hidden states, add a pooling strategy, and train it for retrieval. Decoder-based embedding models do exactly that. But at that point, you have turned the decoder into an embedding model. You are no longer just asking a chat model to give you a vector.
\nSo the next time you reach for an embedding, use the model built for it. Your chat model is built for another task entirely.
\n\n\n", "date_published": "2026-07-16T10:00:00-05:00", "url": "https://llbbl.blog/2026/07/16/what-are-embeddings-really-and.html", "tags": ["AI","Rag","Embeddings","Machine learning"] } ] }I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @logan@llbbl.blog.
\n