Embeddings
-
What Are Embeddings, Really? (And Why You Can't Just Use GPT-4 to Make Them)
If you’ve built a RAG pipeline, a semantic search box, or an AI agent with vector-backed memory, you’ve used embeddings. You probably called a Mistral or Gemini embedding endpoint, took the array of floats that came back, dumped it into pgvector or Qdrant, and moved on with your day. I’ve done this a lot in the last few months.
But what are those numbers? And why do we need a whole separate model to make them? Why can’t I just ask GPT-4 or Claude, which can generate remarkably fluent language, to hand me a vector?
Part of the answer comes down to a fork in the road inside the Transformer architecture. The rest comes down to how a model is trained and how its output gets turned into one vector. Let me walk you through it.
Embeddings Are Just Coordinates
Strip away the jargon and an embedding is a set of coordinates. Picture a giant map of meaning. Similar ideas sit close together, unrelated ideas sit far apart.
- React and Vue land near each other, say around
[0.82, 0.19, -0.45]. - Frontend sits right next door at
[0.79, 0.22, -0.41]. - Python and Sourdough fermentation are off on different continents entirely.
When we turn text into these long lists of numbers, we let a computer measure meaning with plain geometry. Depending on the model, a vector may have hundreds or thousands of dimensions. OpenAI models, for example, have used 1,536 or 3,072, while Mistral Embed uses 1,024. A common comparison tool is cosine similarity. The machine has no idea what “React” means. It just notices that the angle between the React vector and the Vue vector is tiny, so they must be related. That’s the whole trick.
The Fork: Encoders vs. Decoders
The original 2017 Transformer had two halves, and modern models tend to pick one and run with it. This comparison is simplified, but it captures the two families that matter here.
THE ORIGINAL TRANSFORMER (Vaswani et al., 2017) | +-----------------------+-----------------------+ v v ENCODER-ONLY (BERT, DeBERTa) DECODER-ONLY (GPT-style, Llama) - Bidirectional attention - Causal (masked) attention - Every token sees surrounding text - Each position sees current/earlier tokens - Often used for embeddings - Predicts the next token - Strong at classification - Strong at generation, reasoning, chatThat architecture helps explain why a general-purpose chat model is not automatically a good embedding model.
Chat LLMs commonly use decoder-style attention
GPT-style and Llama models use decoder-only architectures. Their job is to predict the next token based on the tokens so far. Decoders use causal masking to prevent a position from attending to later tokens. The hidden state at token #3 can use tokens #1 through #3 to predict token #4, but it cannot use token #4 itself.
That’s ideal for generating text one token at a time, but the raw hidden states are not automatically good whole-passage embeddings. Early token states cannot incorporate later context, although the final token state can represent the entire preceding input. With the right pooling and retrieval training, decoder-based models can still produce strong embeddings.
Many embedding models use bidirectional encoders
Many dedicated embedding models use BERT-style bidirectional attention. They do not use a causal mask between ordinary content tokens during embedding inference, so each token can incorporate context from both before and after it. The system then turns those contextual token states into one vector using mean pooling, a special token, a learned pooling layer, or another strategy.
Encoder architecture alone is not enough, though. A useful embedding space also depends on the pooling method and training objective. One model may start out optimized to predict the next token, while another is trained so whole passages with related meanings end up near each other.
Why You Can’t Just Ask a Chat Model
You can ask GPT-4 or Claude to print a list of numbers, but the response is still generated text. A number like
0.82is emitted as tokens representing that number. The API is not handing you one of the model’s internal vectors.That distinction matters because useful embeddings must share a stable coordinate system. If two passages mean similar things, the model needs to place them near each other, request after request. The vectors need a consistent number of dimensions, a defined pooling strategy, and training that makes distance meaningful.
A chat model’s normal output gives you none of those guarantees. It predicts the next token and sends the generated tokens back. An embedding endpoint returns a fixed-length numerical representation designed for similarity, retrieval, and related tasks.
The chat model is still using vectors internally. Those vectors simply are not what you receive from its chat API. Anthropic does not offer its own embedding model at all, and its documentation points developers to a separate embedding provider.
You can take an open-weight decoder model, extract its hidden states, add a pooling strategy, and train it for retrieval. Decoder-based embedding models do exactly that. But at that point, you have turned the decoder into an embedding model. You are no longer just asking a chat model to give you a vector.
So the next time you reach for an embedding, use the model built for it. Your chat model is built for another task entirely.
Sources
- Vaswani et al. (2017), Attention Is All You Need — The paper that introduced the attention-based Transformer encoder-decoder architecture.
- Devlin et al. (2018), BERT — How bidirectional Transformer encoders build contextual token representations.
- Reimers and Gurevych (2019), Sentence-BERT — How pooling and task-specific training turn BERT into useful sentence embeddings.
- Wang et al. (2022), E5 — An example of weakly supervised contrastive training for general-purpose text embeddings.
- Wang et al. (2024), Improving Text Embeddings with Large Language Models — Evidence that decoder-only models can become strong embedding models with retrieval-specific training.
- Sentence Transformers Loss Overview — The range of contrastive, ranking, regression, distillation, and autoencoding objectives used to train embedding models.
- OpenAI Embedding Models — Examples of 1,536- and 3,072-dimensional embeddings.
- Mistral Embeddings Guide — Mistral Embed’s 1,024-dimensional output and retrieval use cases.
- Google Gemini API Embeddings Guide — How asymmetric query vs. document task types work in practice.
- Anthropic Embeddings Guide — Anthropic’s documentation that Claude does not provide an Anthropic embedding model.
- Hugging Face MTEB Leaderboard — Standard benchmark for comparing embedding models head to head.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
- React and Vue land near each other, say around
-
Embeddings Are Cheap Enough for Personal Wikis Now
My Obsidian vault is my main second brain: around 1,800 Markdown notes. Lately I’ve been less interested in what to put in it and more interested in a different question: what useful tools could I build on top of it?
A few ideas came to mind. Could I do better than Obsidian’s built-in search for the times I remember the idea but not the exact words I used? Could I eventually wrap the whole thing in my own little plugin? Both sound like fun future projects. But the one I wanted to tackle first was better backlinking: getting the system to find related notes and suggest links, so the graph helps maintain its own structure.
So I did what any reasonable person would do. I turned it into a small experiment.
The task was deliberately concrete: given an ordinary note, suggest the topic notes it should link up to. In Obsidian terms, that means adding a frontmatter field like this:
--- up: - "[[TypeScript]]" - "[[Agentic Harness Landscape]]" ---Call them parent backlinks, MOC links, or knowledge graph edges. The label matters less than the shape: each note can have zero, one, or several parents, and the system should never force every note into exactly one folder.
The Setup
My vault had about 1,800 notes, 27 topic hubs (the pages other notes already linked to, like “TypeScript” or “Tailwind CSS”), and a hand-labeled gold set of 38 notes. No vector database. No pre-existing
up:links.Every approach had to emit the same report schema, so I could compare methods mechanically instead of eyeballing a few good examples. And nothing wrote to the vault by default. Applying links was a separate, explicit step. That “report first, apply later” rule is the difference between a useful automation and a spooky one.
The Baseline: Full-Text Search
I started with classic keyword retrieval: index the topic notes, turn each child note into a query, score with Postgres full-text search plus trigram similarity. It’s a good baseline because it’s deterministic, cheap, and explainable.
But it has an obvious weakness in a personal wiki. A note can be about “serverless container hosting” without ever saying “Azure Container Apps.” A note about pyenv and virtual environments might belong under “Python” without naming it. FTS is great when the words line up. Personal notes often don’t.
The Embeddings Version
The embeddings approach was simple: build a text representation for every topic note and every child note, embed both with Google’s Gemini Embedding 2 model, normalize, take the cosine similarity, and keep the top matches above a threshold.
The text prep mattered more than anything else. For child notes I used the title plus opening body. For topic notes, the title, aliases, summary, and first section, stripping wiki-link syntax. MOC pages are often long lists of
[[Some Note]]links, and that’s worse input than a short prose description of what the topic actually means.I did not need vector infrastructure. Vectors were stored as plain arrays in Postgres, cosine similarity computed in Python. With 27 topic vectors and 1,800 child notes, brute force ran in about 6 seconds, and embedding the whole vault plus every test run cost about 30 cents. At this scale, architecture matters less than good text prep.
The Numbers
On the 38-note gold set:
Approach Recall@1 Recall@3 MRR F1 Full-text search 0.519 0.615 0.609 0.424 Embeddings 0.923 0.962 1.000 0.622 Of the 30 intended links across the gold set, embeddings found 28 and missed 2. Full-text search found 18 and missed 12.
The precision wasn’t amazing, and that’s fine. The goal of the first pass is high-recall suggestion, not automatic writing. It produces candidates for review. A later LLM reranker or a human can tighten precision.
One detail surprised me. I tested two query framings: one worded as “search result,” one as “semantic similarity.” Ranking was identical, but the similarity framing had worse precision because it surfaced loosely related matches. For “which topic should this note be filed under?”, the task behaves more like search than duplicate detection. Semantic search and semantic similarity are not the same product requirement.
Thresholds and the Gold Set
My first threshold was too low. Everything looked related to everything. Raising cosine similarity to around 0.70 cut the output to a reviewable set: of 1,805 notes scanned, 682 got at least one suggestion, averaging about 2.4 parents each. The rest got nothing, which is correct. Plenty of notes are fragments, logs, and one-offs. A good backlink suggester should be comfortable saying nothing.
You don’t find that threshold on a model card. You find it on your own notes.
Which brings me to the most valuable artifact in the whole project: the gold set. It wasn’t the embedding index. It was 38 hand-labeled notes, including negative examples with no expected links. Without it, I’d have judged by vibes, and that’s dangerous with embeddings because the good examples look magical. You need the misses in the same report or you’ll overestimate the system. Thirty to fifty labeled examples is enough to expose bad assumptions.
What I’d Actually Do
If I were adding this to another wiki, the order would be:
- Pick 20 to 100 topic notes explicitly, by folder, tag, or inbound link count. Don’t make every note a possible parent.
- Label 30 to 50 examples, negatives included.
- Build an FTS baseline first. If embeddings can’t beat it on your labeled data, your text prep is wrong.
- Build the embedding baseline and compare.
- Tune text prep and threshold.
- Emit a report before any writes.
- Only then consider LLM reranking or a real vector database.
That order keeps you honest. You’ll know whether embeddings help your wiki before you spend time getting Qdrant setup.
Personal wikis are small enough that we can stop treating semantic search as an enterprise architecture problem. The interesting work isn’t the vector database. It’s deciding what “related” means in your own graph, measuring it against your own notes, and building the tool so it suggests, explains, and writes only when asked. Use semantic search to propose structure. Use reports and gold sets to keep it honest. Let the human knowledge base stay human.
I’ll probably keep poking at it.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Building a Linter for Your Obsidian Vault
In software we don’t trust ourselves to keep a codebase clean by hand. We run linters. They catch the dead code, the unused import, the function nobody calls anymore, the link that points at a file that got deleted three commits ago. We’ve decided that rot is inevitable and that a machine should hunt for it on every save.
Our note vaults get no such treatment. They rot exactly the same way, orphan notes with no links in or out, dead
[[wikilinks]]pointing at notes you renamed, half-distilled drafts that have sat untouched for months. The difference is nobody’s running a linter on them. So let’s write one.Folders are a trap. Links are the currency.
The instinct, when a vault gets messy, is to reorganize the folders. Resist it. Folders force a note to live in exactly one place, which is a lie, because most ideas belong to several contexts at once. A note on caching strategy is relevant to a performance project, a systems-design reference, and a half-formed blog post, all at the same time. A folder makes you pick one and bury it.
The health of a vault lives in its links, not its hierarchy. A well-linked note is reachable from a dozen directions. An orphan, a note with zero links in or out, is functionally invisible. You will never stumble back into it. It’s dead the moment you save it. So the single most useful thing a vault linter can report is: which notes are orphans, and which links are broken.
Step one: parse the markdown, not just grep it
You could grep for
[[and call it a day, but you’ll get fooled by links inside code blocks, escaped brackets, and frontmatter. Parse it properly.Two passes per file. First, pull the YAML frontmatter off the top with a library like
python-frontmatter, which hands you the metadata as a dict and the body as a string. That’s where yourstatus,tags, andaliaseslive. Second, run the body through a real markdown parser such asmarkdown-it-pyand walk the token stream, collecting link tokens while ignoring anything inside a fenced code block.Now you have a clean model of the vault: a dict of every note keyed by filename, each with its frontmatter and its outbound links. From there the checks are short:
- Broken links. For every outbound link, confirm the target file exists. If it doesn’t, the note was renamed or deleted. Flag it.
- Orphans. Build the reverse index of inbound links. Any note with no inbound and no outbound links is an orphan. Flag it.
- Stale drafts. Any note still at
status: rawwhose modified time is older than thirty days. Flag it.
That’s a useful linter already, and it’s maybe sixty lines of Python. It runs in a second over a few thousand notes.
Step two: the part grep can’t do
Broken-link detection is mechanical. The interesting failure is the link that should exist and doesn’t. Two notes that are clearly about the same idea, written six months apart, that have no idea the other one exists. No string match will find those, because they don’t share words. They share meaning.
This is where a local embedding model earns its place. Run every note body through something like
sentence-transformersto get a vector per note, then compute pairwise cosine similarity. Any pair that scores above a threshold but has no link between them is a suggested cross-link. The linter doesn’t create the link, it just surfaces the candidate: “these two notes are conceptually close and unconnected, did you mean to link them?”Keep it local. The whole point of a vault is that it’s yours, and you don’t want to ship every private note to an API to find out two of them rhyme. A small embedding model on your own machine handles a personal vault without breaking a sweat, and the vectors never leave the laptop. If you’ve read how vector similarity drives retrieval, this is the same trick pointed at your own notes instead of a document corpus.
Run it like a linter, not a project
The mistake would be to build this as a grand one-time cleanup, run it once, fix everything, and never touch it again. The vault will just rot back. Treat it like the linter it is. Wire it into a weekly job, or a git pre-commit hook if your vault is a repo, and have it print a short report: three broken links, eight orphans, five suggested cross-links. You spend ten minutes acting on the list and the vault stays healthy on its own.
A note vault is a codebase. It accumulates the same entropy, drifts the same way, and benefits from the same discipline we already apply to every other pile of text files we own. We just never thought to point the tools at it. Point them.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Sources
- Andy Matuschak, “Evergreen notes should be densely linked” — the argument that a note’s value comes from its links, and that organizing by hierarchy fights against that.
- Eric Holscher, “python-frontmatter” (GitHub) — library for splitting YAML frontmatter from a markdown body, used for the metadata-parsing pass.
- “markdown-it-py” (GitHub) — a CommonMark parser that exposes a token stream, used here to extract links while skipping code blocks.
- “Sentence Transformers” — framework for generating local sentence and document embeddings, used to flag conceptually similar but unlinked notes.
Python Embeddings Second brain Note-taking Knowledge management
-
How to Pick an Embedding Model (Without Overthinking It)
It’s easy to get deep into vector database comparisons, HNSW vs. IVF, pgvector vs. Pinecone, Qdrant vs. Chroma, and completely skip over the thing that actually matters most: the embedding model.
The way I think about it, the embedding model is the brain of your retrieval system. The vector database is just its filing cabinet. If the model creates poor mathematical representations of your data, no amount of indexing strategy or database performance is going to save you. You’ll get fast, confident, wrong results.
So let’s talk about how to pick a model.
Dimensionality: More Isn’t Always Better
Embeddings are high-dimensional vectors. Common sizes are 384, 768, 1536, or 3072 dimensions. Higher dimensions capture more nuance, but they also mean more storage, more memory, and slower search.
For a lean, local-first setup, something like
all-MiniLM-L6-v2at 384 dimensions gives you a surprisingly good balance of speed and accuracy. You don’t need 3072 dimensions to search your notes. Save the big vectors for when you actually have a reason.Sequence Length: The Silent Data Killer
Sequence length determines how much text the model can look at to create a single vector. If you’re embedding long technical docs or sprawling Markdown files and your model caps out at 512 tokens, it’s just truncating everything past that point. Your carefully written documentation gets chopped, and the embedding only represents the first few paragraphs.
Modern long-context embedding models handle 8k to 32k tokens, which lets you embed entire chapters or large code blocks as single semantic units. If your content is longer than a few paragraphs, check this number before anything else.
Domain Matters More Than You Think
General-purpose models like OpenAI’s
text-embedding-3-smallwork well across most tasks. They’ve been trained on massive, diverse datasets and they’re solid defaults.If you’re searching a codebase or technical documentation, models fine-tuned on programming languages (like
voyage-code-2) will outperform the general ones. The same applies to medical or legal text, where domain-specific jargon means the difference between a relevant result and a completely wrong one.Check MTEB Before You Commit
The Massive Text Embedding Benchmark (MTEB) is the industry standard for comparing models. It breaks performance into sub-categories like Retrieval, Summarization, and Clustering. If you’re building RAG, look at the Retrieval scores specifically. A model that ranks well for clustering might be mediocre at retrieval, and vice versa.
Local vs. API: Pick Your Tradeoff
This decision is as important as the model itself.
- Local models (via HuggingFace or Ollama) keep everything offline. Zero per-request costs, full privacy. Something like
bge-small-en-v1.5running locally is perfect for personal knowledge management or anything where your data shouldn’t leave your machine. - Hosted APIs (OpenAI, Voyage, Cohere) give you the highest performance and longest context windows without managing GPU infrastructure. Better for enterprise scale where you’re willing to trade privacy and recurring costs for accuracy.
Local models make sense for personal projects and hosted APIs make sense when the scale demands it. There’s no universal right answer, but there is a wrong one: picking a deployment model without thinking about where your data lives.
The vector database conversation is important, but it’s second in line to getting the embedding model right first. Everything downstream depends on it.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
- Local models (via HuggingFace or Ollama) keep everything offline. Zero per-request costs, full privacy. Something like