Rag

• Five Places RAG Shows Up in Agentic Systems

  Ask most people what RAG is and they’ll tell you it’s semantic document search. You chunk up a pile of text, embed it, stuff it in a vector database, and pull the relevant bits back at query time. That’s the textbook example, and it’s a good one. But retrieval augmented generation does a lot more work inside agentic systems than “search the docs,” and I think it’s worth walking through where it actually shows up. So let’s talk about it.

  1. High-Precision Semantic Search

  This is the one everybody knows, so let’s get it out of the way first. You take raw text and convert it into high-dimensional vectors, where distance corresponds to conceptual similarity. Store those vectors, index them, and you can look things up by meaning instead of exact keywords.

  The interesting part is how you index them, because the algorithm you pick is a real tradeoff.

  HNSW (Hierarchical Navigable Small World) builds a multi-layer graph. The upper layers have long-distance links for fast routing across the space, and the lower layers have short-distance links for local search. You get low query latency and near-perfect recall. The catch is memory. It wants to keep the raw vectors around, so the footprint gets big.

  IVF-PQ (Inverted File with Product Quantization) goes the other direction. It partitions the vector space into cells using k-means clustering, then compresses the high-dimensional vectors into compact quantized codes. Partition, then squish. That cuts memory consumption dramatically, which makes it a great fit for massive datasets with millions of vectors. The price you pay is recall accuracy, since all that compression throws away detail, and rebuilds get slower when you add new data.

  Neither one is “correct.” You pick based on whether you’re optimizing for recall or for fitting the index in memory.

  2. Tool Selection (RATS)

  Here’s where it gets less obvious. Picture a CLI harness. As your developer toolkit grows, your agent slowly gets “equipped” with dozens or hundreds of possible actions. APIs, database calls, helpers, command executors. At some point you’ve just overloaded the thing with too much stuff.

  Three bad things happen when you do that:

  • Tool space interference (TSI). Overlapping tool descriptions confuse the agent, and it calls the wrong one.
  • Context window saturation. Every tool schema, whether it’s JSON or Markdown, eats thousands of tokens. Pile up enough MCP servers and custom skills and you’re soaking the context window, which drives up cost and latency.
  • The lost-middle problem. Models tend to ignore tools and instructions buried in the middle of a very long prompt.

  Retrieval augmented tool selection fixes this by treating your tools like a corpus. Instead of dumping every schema into the prompt, you retrieve only the handful of tools relevant to the current task. The agent sees a short, sharp menu instead of the entire pantry.

  3. Dynamic Few-Shot Prompting

  Few-shot prompting is a reliable way to enforce formatting constraints like a strict JSON schema, teach reasoning paradigms like chain of thought, or train an agent on error recovery. The problem is that static examples baked into a prompt are a guess. They might not match the task in front of you.

  RAG lets you select the examples at runtime. You curate a database of gold-standard trajectories, each one pairing a specific query or error case with the correct step-by-step reasoning, tool calls, and final output that solved it. When a new task comes in, you search that database using the user’s intent, grab the top few most similar past trajectories, and prepend them to the system instructions.

  So the agent always gets examples that actually resemble what it’s being asked to do, instead of whatever examples you happened to hardcode three weeks ago.

  4. Long-Term Agent Memory

  Work directly with a model and it forgets everything the moment the session closes. Your preferences, your corrections, the choices you already made. Gone. For an agent to be useful, it needs persistent memory across sessions. I’ve written about this before.

  One system here is mem0, which uses a hybrid RAG architecture to persist state. It does asynchronous fact extraction, conflict resolution when new information contradicts old, and grounds the retrieved memories back into the prompt. The retrieval layer is what lets the agent surface the right past fact at the right time instead of replaying the entire history.

  5. Evaluation and Test Harnesses

  Testing AI in production is hard because the outputs aren’t deterministic. So you build evaluation harnesses that run your agent across hundreds of test cases, and RAG turns out to be a quiet workhorse in that loop.

  Two ways it helps:

  • Diffing the test suite. Running every eval on every pull request is slow and expensive. Instead, query a vector index of your test suite using the git diff as the query, and run only the cases relevant to the code you actually touched.
  • Semantic assertions. Exact string matching is useless when you’re verifying something like an agent’s summary. Instead, the harness retrieves historic successful runs and uses vector similarity to ask whether the new output matches the intent and tone of the target, rather than matching it character for character.

  ---

  None of this replaces the document-search version of RAG. It’s the same core trick, embed things, retrieve by similarity, ground the result, pointed at different problems: which tool, which example, which memory, which test. Once you start seeing retrieval as a general-purpose way to feed an agent the relevant slice of a much bigger pile, it shows up everywhere. I’ll probably keep finding more.

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  Sources

  • Yury A. Malkov and Dmitry A. Yashunin, “Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs” (arXiv:1603.09320, 2016) — details on the multi-layer graph architecture and logarithmic complexity of the HNSW index.
  • Hervé Jégou, Matthijs Douze, and Cordelia Schmid, “Product Quantization for Nearest Neighbor Search” (IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011) — explains compressing high-dimensional vectors and combining product quantization with inverted file indexing (IVF-PQ).
  • mem0ai, “mem0: The Memory Layer for Personalized AI” (GitHub) — documentation and codebase for the persistent, self-improving memory layer for AI agents.
  • Mostafa Ibrahim, “Agentic RAG vs Classic RAG: From a Pipeline to a Control Loop” (Towards Data Science, March 2026) — commentary on the shift from static document retrieval to agentic control loops and its associated system failure modes.
  • Microsoft Research, “Tool-space interference in the MCP era: Designing for agent compatibility at scale” — the tool-space interference (TSI) problem from section 2, where overlapping tool descriptions degrade agent tool selection.
  • rewire.it, “Dynamic Tool Allocation for AI Agents (The RATS Pattern)” — the retrieval-augmented tool selection (RATS) pattern from section 2: a router retrieves a relevant subset of tools from a larger catalog.

  Jul 2, 2026 AI Agents Agentic Rag LLM

  Enjoyed this?

• How to Pick an Embedding Model (Without Overthinking It)

  It’s easy to get deep into vector database comparisons, HNSW vs. IVF, pgvector vs. Pinecone, Qdrant vs. Chroma, and completely skip over the thing that actually matters most: the embedding model.

  The way I think about it, the embedding model is the brain of your retrieval system. The vector database is just its filing cabinet. If the model creates poor mathematical representations of your data, no amount of indexing strategy or database performance is going to save you. You’ll get fast, confident, wrong results.

  So let’s talk about how to pick a model.

  Dimensionality: More Isn’t Always Better

  Embeddings are high-dimensional vectors. Common sizes are 384, 768, 1536, or 3072 dimensions. Higher dimensions capture more nuance, but they also mean more storage, more memory, and slower search.

  For a lean, local-first setup, something like all-MiniLM-L6-v2 at 384 dimensions gives you a surprisingly good balance of speed and accuracy. You don’t need 3072 dimensions to search your notes. Save the big vectors for when you actually have a reason.

  Sequence Length: The Silent Data Killer

  Sequence length determines how much text the model can look at to create a single vector. If you’re embedding long technical docs or sprawling Markdown files and your model caps out at 512 tokens, it’s just truncating everything past that point. Your carefully written documentation gets chopped, and the embedding only represents the first few paragraphs.

  Modern long-context embedding models handle 8k to 32k tokens, which lets you embed entire chapters or large code blocks as single semantic units. If your content is longer than a few paragraphs, check this number before anything else.

  Domain Matters More Than You Think

  General-purpose models like OpenAI’s text-embedding-3-small work well across most tasks. They’ve been trained on massive, diverse datasets and they’re solid defaults.

  If you’re searching a codebase or technical documentation, models fine-tuned on programming languages (like voyage-code-2) will outperform the general ones. The same applies to medical or legal text, where domain-specific jargon means the difference between a relevant result and a completely wrong one.

  Check MTEB Before You Commit

  The Massive Text Embedding Benchmark (MTEB) is the industry standard for comparing models. It breaks performance into sub-categories like Retrieval, Summarization, and Clustering. If you’re building RAG, look at the Retrieval scores specifically. A model that ranks well for clustering might be mediocre at retrieval, and vice versa.

  Local vs. API: Pick Your Tradeoff

  This decision is as important as the model itself.

  • Local models (via HuggingFace or Ollama) keep everything offline. Zero per-request costs, full privacy. Something like bge-small-en-v1.5 running locally is perfect for personal knowledge management or anything where your data shouldn’t leave your machine.
  • Hosted APIs (OpenAI, Voyage, Cohere) give you the highest performance and longest context windows without managing GPU infrastructure. Better for enterprise scale where you’re willing to trade privacy and recurring costs for accuracy.

  Local models make sense for personal projects and hosted APIs make sense when the scale demands it. There’s no universal right answer, but there is a wrong one: picking a deployment model without thinking about where your data lives.

  ---

  The vector database conversation is important, but it’s second in line to getting the embedding model right first. Everything downstream depends on it.

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  Apr 29, 2026 AI Rag Embeddings Vector-databases

  Enjoyed this?

• NotebookLM Is Just RAG With a Nice UI

  I’ve been watching AI YouTubers recommend NotebookLM integrations that involve authenticating your Claude instance with some random skill they built. “Download my thing, hook it up, trust me bro.” No details on how it works under the hood. No mention of why piping your credentials through someone else’s code might be a terrible idea. Let’s just gloss over that, I guess.

  So here we are. Let me explain what NotebookLM actually is, because once you understand RAG, the magic disappears pretty quickly.

  What Is RAG?

  RAG stands for Retrieval Augmented Generation. It’s an AI framework that improves LLM accuracy by retrieving data from trusted sources before generating a response.

  The LLM provides the reasoning and token generation. RAG provides specific, trusted context. Combining the two gives you general reasoning grounded in your actual data instead of whatever the model has or hallucinated from its training set.

  The core pipeline looks like this:

  1. Take your trusted data (docs, PDFs, YouTube transcripts, whatever)
  2. Chunk it into pieces
  3. Create vector embeddings from those chunks
  4. Store the vectors in a database
  5. When you ask a question, embed the question into the same vector space
  6. Find the most similar chunks
  7. Feed those chunks into the LLM as context alongside your question

  That’s it. That’s NotebookLM. Steps 1 through 6 are the retrieval half. Step 7 is where the LLM synthesizes an answer. The nice UI on top doesn’t change what’s happening underneath.

  I Accidentally Built Half of It?

  I was interested in the semantic embeddings portion of this pipeline and ended up building something I called Semantic Docs. It handles the retrieval half, steps 1 through 6.

  You point it at a knowledge base, internal company docs, research papers, whatever you’re interested in. It chunks the content, creates vector embeddings, and stores them in a database. When you search, it creates a new embedding from your query, finds the most similar chunks, and returns those as search results.

  The difference between Semantic Docs and NotebookLM is that last step. Semantic Docs gives you the relevant files and passages. It says “here’s where the answers live, go read it.” It doesn’t pipe everything through an LLM to generate a synthesized response. This is a choice, a deliberate choice, not a missing feature.

  Why No Official API Is a Problem

  NotebookLM doesn’t have an official API. People have reverse-engineered how it works, which means every integration you see is built on undocumented behavior that could break at any time. The AI YouTubers recommending these workflows are essentially saying “trust this unofficial thing with your data and credentials.” That should make you uncomfortable.

  If you understand RAG, you can build the parts you actually need. The retrieval half is genuinely useful on its own, and you control the whole pipeline. No third-party authentication. No undocumented APIs. No wondering what happens to your data.

  I’ll probably write more about RAG in the future. It’s a good topic and there’s a lot of noise to cut through. For now, just know that the next time someone tells you NotebookLM is magic, it’s really just vector search with a chat interface on top.

  If you’re a developer, I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week. Or you can find me on Mastodon at @[email protected].

  Apr 20, 2026 AI Programming Rag Notebooklm

  Enjoyed this?