LLM
-
Amnesia-Free Agents: GBrain and the LLM Wiki Idea
Every agent I work with has amnesia.
Open a new terminal session, spin up a subagent, start a fresh chat, and the thing forgets everything. Your project guidelines, your coding style, your database schemas, the meeting where you decided why the schema looks like that. Gone. So you do one of two things: stuff a few thousand tokens into a bloated system prompt, or hand-maintain fragile context files like
CLAUDE.md. I’ve done both. Neither feels good.So when Y Combinator CEO Garry Tan open-sourced GBrain on April 5, 2026, a markdown-first memory layer, I wanted to understand what it does and whether it’s worth the setup.
What GBrain actually is
GBrain is a Postgres-native knowledge brain that acts as a persistent retrieval layer for your coding agents. The interesting part isn’t that it stores stuff, it’s how opinionated it is about the shape of that storage.
Three ideas do the heavy lifting:
- Compiled truth vs. an append-only timeline. Raw daily logs, Slack messages, and emails go in one bucket. The curated “current facts” (contacts, active projects, decisions) live as clean Markdown in another. One is history, the other is truth.
- A self-wiring knowledge graph. It parses your plain Markdown and pulls out relationships (
works_at,invested_in,founded) using local grammars. No LLM API calls to build the map. - The Dream Cycle. An overnight cron job that crawls your daily logs, reconciles contradictions, checks citations, and rebuilds the search index while you sleep.
That last one worth a callout: Memory that maintains itself.
The category error everyone makes
People keep comparing GBrain to runtime agents like OpenClaw or Hermes, and that comparison doesn’t hold up. They’re different layers entirely.
GBrain isn’t an execution engine. It’s closer to a self-writing wiki.
┌────────────────────────────────────────────────┐ │ AGENT RUNTIMES │ │ (OpenClaw, Hermes, Claude Code) │ │ runs code, spawns subagents, hits the shell │ └─────────────────────────┬────────────────────────┘ │ reads & writes via MCP ▼ ┌────────────────────────────────────────────────┐ │ THE LLM WIKI │ │ (GBrain, OpenWiki) │ │ compiles raw history into clean markdown │ └────────────────────────────────────────────────┘The runtime is the hands and eyes. It sandboxes code, runs terminal commands, touches the file system. The wiki is the long-term memory. You wire them together over the Model Context Protocol (MCP), and when the agent needs context, it doesn’t parse your raw email history. It queries the wiki and gets back synthesized, cited facts.
GBrain isn’t the only thing playing this role. OpenWiki, from the LangChain folks, uses the same trick to compile big source repos into navigable markdown. Same pattern, different input.
The payoff: your agent stays lightweight and task-focused, and your knowledge lives in a clean, git-versioned wiki instead of leaking into a system prompt.
What people actually run into
The concept is great. The reality has some sharp edges, and the community forums are honest about them.
It needs babysitting. GBrain moves fast, which means frequent schema updates and migrations. You’ll find yourself running things like
gbrain doctor --remediateto fix database drift. It is not set-it-and-forget-it.Tool sprawl adds up. To feed the Dream Cycle good data, people bolt on voice transcribers, contact enrichers, email scrapers. Most of those are paid SaaS. String enough together and you’ve built yourself a subscription problem.
Setup is a project. A Dockerized Postgres with
pgvector, system keyring wrangling (macOS Keychain or D-Bus), cron schedulers. Budget anywhere from 30 minutes to a lost afternoon.Should you bother?
If you’re juggling complex work across a bunch of codebases, and you want an agent that knows your past architecture decisions and updates its own docs overnight, the GBrain + OpenClaw setup earns its keep. That’s a real moat.
If you just want a lightweight assistant that remembers a few things, you’re better off with plain
MEMORY.mdfiles or a hosted API like Mem0 until the local graph-RAG tooling settles down. I’m somewhere in the middle right now, so I’ll watching it to see where it goes.Amnesia is the default. It doesn’t have to be.
Sources
- GBrain repository
- OpenClaw agent framework
- Hermes Agent (Nous Research)
- /r/openclaw community threads
- Hacker News
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Four Ways to Build Your Own Agentic Harness
The model is the commodity. You rent it. Everything else, the loop, the tools, the state, the permissions, the memory, the orchestration, is the harness. And the harness is the part you get to own.
So the real question, when people say they want to “build their own agent,” isn’t which model. It’s how much of the scaffolding you want to inherit versus how much you want to build yourself. That’s the whole decision. Where you land on that spectrum gives you, more or less, four approaches.
A: Build on top of a full harness
This is the composition-heavy approach, and it’s the one I’ve mostly done. You take a fully featured harness like Claude Code or opencode (and I think a few of the newer agentic IDEs support this now too), and you extend it. Sub-agents, skills, slash commands. You can call these things remotely with CLI commands and wire your own behavior around them.
There are two flavors here. You can build your harness inside the existing one, or you can build it around the existing one. Either way, you own all the agent definitions, the skills, the model choices. But you inherit a lot too: which tools the agent can call, the permission model, all the safety controls. You don’t get to change those, you just get to use them.
And honestly? That’s the appeal. You don’t start from scratch. Somebody already solved the boring, dangerous parts. You show up and build the part you care about.
B: Self-host a fully open harness
This one looks a lot like A, but with one important difference: the underlying harness is open. You’re self-hosting and extending something fully open-source, then polishing and tailoring it to your needs.
The trade is control. In approach A, the tools and permissions and controls are handed to you and you live with them. Here, the entire stack is yours to crack open. If the permission model annoys you, you change it. If you want a tool to behave differently at the loop level, you can reach in and do that. You’re still not building from nothing, but nothing is off-limits either.
C: Assemble from primitives
Now we’re getting low. With this approach you build your harness up from very minimal pieces. Think of it like a full-stack framework in the web world, except for agents. Something like pydantic-ai’s SDK, or pulling aider in as a library rather than running it as a tool.
You don’t start from absolute zero, but you’re close. You define the agent loop yourself. You register the tools. You add the features you need, one at a time. You’re scaffolding basically everything, and all you’re inheriting is a handful of core primitives that you get to shape into whatever you want.
This is the approach for people who have opinions and want to express all of them. It’s more work. It’s also the most yours.
D: An agentic harness framework
The last one is a different animal, and I’ll admit it took me a second to see why it’s its own category. These are orchestration frameworks like LangGraph or Letta (formerly MemGPT). They’re not coding CLIs. They’re SDKs for building a custom agent application: graphs, state machines, first-class memory.
The distinction that finally clicked for me is what you’re building. With A through C, you’re mostly building a personal dev tool, something that helps you write code. With D, the harness is the product. You’re shipping it. A domain agent, a service, a customer-facing thing that happens to be agentic under the hood. The orchestration framework is what you reach for when the agent isn’t your tool, it’s your deliverable.
So where do you land?
There’s no correct answer here, just a trade you’re making on purpose. The more you inherit, the faster you move and the less you control. The more you build, the more it’s yours and the more of the boring, dangerous plumbing you’re now on the hook for.
For most of what I do, A is the sweet spot. Building around and within an existing harness is just so much easier than starting cold, and I’d rather spend my time on the agent definitions than on reinventing a permission model. But if I were shipping an agent as a product instead of a tool, I’d be over in D without thinking twice.
Figure out which thing you’re building first. The approach falls out of that.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Sources
- Anthropic, “Claude Code” (GitHub) — documentation and repository for the terminal-based agentic coding tool that reads codebases, runs commands, and integrates MCP servers.
- Aider AI, “Aider” — official documentation and pair-programming guides for the open-source, git-native AI coding assistant.
- Anomaly Co., “OpenCode” — official homepage and repository for the open-source terminal, desktop, and IDE-based AI coding agent.
- Pydantic, “Pydantic AI” — official documentation and API references for the type-safe Python agent framework designed for structured agent trajectories.
- Letta AI, “Letta” (GitHub) — repository for the stateful agent runtime (formerly MemGPT) that manages persistent memory tiers and agentic state machines.
-
Five Places RAG Shows Up in Agentic Systems
Ask most people what RAG is and they’ll tell you it’s semantic document search. You chunk up a pile of text, embed it, stuff it in a vector database, and pull the relevant bits back at query time. That’s the textbook example, and it’s a good one. But retrieval augmented generation does a lot more work inside agentic systems than “search the docs,” and I think it’s worth walking through where it actually shows up. So let’s talk about it.
1. High-Precision Semantic Search
This is the one everybody knows, so let’s get it out of the way first. You take raw text and convert it into high-dimensional vectors, where distance corresponds to conceptual similarity. Store those vectors, index them, and you can look things up by meaning instead of exact keywords.
The interesting part is how you index them, because the algorithm you pick is a real tradeoff.
HNSW (Hierarchical Navigable Small World) builds a multi-layer graph. The upper layers have long-distance links for fast routing across the space, and the lower layers have short-distance links for local search. You get low query latency and near-perfect recall. The catch is memory. It wants to keep the raw vectors around, so the footprint gets big.
IVF-PQ (Inverted File with Product Quantization) goes the other direction. It partitions the vector space into cells using k-means clustering, then compresses the high-dimensional vectors into compact quantized codes. Partition, then squish. That cuts memory consumption dramatically, which makes it a great fit for massive datasets with millions of vectors. The price you pay is recall accuracy, since all that compression throws away detail, and rebuilds get slower when you add new data.
Neither one is “correct.” You pick based on whether you’re optimizing for recall or for fitting the index in memory.
2. Tool Selection (RATS)
Here’s where it gets less obvious. Picture a CLI harness. As your developer toolkit grows, your agent slowly gets “equipped” with dozens or hundreds of possible actions. APIs, database calls, helpers, command executors. At some point you’ve just overloaded the thing with too much stuff.
Three bad things happen when you do that:
- Tool space interference (TSI). Overlapping tool descriptions confuse the agent, and it calls the wrong one.
- Context window saturation. Every tool schema, whether it’s JSON or Markdown, eats thousands of tokens. Pile up enough MCP servers and custom skills and you’re soaking the context window, which drives up cost and latency.
- The lost-middle problem. Models tend to ignore tools and instructions buried in the middle of a very long prompt.
Retrieval augmented tool selection fixes this by treating your tools like a corpus. Instead of dumping every schema into the prompt, you retrieve only the handful of tools relevant to the current task. The agent sees a short, sharp menu instead of the entire pantry.
3. Dynamic Few-Shot Prompting
Few-shot prompting is a reliable way to enforce formatting constraints like a strict JSON schema, teach reasoning paradigms like chain of thought, or train an agent on error recovery. The problem is that static examples baked into a prompt are a guess. They might not match the task in front of you.
RAG lets you select the examples at runtime. You curate a database of gold-standard trajectories, each one pairing a specific query or error case with the correct step-by-step reasoning, tool calls, and final output that solved it. When a new task comes in, you search that database using the user’s intent, grab the top few most similar past trajectories, and prepend them to the system instructions.
So the agent always gets examples that actually resemble what it’s being asked to do, instead of whatever examples you happened to hardcode three weeks ago.
4. Long-Term Agent Memory
Work directly with a model and it forgets everything the moment the session closes. Your preferences, your corrections, the choices you already made. Gone. For an agent to be useful, it needs persistent memory across sessions. I’ve written about this before.
One system here is mem0, which uses a hybrid RAG architecture to persist state. It does asynchronous fact extraction, conflict resolution when new information contradicts old, and grounds the retrieved memories back into the prompt. The retrieval layer is what lets the agent surface the right past fact at the right time instead of replaying the entire history.
5. Evaluation and Test Harnesses
Testing AI in production is hard because the outputs aren’t deterministic. So you build evaluation harnesses that run your agent across hundreds of test cases, and RAG turns out to be a quiet workhorse in that loop.
Two ways it helps:
- Diffing the test suite. Running every eval on every pull request is slow and expensive. Instead, query a vector index of your test suite using the git diff as the query, and run only the cases relevant to the code you actually touched.
- Semantic assertions. Exact string matching is useless when you’re verifying something like an agent’s summary. Instead, the harness retrieves historic successful runs and uses vector similarity to ask whether the new output matches the intent and tone of the target, rather than matching it character for character.
None of this replaces the document-search version of RAG. It’s the same core trick, embed things, retrieve by similarity, ground the result, pointed at different problems: which tool, which example, which memory, which test. Once you start seeing retrieval as a general-purpose way to feed an agent the relevant slice of a much bigger pile, it shows up everywhere. I’ll probably keep finding more.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Sources
- Yury A. Malkov and Dmitry A. Yashunin, “Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs” (arXiv:1603.09320, 2016) — details on the multi-layer graph architecture and logarithmic complexity of the HNSW index.
- Hervé Jégou, Matthijs Douze, and Cordelia Schmid, “Product Quantization for Nearest Neighbor Search” (IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011) — explains compressing high-dimensional vectors and combining product quantization with inverted file indexing (IVF-PQ).
- mem0ai, “mem0: The Memory Layer for Personalized AI” (GitHub) — documentation and codebase for the persistent, self-improving memory layer for AI agents.
- Mostafa Ibrahim, “Agentic RAG vs Classic RAG: From a Pipeline to a Control Loop” (Towards Data Science, March 2026) — commentary on the shift from static document retrieval to agentic control loops and its associated system failure modes.
- Microsoft Research, “Tool-space interference in the MCP era: Designing for agent compatibility at scale” — the tool-space interference (TSI) problem from section 2, where overlapping tool descriptions degrade agent tool selection.
- rewire.it, “Dynamic Tool Allocation for AI Agents (The RATS Pattern)” — the retrieval-augmented tool selection (RATS) pattern from section 2: a router retrieves a relevant subset of tools from a larger catalog.
-
A Field Glossary for Agentic Knowledge Work
Everywhere you look right now, somebody is saying agentic this and agent that. Harness, scaffold, skill, subagent, agentic OS. The vocabulary is piling up faster than anyone can keep track of, and a lot of it gets used loosely, sometimes by people who don’t actually know what the words mean.
So I figured I’d write down my own working glossary. This isn’t a textbook, and I’m not pretending these are official definitions. It’s how I think about the terms when I’m doing the work. If you’ve been nodding along in conversations without being totally sure what a harness is, this one’s for you.
Agent
There are a lot of flavors of agent, and I’m not going to catalog all of them. Generally speaking, an agent is some code wrapped around an LLM that runs in a loop, has access to tools, and can act using those tools.
That’s the key difference from a plain prompt-and-response call. A direct call to the model gives you one answer and stops. An agent has some degree of autonomy. It can decide to use a tool, look at the result, and keep going. The loop and the tools are what make it an agent instead of a chatbot.
Harness
A harness is the program that sits on top of the model. It manages conversation state, runs the reasoning loop, gives the model access to tools, and enforces the guardrails, things like permissions, controls, and budget.
Here’s an easy way to understand it. The model is the intelligence. The harness is the control on the intelligence. The harness sits between you and the LLM.
A harness can show up in a lot of places. It might be a CLI. It might be a GUI or an app on your phone. It might be a chat thread. You could wire up something like OpenClaw to talk to you in WhatsApp or Telegram, and that chat becomes your harness, while OpenClaw is also a harness underneath. So yes, harnesses can call other harnesses. It’s turtles a little way down.
Scaffold
You’ll hear people say scaffold or scaffolding. This is usually just another word for harness. The prompt, the tools, and the control structure wrapped around the model. Same idea, different label.
Framework or SDK
These are the libraries you build harnesses with. LangChain, the various agent SDKs, or a ready-to-run harness like Claude Code or Hermes.
Worth flagging that framework and SDK mean something specific in regular programming. In the agentic context they’re a little looser. They’re what you build agents and harnesses out of. And it doesn’t have to be off-the-shelf. You can absolutely build your own framework for building your own harnesses if that’s where your head is at.
Context Engineering
This is the big one. The term comes from Karpathy, and while it mattered even more a year ago than it does today, it still applies.
Context engineering is deliberately managing what’s in the context window of the current session. It’s the work of deciding what gets loaded into context and, just as importantly, what gets left out. It’s the successor to what we used to call prompt engineering. The framing shifted because the prompt is only one piece of what the model sees, and the rest of it matters a lot.
MCP
Model Context Protocol. I won’t go deep here, it deserves its own post. MCP is an open protocol for exposing tools and data to an LLM harness. It’s the standard way for your harness to reach out and use third-party software or pull in outside data.
Skills
Skills are a major, important thing, and I’m not going to do them justice in a glossary entry. But here’s the definition.
A skill is a reusable, often self-created capability that bundles up the instructions an agent needs to accomplish a specific task. You can find skills all over the internet now. Everybody’s got their own. You can generate your own pretty easily with the CLI, and harnesses like Claude Code or Hermes can even author their own skills. The word does the work here. Skills are capabilities. It’s how you extend what your agents and harnesses can actually do.
Subagent
A subagent is an isolated child agent spawned from inside a working session.
Say you’re in your harness of choice with a main session running. That orchestration session can fork off a new agent with its own context window, hand it a specific task, and say go do this. The subagent runs on its own, often in parallel with others, working in the background. The main session knows when it finishes and can check the work.
A lot of the time you’ll have a second subagent review the first one’s output. That review loop is the whole idea behind the agentic maturity model, which is a way of thinking about how to structure this kind of work. It’s on GitHub if you want to dig in.
Agentic OS
This is an orchestration layer that combines agents, memory, and tools. It’s not really an operating system, but the name has stuck. You take all these concepts, the skills, the agents, the memory, the tools, and combine them into one organized whole. People are calling that amalgamation an agentic OS.
Second Brain / PKM
A personal knowledge vault. I posted about this just this week. It’s a personal knowledge base that the model can read, search, and extend. Your notes, your references, your accumulated thinking, made available to the agent.
Vibe Coding vs. Agentic Engineering
This is the distinction I care about a lot.
Vibe coding is not really knowing what you’re doing or how it’s being done. Anyone can vibe code. You describe what you want and you accept what comes back.
Agentic engineering is knowing what you’re doing and caring about how it gets done. Not everyone can do that part.
The way people put it is that vibe coding raises the floor and agentic engineering raises the ceiling. Vibe coding lets anyone build something. Agentic engineering lets a professional move a lot faster than they used to. Both are real. They’re not the same thing.
That’s the glossary, at least the version that lives in my head. None of these terms are settled, and half of them will probably mean something slightly different in six months. But hopefully this has helped!
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Sources
- Vibe coding — Wikipedia — the term coined by Andrej Karpathy (Feb 2025); the “vibe coding vs. agentic engineering” distinction the post draws.
- Andrej Karpathy, "+1 for ‘context engineering’ over ‘prompt engineering’" (X, June 25, 2025) — the origin of the term “context engineering” as the successor to prompt engineering.
- Model Context Protocol — Wikipedia — MCP as the open standard introduced by Anthropic (Nov 2024) for exposing tools and data to LLM harnesses.
- Emma Roth, “Anthropic launches tool to connect AI systems directly to datasets” (The Verge, Nov 25, 2024) — news coverage of the MCP launch.
- Agentic Maturity Model — GitHub — the AMM referenced in the Subagent entry; Level 3 describes the worker-plus-reviewer subagent pattern.
-
Karpathy's LLM Wiki: Your Second Brain, Maintained by the Machine
A few months ago Andrej Karpathy dropped a GitHub gist that coined a term I haven’t stopped thinking about: the LLM Wiki. The pitch is simple enough to fit on a napkin. Obsidian is the IDE, the LLM is the programmer, and the wiki is the codebase.
Sit with that for a second. It reframes your second brain as something the model builds and maintains, not something you query. When I walked through the history of the second brain, this is the corner I promised to come back to.
Compile, Don’t Retrieve
The usual move with a big pile of notes is query-time RAG. You ask a question, some vector embeddings go find the closest chunks, and the model stitches together an answer on the spot. It works, but the knowledge never gets organized. You’re re-deriving structure every single time you ask.
Karpathy flips it. Instead of retrieving at query time, the LLM compiles the knowledge ahead of time and keeps it current. The result is a persistent, cross-linked markdown wiki. The model doesn’t get bored, doesn’t skip the boring summary, doesn’t forget to update the index. It just keeps the thing tidy.
Google landed in a similar spot with their Open Knowledge Format (OKF), which formalizes the same idea as a curated markdown bundle with an open spec. So this isn’t one person’s hot take. The pattern is in the water.
The Three Components
Karpathy’s wiki has three parts, and the separation is the whole point.
Raw sources. Your curated collection of source documents. These are read-only. The LLM reads them but never edits them. He recommends a
raw/directory, with subdirectories for non-text files. This is your ground truth, and keeping it untouchable is what keeps the rest honest.The wiki. A directory of LLM-generated markdown: summaries, entity pages, concept pages, comparisons, overviews. The model owns this entirely. It creates the pages, maintains the cross-references, and enforces consistency. You’re not in here hand-editing.
The schema. A document like
CLAUDE.mdorAGENTS.mdthat tells the LLM how the wiki is structured, what the conventions are, and how the workflows run. It evolves as you use the system. Think of it as the contract between you and the machine.The Three Operations
The classic Second Brain CODE method has four steps. Karpathy’s version lands on three operations, which I appreciate.
Ingest. You add a source. The LLM reads it, writes a summary page, and updates the index. While it’s in there, it can also touch up related entity and concept pages and log the event. One new document ripples outward into everything it connects to.
Query. You ask a question against the wiki. The model finds the relevant pages, reads, and synthesizes an answer as markdown, tables, whatever fits. Here’s the part I like: a useful exploration can be filed as a new page. So the act of asking a good question makes the next answer easier to produce. The knowledge base gets smarter by being used.
Lint. A periodic health check. The model hunts for contradictions, stale claims, orphan pages, broken references, and missing concepts. The gaps. Then it suggests new questions to ask and new sources to chase. It’s the maintenance pass you’d never do yourself.
The Supporting Cast
A few files make the whole thing run:
index.mdis the content catalog. Every page, a link, a one-line summary, and some metadata, grouped by category. The model reads this first on a query.log.mdis an append-only chronological log with greppable entries like## [DATE] operation | title.qmdis a local markdown search engine (BM25 plus vector plus an LLM re-rank), available as a CLI or MCP server, for when the wiki outgrows a plain index lookup.
And because this lives in Obsidian, you get some nice perks for free. The Web Clipper turns a browser tab into markdown straight into
raw/. Graph View is a visual lint, hubs and orphan pages pop right out. Dataview pulls frontmatter into dynamic tables. And Marp spits out slides directly from your wiki pages.Why This Clicks for Me
The thing I keep coming back to is the read-only
raw/boundary. So much of the anxiety around AI touching your notes is “what if it mangles something I care about.” Splitting sources from the generated wiki means the model can be as aggressive as it wants in the wiki layer, and your source of truth never moves. The worst case is you regenerate a summary page. No harm done.I haven’t fully committed my own vault to this yet, but the shape is right. A knowledge base that maintains itself, where asking good questions leaves the place better than you found it. I’ll probably keep poking at it.
Sources
- Andrej Karpathy, LLM Wiki: A Pattern for AI-Maintained Knowledge Bases — the three components (raw/wiki/schema), the three operations (ingest/query/lint),
index.md,log.md,qmd, and the Obsidian tooling (Web Clipper, Graph View, Dataview, Marp). - Sam McVeety & Amir Hormati, “Introducing the Open Knowledge Format” (Google Cloud Blog, June 12, 2026) — Google’s OKF as a curated markdown bundle with an open spec, explicitly formalizing the same LLM-wiki pattern.
- Tiago Forte, Building a Second Brain (2022) — the CODE method (Capture, Organize, Distill, Express), the “classic” four-step workflow this post lines up against Karpathy’s three operations.
- Roger Montti, “Google Cloud Announces The Open Knowledge Format” (Search Engine Journal, June 15, 2026) — independent news coverage of OKF, including the quote linking it back to Karpathy’s LLM Wiki gist.
- Cecilia Meis, “Google Launches Open Knowledge Format, an AI Standard” (Semrush Blog, June 23, 2026) — news coverage framing OKF as a vendor-neutral markdown spec for AI agent knowledge.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
LangChain and LLM Routers, the Short Version
LangChain is important to know and understand in the age of agents. Also, LLM routing. They’re related but they’re not the same thing, and the distinction matters.
So lets break it down.
LangChain is the Plumbing
Out of the box, an LLM is a text-in, text-out engine. It only knows what it was trained on. That’s it. LangChain is an open-source framework that connects that engine to the outside world.
It gives you standardized tools to build pipelines:
- Models: Interfaces for talking to different LLMs (Gemini, Claude, OpenAI, whatever you’re using)
- Prompts: Templates for dynamically constructing instructions based on user input
- Memory: Letting the LLM remember past turns in a conversation
- Retrieval (RAG): Connecting the LLM to external databases, PDFs, or the internet so it can answer questions about your data
- Agents & Tools: Letting the LLM actually do things, like execute code, run a SQL query, or send an email
You could wire all of this up yourself, but LangChain gives you the standard pieces so you’re not reinventing the plumbing every time.
LLM Routers are the Traffic Controller
A router is an architectural pattern you build on top of that plumbing. Instead of sending every request through the same prompt to the same massive model, a router evaluates the request and directs it to the right destination. Simple concept, big impact.
Three reasons you’d want one:
- Cost: You don’t need a giant, expensive model to answer “Hello!” or look up a basic fact. Send simple queries to a smaller, cheaper model. Save the heavy model for complex reasoning.
- Specialization: Maybe you have one prompt for writing code and another for searching a company HR manual. The router makes sure the query hits the right expert system.
- Speed: Smaller models and direct database lookups are faster. Routing makes your whole application more responsive.
How Routing Actually Works
In LangChain, there are two main approaches:
Logical Routing uses a fast LLM to read the user’s prompt and categorize it. You tell the router LLM something like: “If the user asks about math, output MATH. If they ask about history, output HISTORY.” LangChain then branches to a specialized chain based on that output.
Semantic Routing skips the LLM entirely for the routing decision. It converts the user’s text into a vector (an array of numbers representing the meaning of the text) and compares it to predefined routes to find the closest match. This is significantly faster and cheaper than asking an LLM to make the call.
LangChain provides
RunnableBranchin LCEL (LangChain Expression Language, their declarative syntax for chaining components) for this, basically if/then/else logic for your AI pipelines. Worth digging into if you’re building with LangChain.Routing is what makes AI applications practical at scale. LangChain is one way to build it. They’re complementary, not interchangeable.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].