Developer tools
-
Onboarding a Second Engineer to an Agent-Heavy Repo
How do you onboard a new engineer to a project when most of what they need to know is not checked into the code at all? It’s the agent configuration. It’s the rules about commands, the conventions the agent applies automatically because it’s stored in your memory. That isn’t shared.
I think we’re still figuring out the answers to this. I’m not sure that we should be sharing the context file. I don’t think we should be sharing subagents. I could see skills being a shared resource.
I think it’s more important that you don’t share everything. The things that should be shared, share cleanly. Set it up like you would set up linting rules on your pipeline.
Understanding how the other person intends to be productive will impact what items need to be shared. Having access to a good memory system is way more important than checking in your context file.
When a new person joins, of course, give them access to the code. Of course, give them access to things that you’ve agreed upon should be shared. But you should also talk with them and understand what their expectations are of a real workflow from end to end. Do they have the tools that they need in order to build reliable and repeatable workflows that will get results?
Access to the tools that they need is more important than sharing every single tiny detail.
People responsible for making changes to a system can learn a lot by just doing. You need to understand the tools before you can understand the system.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Developer-tools Documentation Agentic-coding Engineering-teams Onboarding
-
Boring Is a Feature
What does boring look like in the age of AI? And I’m not talking about uninteresting. I’m talking about highly maintainable.
JavaScript?
I mean I guess models are good at it. Everybody knows it. It runs everywhere. The biggest problem with JavaScript is that TypeScript is better.
Certainly it’s better than picking a novelty language that you haven’t built anything with before. Does anybody on the team actually know Haskell? And how long ago did they know Haskell? You need to evaluate the cost of adopting it just as you would evaluate how long it would take to learn it and train the team on it.
The upfront cost can be easier to measure. But the recurring ones are much harder to predict. What happens when a maintainer moves on from a package that everyone uses, and the speed it takes to find a new maintainer is not as fast as you need it to be?
Boring tools are ones where the recurring cost of maintenance is as close to zero as you can get it. If you come back to it in eight months, it should work the way you remember it. This is a fairy tale that we tell ourselves, that nothing ever changes and we can control that change.
Is boring even possible in the age of AI, when it feels like everyone has their own particle beam cannon that they can point at your codebase?
I think it’s worth talking about what I mean by boring, because it can be used as a synonym for old, but that’s not what I mean. Boring means predictability.
Take all your npm packages. Can you answer these questions about all of them? Probably not.
- How often do the release notes contain the word “breaking”? Skim a year of them. This is the single best signal available and it takes ten minutes.
- How many people can merge? One is a risk regardless of how good that one person is. People change jobs, burn out, and lose interest.
- What happens to old versions? A project that supports the previous major for a while is telling you something about how it thinks about your time.
- Can you read the source? Not all of it. Enough to fix something yourself when you’re blocked and nobody’s answering.
Learn the new tool. Experiment. Try new things. Stay passionate about software. Just because you can use the new thing doesn’t mean you should.
Don’t always pick the boring option, just like you don’t always pick the new option. It takes wisdom to know what the right answer is.
You have to understand your failure modes, and when it’s an appropriate time to take a risk, and the scale of the risk.
Pick boring for the parts you don’t want to think about. Save the interesting decisions for the places where being interesting is the point.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
DevOps Developer-tools Software-development Engineering Tooling
-
Cost and Latency Belong in the Score
Congratulations, the best model available passed your eval. That’s not the question you should be answering. What other models could have achieved equivalent results? How much did it cost to run those other models? How long did it take for those other models to achieve equivalent results? All of these are the questions that you should be asking yourself.
If the only thing you’re measuring is if the output is correct or not, then we already know the ranking. The bigger models are going to score higher on the majority of tasks. You don’t need an eval harness to answer that question. Quality scoring is only part of the answer that you actually care about. On every eval run, you should be recording at least three things per task.
- Quality. Whatever your pass or fail criteria are. This is the part you already have.
- Cost. What the run consumed. Tokens in, tokens out, and the price attached to them.
- Latency. Wall clock, start to finish.
These aren’t particularly hard to capture. Cost and latency are usually right there in the response and you’re probably throwing them away. Latency is a hill to climb in its own right, given that it can vary depending on the reliability of the models. Latency at different times of day also matters more than at others. If the human operator is asleep, then latency matters less.
Once you start recording these three dimensions, the promise is you’ll be able to more accurately answer the question: which model is best?
I mean that’s the whole point of all this, right?
What you’re really building is a system, or an attempt at a system, for predicting the output of a non-deterministic system.
Good luck.
A global spending cap for the month doesn’t answer the question: are you using the wrong model for particular tasks?
Task routing is complicated. I can see it being a complex problem to try to solve.
You’re going to find that cheap models win more often than you’d think. There’s a whole bunch of types of work where you don’t need a frontier model, where you’re just using the wrong tool for the task at hand. Think about reformatting, or extracting data from structured text, or classifying something into a bucket, or summarizing a document, or renaming things. All of these are narrow tasks that don’t require a lot of reasoning between the input and the output.
One thing to try is to do it with the cheap one first and then escalate to a larger model if it fails. You’ll often find that scaling up is a lot easier than scaling down in terms of model intelligence.
The takeaway here is that cost and latency belong in or alongside quality in an eval harness.
Once you start recording all three, then you can start measuring and deciding between models.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Your Vibes Are Not an Agent Eval
Do you really know what you’re doing? You swapped a model, you tuned an agentic workflow, you had the agent rewrite a chunk of a system prompt. You added a skill. Now the output feels sharper. Is that feeling a measurement? No, it’s an impression. This is when the vibes start seeping into your agentic engineering world view.
How Impressions Fail
A subjective assessment isn’t necessarily useless, since it’s how you can notice something is wrong in the first place, and it can be a good signal, or an early signal that leads to a corrective action.
A few things here conspire to work against you.
Recency. You remember the last three runs vividly and the forty before them not at all. If the last three happened to be easy tasks, the model got better. If they were gnarly, it got worse.
Confirmation. You just spent an hour rewriting a prompt. You are not a neutral judge of whether that hour helped. Nobody is.
Prompt drift. This one is sneakier. You’re not asking the same thing you asked last month. Your prompts got better because you got better at prompting, and that improvement gets silently credited to the model.
Task drift. The work changed. You were doing greenfield scaffolding in June and you’re doing debugging in August. Those exercise completely different capabilities, and comparing across them tells you nothing.
All of these will sneak up and bite you in the ass. A decent working knowledge of the system is not a measurement.
The actual risk with an agentic workflow isn’t a sharp and dramatic decline in quality. It’s a slow regression over time as you start missing things that slip through the cracks when you’re not paying attention as closely as you should on that day.
I talked about evals that are worth building in previous posts. You should go have a look at some examples there on how to get started.
How do you test a harness? You need to separate the model from the harness. It turns out the harness changes frequently along with the model. Is it even worth testing the harness?
A model swap tripwire is a good place to get started. A tripwire asks whether this specific change made things worse. It’s a binary operation. You run it before the swap, save the results, and run it again after the swap, and compare the results. Same task, same prompt, only the model changed.
If you keep going on vibes, they will keep telling you things. That may or may not matter. At the end of the day, vibes are a decent smoke alarm, but make for a terrible way to measure quality.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Start With Ten Tasks You Actually Do
Public benchmarks of large language models are a fine way to compare models in the abstract, but they’re close to useless for answering questions about things that actually matter. Generally, it’s helpful to know which model is the best in general at a specific benchmark, but it doesn’t answer the question of which model is the best at that specific thing you ask it to do all day long.
You need to build a golden set. Ten tasks where you understand the input and the output.
Scoring the code is not the hard part. Picking the ten tasks is going to be the hardest thing. How do you pick something that has a true pass or fail, but also applies to your specific problem?
The temptation is to sit down and try to come up with representative tasks. Chances are you’re going to waste a ton of time and not produce any better results.
Instead, you should be harvesting your tasks from a variety of sources.
What happens when the benchmark that matters is your last ten pull requests?
- Git history. What have you actually been changing? A month of commits will show you the shape of your work faster than introspection will.
- Issues and TODOs. These are tasks somebody already wrote down in task-shaped language.
- Prior agent sessions. If you have logs, this is the best source, because it’s literally the distribution you’re trying to measure.
- The things you retry. Anything you’ve asked an agent twice because the first answer was wrong is a high-value task. It’s already demonstrated it can discriminate.
A good eval task is one where you already know what the failure is and that it’s possible, because you’ve seen it fail.
Scoreable Means Checkable
So now you have your tasks, and it’s a different problem. You need to decide how to score the task and whether or not the agent got it right. This doesn’t necessarily mean you have to build automation from day one. Some good starting points that would qualify are the following:
- Tests pass or don’t
- Output parses or doesn’t
- The right files changed and no others
- A required field is present and well-formed
- The result matches a known-good output you saved earlier
Keep the amount of comparisons small. Don’t expand and keep evaluating. You can keep your old evaluations, but they shouldn’t impact future decisions forever.
Some amount of change in passing or failing over time is representative of a healthy set.
Here’s how you can get started.
Open your git log. Find things that the agent did well and things the agent could have done better. Write a definition of done.
You now have a small golden set that’s going to be more relevant and useful than any leaderboard online, because it was built from the results of your actual work.
The leaderboard tells you which model wins on average. You are not the average.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Your Agent Needs a Dollar Limit, Not a Token Budget
If you let an autonomous coding agent run in an unbounded loop, I have bad news for you, or rather, your wallet.
It happens easily. An agent gets handed a task, runs into an unhandled error or a failing test, and gets stuck in a retry loop. It re-reads the same files, attempts the same broken patch, and streams tokens the whole time while nobody is watching.
We solved this problem in cloud infrastructure ten years ago. Nobody deploys a Lambda function without an execution timeout. Nobody configures a Horizontal Pod Autoscaler without setting
maxReplicas. Let a container run wild without bounds and your infra team will revoke your deployment credentials before breakfast.Yet here we are, handing autonomous agents full access to our terminal and our API keys with a carrot “go do the task, good luck!”
If we want agents to be production-grade tools, we need to treat token spend like compute spend. The most important feature an agent harness can ship is a first-class, per-task spend ceiling.
Claude Code Shows the Path
Anthropic is paving the way here, and it’s worth talking about.
In Claude Code, you can set a USD ceiling on a single invocation using the
--max-budget-usdflag:claude -p --max-budget-usd 2.00 "refactor auth module"This flag only works in print mode — that’s the
-pabove, which is short for--print.This cap is apparently aware of any fan-out that might occur from subagents, so spend on subagents counts against the same ceiling. Claude Code will then kill the background subagents that are still running if it hits the budget limit. For this feature to work, you need to be running Claude Code v2.1.217 or later.
So if you’re building CLI harnesses on top of Claude Code, this will be a huge quality-of-life feature for you to implement. This way you can kick off a background task and rest assured that the harness will not result in a big surprise on your API bill.
The other one is not a cap at all
Anthropic also has something on the raw API side called the task budget.
However, it is not the same and it will not protect you. Task budgets are in beta, and they hand the model a token allowance for it to run its full agentic loop. It tries to wrap things up gracefully rather than being cut off in the middle of a tool call.
The task budget on the API side is in tokens, not dollars. So it’s fine for getting an idea of whether something is possible within a given token budget, but it’s not going to save you from any surprises on the API bill side of things.
The Gap in Codex and OpenCode
The rest of the CLI agent ecosystem hasn’t caught up.
OpenAI Codex CLI (
codex exec) has no native--max-budget-usdflag or budget setting. You can pin a cheaper model profile, but you cannot set a hard dollar limit on a per-task basis.OpenCode (
opencode run) is in a similar spot, which is strange, given that OpenCode has done a great job of adding features to their CLI harness. Unfortunately, there’s no way to pass a pre-execution cap to OpenCode before you launch a task. It kind of feels like a missed opportunity, or one that they will add soon, given that OpenCode already tracks consumption inside the CLI if you’re using it directly.
How We Hack Around It Today
So how do you enforce a dollar cap on non-Anthropic models right now? You push the problem down a layer and let a gateway handle it — which is one more reason your AI stack probably wants a gateway anyway.
The infrastructure side of things has solved this already with the proxies that are available. They expose the functionality that you need in order to set hard per-key budgets. LiteLLM Proxy will start rejecting calls after that budget has been exceeded, with a
400and abudget_exceedederror type. If you’re using Cloudflare AI Gateway, they shipped a dollar-denominated spend limit in June of this year, which returns a429once you cross the line. You can scope it by model, provider, or other custom metadata, and you can configure it to fail over to a cheaper model instead of blocking, which is a nice to have.
Shift-Left till you get to FinOps
FinOps is what happens when you keep shifting left.
Eventually, we’re going to get tired of the bill.
The gateway vendors have come prepared, and the CLI harnesses have yet to fully adopt a decent token/thinking/dollar budget flag system.
We’ll get this figured out one of these days.
Sources
- Claude Code CLI reference —
--max-budget-usd, its print-mode constraint, subagent spend counting toward the cap, and the v2.1.217 enforcement requirement; verified locally againstclaude --helpon v2.1.220, 2026-07-26. - Anthropic: Task budgets — the advisory, token-denominated API feature; source of the “soft hint, not a hard cap” language and the note that task budgets are unsupported on Claude Code.
- OpenAI Codex CLI: local inspection of
codex exec --help, 2026-07-26 — no budget or spend-cap flag. - OpenCode CLI: local inspection of
opencode run --helpandopencode stats, 2026-07-26 — post-run cost reporting, no pre-run cap. - LiteLLM: Budgets, Rate Limits — virtual key
max_budget,duration, and thebudget_exceededrejection. - Cloudflare AI Gateway: Spend limits — dollar budgets scoped by model, provider, or metadata,
429on block, optional cheaper-model fallback. - Your AI bill is out of control. Cloudflare can fix it now. — the June 2026 launch announcement.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
- Claude Code CLI reference —
-
Where Should This Agent Knowledge Live?
Every agent has a junk drawer.
It usually starts with project instructions. Then I added build commands, personal preferences, database warnings, old architecture decisions, and things to fix later.
The agent could see everything if I wanted it to, but then it would have to read a small novel before touching the code, recurring workflows were buried between random facts, and completed work kept hanging around like it was still relevant. I had given the agent more context and somehow made it less informed.
The problem was not missing knowledge. The problem was putting every kind of knowledge in the same place. An instruction, a skill, a memory, and an issue can all be written in Markdown.
They do four completely different jobs.
Four Places, Four Jobs
A clean version looks like this:
What the agent needs Where it belongs A rule that must apply during every relevant session Instructions A reusable procedure for a particular kind of work Skill A durable fact that may become relevant later Memory A commitment that remains open until completed Issue tracker In practice, this is messier than a table… That’s where the engineering and attention to detail really matter.
So the useful questions are Does the agent need to know about this? and What bucket does this knowledge belong in?
Instructions Are Guaranteed and Expensive
Project instruction files are the things your agent loads automatically. Depending on the tool, that might be
AGENTS.md,CLAUDE.md, or another repository-level file.This is your guaranteed layer. The agent (or harness) doesn’t have to remember to search for it. If a session starts in the project, the rules are sitting in context.
Use that guarantee for knowledge that must shape nearly every relevant action:
- the preferred package manager and command runner,
- where the main source and tests live,
- dangerous commands that require explicit approval,
- the authoritative source for important data,
- mandatory validation before work counts as complete,
- a pointer telling the agent when to load a skill or recall a memory.
The guarantees come with a cost. Every line added to the context is loaded on every session, even when all you really need is a lightweight session where that context doesn’t matter.
Be diligent about cleaning up and maintaining your guaranteed context window, especially if you don’t have a memory layer in place.
Skills Are Procedures With Judgment
A skill answers a different question: how should the agent perform this kind of work?
Publishing a blog post, reviewing a pull request, applying a database migration, preparing a release, updating dependencies. Those are workflows. They have an entry condition, a sequence, safety rules, and a way to verify the result. That’s more than a fact. It’s operational judgment packaged for reuse.
Before we had skills, we had playbooks. Now we can make playbooks out of anything.
A good skill tells the agent when the workflow applies, what to inspect before acting, which steps and tools are appropriate, what must never happen silently, and what evidence proves the work succeeded.
Maybe the deployment instructions can now stay short; when doing a deployment, load the deployment skill.
Instructions are guaranteed. Skills are conditional.
Memory Is Context, Not Policy
Memory is where durable facts live without being injected into every session.
I prefer pnpm for JavaScript and most TypeScript projects. I prefer uv, and sometimes Poetry, for Python. These are facts that shouldn’t have to be repeated.
What about that time you had to troubleshoot an integration and observed some strange behavior? What about when you changed the database design and it broke the support layer? None of this deserves to be injected into every prompt, but it deserves a place where the details can be accessed later.
A semantic memory system can store and retrieve the relevant durable facts and give them to the agent when it asks. I described this earlier.
Memory can be large, and it can be flexible. But it’s also not guaranteed. The agent might not use the right keyword. You might have a problem with the vendor. A critical dependency could go down and take the memory system offline.
Don’t put safety-critical policies in memory. It’s good to have backups. If preferences get lost, they can be recreated, but absolutes like never print secret values belong in several places. If it has anything to do with security, cover your ass.
Memory is best for facts, preferences, relationships, explanations, and past decisions.
Issues Are Promises, Not Storage
An issue tracker tells the agent what needs to be done.
Issues have always been little documentation vaults. We write the history of the bug as it travels through the system. We link back to the issue as it maintains relevance.
An issue should preserve the context for a decision. It should act as a durable property of the project, recording the circumstances around a decision point.
Don’t make it a container for everything the agent did along the way, but I think it’s totally fine if you use it to publish an implementation plan.
Just, you know, you gotta read it.
Our job now is reading about software. The issue trackers are our corpus.
Route the Knowledge With Four Questions
When I don’t know where something belongs, these four questions can help.
1. Must the agent know this before it acts?
Instruction
2. Is this about performing a recurring kind of work?
Skill
3. Is this a durable fact that may help later?
Memory
4. Is this unfinished work or a commitment?
Issue
Sometimes the answer can be more than one place. But don’t copy the content blindly between locations.
All the files may be Markdown, but maintaining the architecture now means knowing where to put the information.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Pi and Hermes Are Trying to Solve Different Problems
I went looking for a talk from Mario Zechner, the creator of Pi, because I wanted to understand why someone would build another coding agent when we already have a pile of them. I found: a talk called “Building pi in a World of Slop.”
Zechner described Pi as a minimal, extensible coding agent that should fit your workflows instead of forcing you into its workflow. He also made a point that should be printed on the box of every AI coding tool: code is not free. The model can produce it quickly, sure. You still own the review, the maintenance, the weird edge cases, and the next person trying to understand it six months later.
That framing explains Pi better than any feature list does.
I’ve also been reading about Hermes Agent, from Nous Research. Hermes is a useful comparison because it’s also an open, provider-flexible agent harness. But it isn’t trying to be Pi with a few extra switches turned on.
Pi and Hermes are trying to solve different problems.
Pi Gives You a Small Place to Start
Pi is a terminal coding harness with a deliberately small default: read files, write files, edit files, run shell commands. Underneath that CLI is a set of TypeScript packages for model access, the agent loop, sessions, and the terminal UI. You can use the CLI, run it through JSON/RPC, or embed the SDK in something else.
That last part is the point.
Pi deliberately leaves out things a lot of agent products treat as table stakes: MCP in the core, subagents, plan mode, permission popups, to-do lists, background shell work. This can look like a missing-feature list if you evaluate it like Claude Code or another finished product.
I don’t think that’s the right test.
Those omissions are Pi’s design. It’s saying: a harness should give you a stable loop, a tool boundary, sessions, and enough extension points to build the workflow you actually need. Then it should get out of the way.
Want MCP? Add it. Want a planning workflow? Make one. Want agents that coordinate over a message bus, work in separate git worktrees, or run in a weird internal deployment? You own the composition. Pi has extensions and packages for that, and now an explicitly experimental orchestration package, but none of it is presented as the one true way to work.
That’s a compelling idea if you’re building a specialized system. It’s also work. Both things can be true.
Hermes Starts With the System
Hermes starts from almost the opposite direction. It’s an integrated autonomous-agent platform with persistent memory, learned skills, built-in delegation, MCP support, scheduling, multiple execution environments, and surfaces that extend beyond the terminal into messaging and desktop interfaces.
Hermes is asking a larger question: what does an agent need to keep working over time, across channels, with memory of what it has already learned?
That’s not just a bigger Pi configuration.
When Hermes includes persistent memory and skill creation, it’s making those things part of the product contract. When it includes subagents and scheduling, it’s giving you an operating model for delegation and recurring work. You get more out of the box, and you inherit more of the system’s assumptions.
For a lot of people, that’s exactly right. If you want an agent to run continuously, show up in Slack or Telegram, remember prior work, and execute recurring workflows, building all of that from Pi primitives would be a very committed hobby.
Good for you, but I think most teams shouldn’t volunteer for that job unless the control model is part of what they’re building.
The Comparison That Matters
Here’s the version I keep coming back to:
Pi Hermes Default posture Minimal programmable harness Integrated autonomous-agent platform Core workflow You compose the pieces The product ships an opinionated system Multi-agent work Extensions, packages, or your own topology Built-in delegation and parallel work Memory Session primitives and JSONL history Persistent memory and skill-learning features Best fit A workflow or control plane you need to own A capable agent system you want to operate This isn’t a scorecard. Hermes isn’t “better” because it has more rows filled in, and Pi isn’t “purer” because it has fewer.
The question is where you want the complexity to live.
With Pi, much of it lives in the system you build around the harness. You have to decide how agents coordinate, what gets remembered, which tools are safe, and how approval works. In exchange, the result can fit your environment instead of being a very configurable version of someone else’s environment.
With Hermes, more of that complexity is already in the platform. You spend less time assembling basic capabilities, but you should understand its memory model, delegation model, security posture, and operational boundaries before you give it real work.
Neither choice removes responsibility. It just changes the shape of it.
Don’t Build a Harness Because It Sounds Fun
Agent harnesses are one of those things that sound like a great weekend project. You wire up a model, give it a few tools, add memory, spawn a couple subagents, and suddenly you have a tiny digital organization running in your terminal.
Then Monday happens.
The agent needs a permission model. It needs observability. It needs a way to recover from bad state. It needs sensible defaults for credentials and logs. It needs evaluation. It needs someone to own the changes when a provider API shifts or an extension becomes a security problem.
That’s why I like the Pi and Hermes comparison. It makes the tradeoff visible.
Use Hermes when you want an agent platform. It already has an opinion about the features an always-on, multi-surface agent needs.
Use Pi when the workflow itself is the product, or when the product assumptions are exactly what you need to escape. Pi’s small core is valuable because it leaves room for a different control plane.
And if all you need is a better code-review prompt or a way to query one internal system, build that inside the harness you already use. A skill, extension, or MCP server is usually a better answer than inventing an agent platform because you wanted one new capability.
This is the same point I landed on in a recent post: you think you want to build your own harness, but what you usually want is a wrapper around the one you already have.
Code is not free. Neither is a harness.
Sources & References
- “Building pi in a World of Slop” — Mario Zechner (talk) — Pi’s design philosophy, workflow fit, and the cost of generated code.
- Pi documentation — current product scope, installation, extensions, and operating modes.
- Pi usage documentation — default tool surface and deliberate core omissions.
- Pi monorepo — TypeScript package architecture and experimental orchestrator package.
- Hermes Agent documentation — persistent memory, skills, delegation, MCP, execution environments, and surfaces.
- Hermes Agent repository — open-source project and implementation reference.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Don't Build a Full Agentic Harness. Wrap One Instead.
I keep seeing people talk about building an “Agentic OS”. A personal system where agents get tools, memory, sub-agents, long-running tasks, permissions, and maybe a little dashboard with colored dots so you know the robots are thinking.
I get it. It sounds fun. It is fun. I like building things too.
So first, a distinction, because the word gets thrown around loosely. Wrapping an existing harness is fine. A little script that shells out to Claude Code or Codex to do one job is a wrapper, and most people should build those. Plenty of people build one by accident and never call it a harness. I’ve got three small ones driving the pipeline that publishes this blog, each just handing a task to Claude Code and getting out of the way. That’s not the trap.
The trap is starting from a raw model API and trying to rebuild the whole thing, replicating Codex or Claude Code or OpenCode from scratch because you want a better way to work with agents. Do that and chances are you’re about to spend several weekends building a worse version of the tool you already have.
No-thX.
So the useful question isn’t “should I build a harness?” It’s how much of the harness do I need to own?
You’re Choosing an Ownership Level
A harness is all the stuff around the model: the agent loop, tool execution, sessions, permissions, memory, context management, orchestration. The model is the part you rent. The harness is the part you choose to own.
And that choice is a slider, not a binary.
What you do What you own What you inherit Extend a full harness Skills, sub-agents, MCP servers, conventions The loop, permissions, sessions, tool execution Run an open harness Deployment, provider, config Most core agent machinery Start from primitives The loop, workflow-specific behavior A small SDK and a few tools Build an agent app Everything that makes it a product Framework primitives, maybe Most people should start at the top and work down only when they hit a real reason to.
Start by Composing What Already Works
If you already use Claude Code, Codex, Gemini CLI, or another complete coding harness, you have more leverage than you think.
You can add specialized agents. You can write skills for recurring work. You can connect MCP servers for memory, documents, databases, browser tooling, whatever you need. You can set project conventions so the agent doesn’t rediscover the same rules every time.
That’s not “just configuration.” Configuration is how you shape a system without becoming responsible for every moving part inside it. The host already knows how to run the model loop, ask permission before risky actions, manage sessions and context, stream tool output, handle files and diffs, and coordinate sub-agents.
I’ve got a vault full of agent instructions, task-specific skills, persistent memory, and a few specialized agents. None of it required me to write a scheduler, a context-compaction system, or an approval UI from scratch. Good. I have other things to do.
The DIY Tax Is Real
A basic agent loop looks almost insultingly simple:
send prompt → receive tool call → run tool → send result → repeatYou can get that running in an afternoon.
What happens when a tool hangs? When the user cancels halfway through a long task? Where do sessions live, and how do you resume them? How do you show the user what changed? How do you stop an agent from reading the wrong file, deleting the wrong directory, or spending five dollars retrying the same broken command?
Then you need permissions. Sandboxing. Tool schemas. Retries. Logging. Secret handling. Context limits. Model fallbacks. Observability. A way to update all of it without turning your harness into the largest unmaintained project in your life.
It works. But “it works” and “it’s a good idea” are two very different things.
When You Should Go Lower
There are good reasons to own more of the stack. Maybe you need provider independence, routing cheap models to bulk work and expensive ones to the hard problems. Maybe you need an agent running persistently on your own infrastructure. Maybe the agent has to live inside another product, not a coding CLI. Maybe your workflow is weird: several specialized agents passing structured work between each other, a custom approval model, durable state that’s part of the thing you’re selling.
Those are all real reasons to move down the slider.
The trap is that people move down the slider because they’re curious, not because they hit a constraint. Curiosity is a great reason to build a prototype. It is not automatically a great reason to make yourself responsible for a runtime.
The Path That Doesn’t Make You Miserable
- Extend the harness you already use. Add a few good skills, focused sub-agents, the tools and memory you need. Do this first, because it shows you which parts of the workflow are painful before you replace anything.
- Add a model-agnostic harness for the jobs that hurt. When billing, deployment, or long-running automation become a real problem, reach for an open, self-hostable harness like Goose, OpenHands, Hermes Agent, or Pi. You still inherit the hard machinery but get control over providers and hosting. (This is also the layer where a model gateway like OpenRouter or LiteLLM slots in underneath, so you’re not locked to one vendor.)
- Drop to primitives for one narrow workflow. Skip the harness entirely and write the loop yourself on a thin SDK, the Vercel AI SDK or Anthropic’s Claude Agent SDK, when you need an embedded agent or a topology existing tools can’t represent cleanly. Build the smallest thing that proves the point. Don’t start by recreating a general-purpose coding agent.
- Reach for a framework when you’re shipping an agent product. If the harness itself is the product, then yes, you probably need graphs, durable state, domain models, and all the rest. This is where something like LangGraph or CrewAI earns its weight. That’s a different project from improving your own workflow.
One tool blurs steps 1 through 3 on purpose, and it’s worth calling out: Pi (pi.dev). It’s a coding-agent CLI you can use today and a TypeScript SDK you build your own harness on top of, provider-agnostic, with a “primitives, not features” core. If you already know you’ll want to customize, Pi lets you start by using it and grow into owning the loop, one extension at a time, without ever switching tools. Hermes Agent sits in similar territory for the self-hosted, model-agnostic case. Either one is a saner on-ramp than a from-scratch build on day one.
Own the Part That Makes You Different
The more of the harness you own, the more control you have. You also own more bugs, more security decisions, more context problems, and more ways for an agent to fail that are hard to explain.
So my strategy is boring in the good way. Start with a full harness. Compose it around your work. Add a lower-level tool only when you can name the limitation it solves. Build the loop yourself only when owning the loop is the point.
You don’t need an Agentic OS to get serious value out of agents. You need a workflow that helps you finish work without becoming one more system you have to maintain.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Your Coding Agent Should Not Own Your Editor
Every coding agent wants to become the place where you work. It starts life as a command-line tool. Then it grows a chat interface, a diff viewer, a permission system, a terminal, a model picker, and eventually an editor integration. Meanwhile, every editor has to build a separate adapter for every agent its users might want. Ten editors, ten agents, and suddenly you’re staring at a hundred bespoke pairings nobody wants to maintain.
We’ve seen before, it’s called a shared boundary.
That’s what the Agent Client Protocol (ACP) is. An agent implements ACP once. An editor implements ACP once. Now you can run the agent inside the editor without either product having to swallow the other whole. It sounds like plumbing because it is plumbing. Plumbing is also the thing that makes ecosystems possible.
ACP in One Sentence
ACP is an open protocol for communication between an AI coding agent and the application presenting that agent to you.
Let’s be specific:
- The client is usually the editor or IDE. It owns the interface, your local environment, and the interaction with you.
- The agent is the coding-agent process. It owns the model loop, the conversation state, and the tool-use logic.
That word agent carries a lot of baggage, so it’s worth nailing down. In everyday AI talk it can mean the raw model, a harness like Claude Code or Opencode that wraps a model, or a subagent that a larger agent spawns to handle a subtask. ACP means the narrow one: the coding-agent process, the harness itself. You’re almost never talking to the model directly in this picture. You talk to the harness, and it drives the model on the other side. An ACP agent also isn’t a subagent. Subagents are an internal detail of whatever the harness does behind its own loop, invisible to the protocol. ACP draws its boundary one level up, between the editor and the whole coding-agent process, not between an agent and its helpers.
In the common local setup, the editor launches the agent as a subprocess and they trade newline-delimited JSON-RPC messages over stdin and stdout.
flowchart LR U[Developer] <--> C[ACP client<br/>Editor or IDE] C <-->|JSON-RPC over stdio| A[ACP agent<br/>Coding-agent process] A <--> M[Model provider] A <--> T[MCP servers and tools] C <--> W[Workspace, buffers, terminals]The whole design is in that picture: the editor and the agent stay separate programs. The agent doesn’t rebuild a serious code-review UI. The editor doesn’t reimplement the reasoning loop. Each side keeps the part it already understands.
The LSP Analogy Only Gets You Halfway
The usual pitch borrows from the Language Server Protocol, and the economics do rhyme. Before LSP, editors built language support one language at a time. After, one language server worked across many editors. ACP applies the same trick to agents: agents stop maintaining an integration per editor, editors stop maintaining one per agent, and you can swap agents without changing where you review code.
But don’t take the analogy too literally. A language server answers bounded questions. Where is this symbol defined? What completions apply here? A coding agent is a long-running, stateful thing. It streams text, announces plans, calls tools, asks permission, edits files, starts processes, and sometimes needs to be interrupted mid-turn. So ACP has to standardize more than request-and-answer. It standardizes enough of the experience of supervising an agent for the client to render it well.
What Happens in a Session
A connection opens with
initialize, where both sides negotiate a version and advertise capabilities. This is deliberately not all-or-nothing. Both programs are expected to cope with optional features being absent.Then the client opens a conversation with
session/newand gets a session ID back. One connection can carry several independent sessions. The client sends your message withsession/prompt, and while the agent works it streamssession/updatenotifications: assistant chunks, thoughts and progress, a plan and edits to that plan, tool calls and their status, mode changes. If a tool call needs a sign-off, the agent sendssession/request_permissionand the editor shows you the choice. Cancel a turn and the client firessession/cancel.That bidirectional flow is the whole difference between ACP and a thin chat API. The agent isn’t just handing back text. It’s exposing a structured account of what it’s doing so the editor can turn that into something you can watch and steer.
The Editor Stops Being a Chat Window
An agent in a plain terminal sees the files on disk. An editor knows more than disk. It has unsaved buffers, syntax highlighting, diagnostics, symbol navigation, a diff UI, and a model of what you’re reviewing right now. ACP’s filesystem methods let the agent ask the client to read or write text including editor state that hasn’t hit disk yet. Its terminal methods let the agent request a command, get a handle, read bounded output, wait for exit, or kill it, while the editor keeps ownership of the process and shows output in its own native terminal.
That split is a lot healthier than every agent inventing its own janky approximation of an IDE. The agent brings intent and execution. The editor brings visibility and control.
ACP and MCP Are Not the Same Thing
People mix these up because both use JSON-RPC and both show up in agent tooling. They sit on different boundaries.
- MCP answers: what can the agent use? Databases, issue trackers, browsers, internal APIs.
- ACP answers: where and how do you work with the agent?
They’re complementary. During
session/new, the ACP client can hand MCP server config to the agent, which then connects to those servers itself. Tools arrive through MCP. Agent work arrives through ACP. Clean.The Interface Is Becoming Its Own Layer
Strip away the JSON-RPC and the method names and the idea is simple: the coding agent and the interface you use to supervise it are different products. Improve the editor without waiting on every agent vendor. Stay in the environment where your code, terminal, and review workflow already live.
We should want coding agents to get better. We should also want the things we use to control them to get better. Those two will move faster if they’re allowed to move apart. That’s the actual promise of ACP. Not one universal agent, but a boundary that stops any single agent from owning the entire way we work.
Sources & References
- Agent Client Protocol introduction and architecture — overview, subprocess model, sessions, permissions, and the relationship to LSP and MCP.
- ACP v1 overview, session setup, and prompt turn — methods, streaming updates, cancellation, and MCP handoff.
- ACP filesystem and terminal methods — unsaved editor state, tracked writes, and process control.
- The ACP Registry is Live — distribution for compatible agents and clients.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].