Agentic-coding
-
Cost and Latency Belong in the Score
Congratulations, the best model available passed your eval. That’s not the question you should be answering. What other models could have achieved equivalent results? How much did it cost to run those other models? How long did it take for those other models to achieve equivalent results? All of these are the questions that you should be asking yourself.
If the only thing you’re measuring is if the output is correct or not, then we already know the ranking. The bigger models are going to score higher on the majority of tasks. You don’t need an eval harness to answer that question. Quality scoring is only part of the answer that you actually care about. On every eval run, you should be recording at least three things per task.
- Quality. Whatever your pass or fail criteria are. This is the part you already have.
- Cost. What the run consumed. Tokens in, tokens out, and the price attached to them.
- Latency. Wall clock, start to finish.
These aren’t particularly hard to capture. Cost and latency are usually right there in the response and you’re probably throwing them away. Latency is a hill to climb in its own right, given that it can vary depending on the reliability of the models. Latency at different times of day also matters more than at others. If the human operator is asleep, then latency matters less.
Once you start recording these three dimensions, the promise is you’ll be able to more accurately answer the question: which model is best?
I mean that’s the whole point of all this, right?
What you’re really building is a system, or an attempt at a system, for predicting the output of a non-deterministic system.
Good luck.
A global spending cap for the month doesn’t answer the question: are you using the wrong model for particular tasks?
Task routing is complicated. I can see it being a complex problem to try to solve.
You’re going to find that cheap models win more often than you’d think. There’s a whole bunch of types of work where you don’t need a frontier model, where you’re just using the wrong tool for the task at hand. Think about reformatting, or extracting data from structured text, or classifying something into a bucket, or summarizing a document, or renaming things. All of these are narrow tasks that don’t require a lot of reasoning between the input and the output.
One thing to try is to do it with the cheap one first and then escalate to a larger model if it fails. You’ll often find that scaling up is a lot easier than scaling down in terms of model intelligence.
The takeaway here is that cost and latency belong in or alongside quality in an eval harness.
Once you start recording all three, then you can start measuring and deciding between models.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Your Vibes Are Not an Agent Eval
Do you really know what you’re doing? You swapped a model, you tuned an agentic workflow, you had the agent rewrite a chunk of a system prompt. You added a skill. Now the output feels sharper. Is that feeling a measurement? No, it’s an impression. This is when the vibes start seeping into your agentic engineering world view.
How Impressions Fail
A subjective assessment isn’t necessarily useless, since it’s how you can notice something is wrong in the first place, and it can be a good signal, or an early signal that leads to a corrective action.
A few things here conspire to work against you.
Recency. You remember the last three runs vividly and the forty before them not at all. If the last three happened to be easy tasks, the model got better. If they were gnarly, it got worse.
Confirmation. You just spent an hour rewriting a prompt. You are not a neutral judge of whether that hour helped. Nobody is.
Prompt drift. This one is sneakier. You’re not asking the same thing you asked last month. Your prompts got better because you got better at prompting, and that improvement gets silently credited to the model.
Task drift. The work changed. You were doing greenfield scaffolding in June and you’re doing debugging in August. Those exercise completely different capabilities, and comparing across them tells you nothing.
All of these will sneak up and bite you in the ass. A decent working knowledge of the system is not a measurement.
The actual risk with an agentic workflow isn’t a sharp and dramatic decline in quality. It’s a slow regression over time as you start missing things that slip through the cracks when you’re not paying attention as closely as you should on that day.
I talked about evals that are worth building in previous posts. You should go have a look at some examples there on how to get started.
How do you test a harness? You need to separate the model from the harness. It turns out the harness changes frequently along with the model. Is it even worth testing the harness?
A model swap tripwire is a good place to get started. A tripwire asks whether this specific change made things worse. It’s a binary operation. You run it before the swap, save the results, and run it again after the swap, and compare the results. Same task, same prompt, only the model changed.
If you keep going on vibes, they will keep telling you things. That may or may not matter. At the end of the day, vibes are a decent smoke alarm, but make for a terrible way to measure quality.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Start With Ten Tasks You Actually Do
Public benchmarks of large language models are a fine way to compare models in the abstract, but they’re close to useless for answering questions about things that actually matter. Generally, it’s helpful to know which model is the best in general at a specific benchmark, but it doesn’t answer the question of which model is the best at that specific thing you ask it to do all day long.
You need to build a golden set. Ten tasks where you understand the input and the output.
Scoring the code is not the hard part. Picking the ten tasks is going to be the hardest thing. How do you pick something that has a true pass or fail, but also applies to your specific problem?
The temptation is to sit down and try to come up with representative tasks. Chances are you’re going to waste a ton of time and not produce any better results.
Instead, you should be harvesting your tasks from a variety of sources.
What happens when the benchmark that matters is your last ten pull requests?
- Git history. What have you actually been changing? A month of commits will show you the shape of your work faster than introspection will.
- Issues and TODOs. These are tasks somebody already wrote down in task-shaped language.
- Prior agent sessions. If you have logs, this is the best source, because it’s literally the distribution you’re trying to measure.
- The things you retry. Anything you’ve asked an agent twice because the first answer was wrong is a high-value task. It’s already demonstrated it can discriminate.
A good eval task is one where you already know what the failure is and that it’s possible, because you’ve seen it fail.
Scoreable Means Checkable
So now you have your tasks, and it’s a different problem. You need to decide how to score the task and whether or not the agent got it right. This doesn’t necessarily mean you have to build automation from day one. Some good starting points that would qualify are the following:
- Tests pass or don’t
- Output parses or doesn’t
- The right files changed and no others
- A required field is present and well-formed
- The result matches a known-good output you saved earlier
Keep the amount of comparisons small. Don’t expand and keep evaluating. You can keep your old evaluations, but they shouldn’t impact future decisions forever.
Some amount of change in passing or failing over time is representative of a healthy set.
Here’s how you can get started.
Open your git log. Find things that the agent did well and things the agent could have done better. Write a definition of done.
You now have a small golden set that’s going to be more relevant and useful than any leaderboard online, because it was built from the results of your actual work.
The leaderboard tells you which model wins on average. You are not the average.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
The Decision Log: A Lightweight Artifact for Agentic Coding
Coding agents are remarkably good at reopening decisions you already made.
Imagine a content pipeline where posts live as local Markdown files and a database holds the scheduling metadata. You open a fresh session. Which one does the agent think is authoritative?
It has to guess. And the database looks like the better answer, because databases usually are. So it proposes the obvious cleanup: make the database the source of truth and treat the files as an export format.
It’s wrong, for a reason the code never states. You can rebuild the database from the files. You cannot rebuild the files from the database. Drop the database and you re-index from markdown, losing some operational history on the way. Lose the markdown and the posts are gone.
Some questions are still better answered by the database, scheduling among them. But the authoritative copy is whichever one you could regenerate the other from, and that asymmetry is the entire argument. It appears nowhere in the schema, nowhere in the file layout, and nowhere in the diff.
So you explain it. The agent takes the point immediately, drops the idea, and gets on with the actual work. Good outcome.
Then the session ends, and the next one arrives with the same instincts and the same blank slate. Nothing’s wrong with its reasoning. It’s missing the one piece of context the repository doesn’t contain: that road has already been walked.
That’s what a decision log is for.
Issues and Skills Leave a Gap
Agentic projects tend to accumulate two useful artifacts.
Issues describe work. Add a field, fix the retry logic, move the cache, update the API client. They tell an agent what needs to change and, if the issue is any good, what done looks like.
Skills and project instructions describe process. Use this package manager. Run these tests. Never call this destructive command. They tell an agent how work should happen, repeatedly.
Both matter. Neither is a natural home for, “We considered Redis, rejected it because this service has to run without another dependency, and we’ll reconsider if the process moves to multiple instances.”
Artifact The question it answers Issue What needs to change? Skill or instruction How should work be done? Decision log Why this path instead of another one? The code records the result. Git records the diff. A closed issue might contain the discussion, if somebody thinks to go looking for it. The decision log keeps the conclusion and the rejected alternatives somewhere an agent can find them before it starts planning.
The Rejected Option Is the Important Part
I wrote recently about decision debt, the gap created when code gets produced faster than anyone records what it means or why it exists. A decision log is one small way to pay that debt as you go.
The useful part isn’t “We chose SQLite.” The repository already contains a SQLite database. The useful part is why SQLite won, which alternatives lost, and what would have to change before the decision should be reopened.
Without that negative history, an agent sees an absence and treats it as an oversight. No Redis? Add Redis. No abstraction around this HTTP client? Generate one. No Kubernetes deployment? Surely the project just hasn’t matured enough yet.
Sometimes the missing thing is missing on purpose.
Humans do this too, of course. We reopen old arguments when the people who remember them leave, or when the conclusion is buried in a meeting recording. Agents just compress the cycle. Every fresh session is a new developer joining the project with excellent technical instincts and absolutely no institutional memory.
Keep the Entry Small
Architecture Decision Records have been around since Michael Nygard described the pattern in 2011. They preserve the status, context, decision, and consequences of an important architectural choice. Lightweight templates like MADR also capture the options considered, the decision drivers, and why one option won.
That’s the right idea. I just don’t need a formal architecture record for every consequential choice in a personal project.
I’d start with some sort of log file, holding entries like this:
## 2026-07-26: Keep Markdown as the publishing source of truth **Context:** Posts exist as local files and as database records. **Decision:** Frontmatter controls publish state. The database owns scheduling metadata and is not authoritative for publishing. **Rejected:** Making the database authoritative, or newest-write-wins. **Why:** The database can be rebuilt from the files. The files cannot be rebuilt from the database. **Revisit when:** Editing moves to a multi-user hosted application. **Links:** the repository module, the publishing documentation.That’s the whole artifact.
An agent that reads that before it starts planning knows the database-as-source-of-truth idea isn’t a fresh insight. It also knows exactly what would have to change before it becomes one again.
I’ll be honest that I haven’t started doing this yet, but it sounds like a good idea right?
The entry doesn’t need a transcript of the debate. It needs enough context for a future person or agent to understand that the alternative was considered, why it lost, and which changed condition would make it worth discussing again.
What Belongs in the Log
If every choice becomes an entry, the log turns into another file nobody reads. I’d record a decision when at least one of these is true:
- Two or more reasonable approaches existed
- A future agent is likely to propose the rejected option again
- The choice establishes a source of truth, security boundary, schema, dependency, or workflow
- Reversing it later would be expensive or dangerous
- The reason isn’t obvious from the code
Don’t log naming arguments, routine implementation details, or every library function you picked. “Used a dictionary here” isn’t institutional knowledge. “Kept provider integrations on raw HTTP because the SDK doesn’t support the endpoint we need” might be.
Major decisions can still become full ADRs in
docs/decisions/or whatever. The lightweight log is for the big middle ground between an architectural record and a comment somebody vaguely remembers leaving on a pull request.
Make Agents Read It at the Right Time
Creating the file isn’t enough. The agent needs a retrieval rule. One sentence in the project instructions:
Before proposing changes to architecture, dependencies, data ownership, security boundaries, or core workflows, search `docs/decisions.md` for related decisions and revisit conditions.Then the other half:
After a consequential decision is approved, propose a short decision-log entry. Do not record a new project policy without human confirmation.The instruction defines the recurring behavior. The log supplies the project-specific facts. That keeps settled choices out of a giant instruction file while still putting them in the agent’s path when they matter.
When a decision changes, don’t quietly rewrite history. Add a new entry that supersedes the old one and say which revisit condition actually showed up. Version control preserves the edit, but the document should make the change legible without requiring repository archaeology.
Not Another Memory System
A chat transcript contains every false start, tool result, and half-formed idea. It’s too noisy to act as a project constitution. Agent memory can help retrieve past context, but it might be private to one tool, unavailable to a collaborator, or hard to review in a pull request.
A decision log needs to be deliberately boring. Plain text. Searchable. Reviewable. Stored next to the code, or accessible by it. Any human or agent can read the same entry and argue with it in the open.
None of this guarantees an agent will make the right call. It removes one wasteful failure mode: spending another session rediscovering a settled tradeoff and confidently proposing the option the project already rejected.
Issues tell the agent where to go. Skills tell it how to move. The decision log marks the roads we already closed, why we closed them, and when they might be worth opening again.
That feels like context worth keeping.
Sources
- Documenting Architecture Decisions — Michael Nygard’s original 2011 ADR proposal.
- MADR — a lightweight decision-record format covering options, rationale, consequences, and revisit conditions.
- The GDS Way: Documenting architecture decisions — keeping decision rationale in the repository while using issues to track implementation.
- How Claude Code works — fresh session context, compaction, project instructions, and persistent memory.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Code Is Cheap Now. Decisions Aren't
I’ve spent a lot of time working on software backlogs with coding agents. Pick an issue, inspect the code, make the change, run the tests, close it, move to the next one.
The strange part is how often writing the code is no longer the hard part.
An agent can add a field to a model, update the database, wire it through a client, and write a pile of tests before I check on it again. That would have been hours or days of time before. Now it happens in one focused session.
Great.
But the feature still needs someone to decide what the field means. Is it nullable? Which system owns it? Can it be inferred from another value? What happens when the provider stops sending it? Does an older local value survive, or should it be cleared?
The typing got cheaper. The decisions did not.
The Keyboard Was Never the Whole Job
We’ve spent decades measuring software work by its visible output. Lines of code, commits, pull requests, story points, features shipped. Code was expensive to produce, so counting the artifacts at least felt related to the effort.
That relationship is falling apart.
You can ask an agent for three implementations of the same feature and get all three before lunch. You can generate a REST API, a migration, a test suite, and a deployment manifest in one sitting. If you don’t like the architecture, ask for another one.
Now you have four implementations and a new problem: which one should exist?
Software development was never just converting requirements into syntax. It was deciding which requirements were real, where the boundaries belonged, what failure looked like, and which tradeoffs you were willing to own. Code generation doesn’t remove that work. It removes the part that used to hide it.
One Small Change, Twelve Real Questions
Let me give you an example.
Say an external API adds an optional date. Adding it to your application sounds trivial. Nullable column, property on the model, parse the response, write a migration, done.
Except the provider already has another date with a similar name. One is when the item was published. The other is when the user interacted with it. They’re both dates. They are not interchangeable.
So now the real questions start:
- Do we store only the explicit value, or infer it when it’s missing?
- Is a missing field the same as a field containing
null? - If the remote value disappears, do we delete the local value?
- Does this field participate in unchanged-record detection?
- Will a dry run exercise the same comparison path as a real sync?
- Can older installations run safely before the migration is applied?
- What happens when a second provider represents the same idea differently?
None of those are unique to any language. The agent can explain the options and implement whichever answer you pick, but it can’t discover your intended data contract from the shape of a JSON response.
If you skip those decisions, you still get code. You just get code that quietly invents policy for you.
Cheap Code Creates Expensive Options
This is the part that you might underestimate.
When implementation was slow, the cost naturally limited how many ideas made it into a codebase. You might sketch three approaches, but you probably built one. The friction forced a little restraint.
Agents remove that friction. A feature that would have been rejected as “not worth a week” now looks attractive because the first version only takes an hour. A speculative abstraction feels harmless because the agent can generate it in minutes. A new internal tool seems reasonable because the prototype already works.
It worked. But “it works” and “it’s a good idea” are two very different things.
The generated code still has to be reviewed. Its dependencies still need updates. Its behavior still needs tests. Someone still has to understand it when the surrounding system changes. If it handles credentials or customer data, somebody owns that risk too.
The implementation cost may be close to zero. The ownership cost is not.
This is how you end up with a backlog full of individually reasonable features that collectively make the project worse. Each one was cheap enough to add. Nobody stopped to price the maintenance costs associated with each successive feature.
Decision Debt
We already have a name for shortcuts in implementation: technical debt. You move quickly today and accept that the code will cost more to change later.
Agentic development creates another kind. Call it decision debt.
Decision debt is what happens when code gets produced faster than anyone can answer why it exists, what promise it makes, and who is responsible for it. The implementation is complete, but the boundaries are fuzzy. The tests prove what the code currently does, but nobody has decided whether that’s what it should do.
You can see it in questions like these:
- Are we supporting this behavior or merely tolerating it?
- Is this data authoritative, derived, or cached?
- Is this workflow meant for one person or every user?
- Is this abstraction solving a repeated problem or predicting one?
- What would make us remove this feature?
An agent can help you reason through every one of those. That’s one of the best uses of the technology. Ask it to inspect the repository, find conflicting assumptions, model failure cases, and challenge the proposed design.
Just don’t confuse receiving an answer with making a decision.
The Human Work Moves Upstream
If agents keep getting better at implementation, the valuable human work moves toward choosing and framing the work.
That means writing a clear definition of done before generating code. Deciding which system is the source of truth. Recognizing when two similar concepts need separate names. Looking at a working implementation and saying, “No, this does not belong here.”
It also means treating restraint as engineering work.
Closing a stale issue without implementing it can be more valuable than shipping the feature. Reusing an existing boundary can beat introducing a cleaner new abstraction. Deleting a half-maintained tool can improve a system more than generating its replacement.
None of those choices produce an impressive diff. That’s fine. The diff was never the product.
What Good Looks Like Now
I don’t think the answer is to slow agents down, or to pretend generated code is somehow less legitimate than code typed by hand. The code doesn’t care who wrote it. If it’s correct, understandable, tested, and worth owning, ship it.
The change is in where we spend our attention.
Spend less time admiring how quickly the implementation appeared. Spend more time checking the assumptions it smuggled in. Ask what new state the system owns, what promise the interface makes, what can fail, and whether the feature deserves to survive its prototype.
Agents make it possible to build almost anything you can describe. Our job is deciding what should still exist six months later.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
AI Software-development Engineering Agentic-coding Technical-debt