Finops
-
Cost and Latency Belong in the Score
Congratulations, the best model available passed your eval. That’s not the question you should be answering. What other models could have achieved equivalent results? How much did it cost to run those other models? How long did it take for those other models to achieve equivalent results? All of these are the questions that you should be asking yourself.
If the only thing you’re measuring is if the output is correct or not, then we already know the ranking. The bigger models are going to score higher on the majority of tasks. You don’t need an eval harness to answer that question. Quality scoring is only part of the answer that you actually care about. On every eval run, you should be recording at least three things per task.
- Quality. Whatever your pass or fail criteria are. This is the part you already have.
- Cost. What the run consumed. Tokens in, tokens out, and the price attached to them.
- Latency. Wall clock, start to finish.
These aren’t particularly hard to capture. Cost and latency are usually right there in the response and you’re probably throwing them away. Latency is a hill to climb in its own right, given that it can vary depending on the reliability of the models. Latency at different times of day also matters more than at others. If the human operator is asleep, then latency matters less.
Once you start recording these three dimensions, the promise is you’ll be able to more accurately answer the question: which model is best?
I mean that’s the whole point of all this, right?
What you’re really building is a system, or an attempt at a system, for predicting the output of a non-deterministic system.
Good luck.
A global spending cap for the month doesn’t answer the question: are you using the wrong model for particular tasks?
Task routing is complicated. I can see it being a complex problem to try to solve.
You’re going to find that cheap models win more often than you’d think. There’s a whole bunch of types of work where you don’t need a frontier model, where you’re just using the wrong tool for the task at hand. Think about reformatting, or extracting data from structured text, or classifying something into a bucket, or summarizing a document, or renaming things. All of these are narrow tasks that don’t require a lot of reasoning between the input and the output.
One thing to try is to do it with the cheap one first and then escalate to a larger model if it fails. You’ll often find that scaling up is a lot easier than scaling down in terms of model intelligence.
The takeaway here is that cost and latency belong in or alongside quality in an eval harness.
Once you start recording all three, then you can start measuring and deciding between models.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Your Agent Needs a Dollar Limit, Not a Token Budget
If you let an autonomous coding agent run in an unbounded loop, I have bad news for you, or rather, your wallet.
It happens easily. An agent gets handed a task, runs into an unhandled error or a failing test, and gets stuck in a retry loop. It re-reads the same files, attempts the same broken patch, and streams tokens the whole time while nobody is watching.
We solved this problem in cloud infrastructure ten years ago. Nobody deploys a Lambda function without an execution timeout. Nobody configures a Horizontal Pod Autoscaler without setting
maxReplicas. Let a container run wild without bounds and your infra team will revoke your deployment credentials before breakfast.Yet here we are, handing autonomous agents full access to our terminal and our API keys with a carrot “go do the task, good luck!”
If we want agents to be production-grade tools, we need to treat token spend like compute spend. The most important feature an agent harness can ship is a first-class, per-task spend ceiling.
Claude Code Shows the Path
Anthropic is paving the way here, and it’s worth talking about.
In Claude Code, you can set a USD ceiling on a single invocation using the
--max-budget-usdflag:claude -p --max-budget-usd 2.00 "refactor auth module"This flag only works in print mode — that’s the
-pabove, which is short for--print.This cap is apparently aware of any fan-out that might occur from subagents, so spend on subagents counts against the same ceiling. Claude Code will then kill the background subagents that are still running if it hits the budget limit. For this feature to work, you need to be running Claude Code v2.1.217 or later.
So if you’re building CLI harnesses on top of Claude Code, this will be a huge quality-of-life feature for you to implement. This way you can kick off a background task and rest assured that the harness will not result in a big surprise on your API bill.
The other one is not a cap at all
Anthropic also has something on the raw API side called the task budget.
However, it is not the same and it will not protect you. Task budgets are in beta, and they hand the model a token allowance for it to run its full agentic loop. It tries to wrap things up gracefully rather than being cut off in the middle of a tool call.
The task budget on the API side is in tokens, not dollars. So it’s fine for getting an idea of whether something is possible within a given token budget, but it’s not going to save you from any surprises on the API bill side of things.
The Gap in Codex and OpenCode
The rest of the CLI agent ecosystem hasn’t caught up.
OpenAI Codex CLI (
codex exec) has no native--max-budget-usdflag or budget setting. You can pin a cheaper model profile, but you cannot set a hard dollar limit on a per-task basis.OpenCode (
opencode run) is in a similar spot, which is strange, given that OpenCode has done a great job of adding features to their CLI harness. Unfortunately, there’s no way to pass a pre-execution cap to OpenCode before you launch a task. It kind of feels like a missed opportunity, or one that they will add soon, given that OpenCode already tracks consumption inside the CLI if you’re using it directly.
How We Hack Around It Today
So how do you enforce a dollar cap on non-Anthropic models right now? You push the problem down a layer and let a gateway handle it — which is one more reason your AI stack probably wants a gateway anyway.
The infrastructure side of things has solved this already with the proxies that are available. They expose the functionality that you need in order to set hard per-key budgets. LiteLLM Proxy will start rejecting calls after that budget has been exceeded, with a
400and abudget_exceedederror type. If you’re using Cloudflare AI Gateway, they shipped a dollar-denominated spend limit in June of this year, which returns a429once you cross the line. You can scope it by model, provider, or other custom metadata, and you can configure it to fail over to a cheaper model instead of blocking, which is a nice to have.
Shift-Left till you get to FinOps
FinOps is what happens when you keep shifting left.
Eventually, we’re going to get tired of the bill.
The gateway vendors have come prepared, and the CLI harnesses have yet to fully adopt a decent token/thinking/dollar budget flag system.
We’ll get this figured out one of these days.
Sources
- Claude Code CLI reference —
--max-budget-usd, its print-mode constraint, subagent spend counting toward the cap, and the v2.1.217 enforcement requirement; verified locally againstclaude --helpon v2.1.220, 2026-07-26. - Anthropic: Task budgets — the advisory, token-denominated API feature; source of the “soft hint, not a hard cap” language and the note that task budgets are unsupported on Claude Code.
- OpenAI Codex CLI: local inspection of
codex exec --help, 2026-07-26 — no budget or spend-cap flag. - OpenCode CLI: local inspection of
opencode run --helpandopencode stats, 2026-07-26 — post-run cost reporting, no pre-run cap. - LiteLLM: Budgets, Rate Limits — virtual key
max_budget,duration, and thebudget_exceededrejection. - Cloudflare AI Gateway: Spend limits — dollar budgets scoped by model, provider, or metadata,
429on block, optional cheaper-model fallback. - Your AI bill is out of control. Cloudflare can fix it now. — the June 2026 launch announcement.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
- Claude Code CLI reference —