AI
-
Cost and Latency Belong in the Score
Congratulations, the best model available passed your eval. That’s not the question you should be answering. What other models could have achieved equivalent results? How much did it cost to run those other models? How long did it take for those other models to achieve equivalent results? All of these are the questions that you should be asking yourself.
If the only thing you’re measuring is if the output is correct or not, then we already know the ranking. The bigger models are going to score higher on the majority of tasks. You don’t need an eval harness to answer that question. Quality scoring is only part of the answer that you actually care about. On every eval run, you should be recording at least three things per task.
- Quality. Whatever your pass or fail criteria are. This is the part you already have.
- Cost. What the run consumed. Tokens in, tokens out, and the price attached to them.
- Latency. Wall clock, start to finish.
These aren’t particularly hard to capture. Cost and latency are usually right there in the response and you’re probably throwing them away. Latency is a hill to climb in its own right, given that it can vary depending on the reliability of the models. Latency at different times of day also matters more than at others. If the human operator is asleep, then latency matters less.
Once you start recording these three dimensions, the promise is you’ll be able to more accurately answer the question: which model is best?
I mean that’s the whole point of all this, right?
What you’re really building is a system, or an attempt at a system, for predicting the output of a non-deterministic system.
Good luck.
A global spending cap for the month doesn’t answer the question: are you using the wrong model for particular tasks?
Task routing is complicated. I can see it being a complex problem to try to solve.
You’re going to find that cheap models win more often than you’d think. There’s a whole bunch of types of work where you don’t need a frontier model, where you’re just using the wrong tool for the task at hand. Think about reformatting, or extracting data from structured text, or classifying something into a bucket, or summarizing a document, or renaming things. All of these are narrow tasks that don’t require a lot of reasoning between the input and the output.
One thing to try is to do it with the cheap one first and then escalate to a larger model if it fails. You’ll often find that scaling up is a lot easier than scaling down in terms of model intelligence.
The takeaway here is that cost and latency belong in or alongside quality in an eval harness.
Once you start recording all three, then you can start measuring and deciding between models.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Your Vibes Are Not an Agent Eval
Do you really know what you’re doing? You swapped a model, you tuned an agentic workflow, you had the agent rewrite a chunk of a system prompt. You added a skill. Now the output feels sharper. Is that feeling a measurement? No, it’s an impression. This is when the vibes start seeping into your agentic engineering world view.
How Impressions Fail
A subjective assessment isn’t necessarily useless, since it’s how you can notice something is wrong in the first place, and it can be a good signal, or an early signal that leads to a corrective action.
A few things here conspire to work against you.
Recency. You remember the last three runs vividly and the forty before them not at all. If the last three happened to be easy tasks, the model got better. If they were gnarly, it got worse.
Confirmation. You just spent an hour rewriting a prompt. You are not a neutral judge of whether that hour helped. Nobody is.
Prompt drift. This one is sneakier. You’re not asking the same thing you asked last month. Your prompts got better because you got better at prompting, and that improvement gets silently credited to the model.
Task drift. The work changed. You were doing greenfield scaffolding in June and you’re doing debugging in August. Those exercise completely different capabilities, and comparing across them tells you nothing.
All of these will sneak up and bite you in the ass. A decent working knowledge of the system is not a measurement.
The actual risk with an agentic workflow isn’t a sharp and dramatic decline in quality. It’s a slow regression over time as you start missing things that slip through the cracks when you’re not paying attention as closely as you should on that day.
I talked about evals that are worth building in previous posts. You should go have a look at some examples there on how to get started.
How do you test a harness? You need to separate the model from the harness. It turns out the harness changes frequently along with the model. Is it even worth testing the harness?
A model swap tripwire is a good place to get started. A tripwire asks whether this specific change made things worse. It’s a binary operation. You run it before the swap, save the results, and run it again after the swap, and compare the results. Same task, same prompt, only the model changed.
If you keep going on vibes, they will keep telling you things. That may or may not matter. At the end of the day, vibes are a decent smoke alarm, but make for a terrible way to measure quality.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Start With Ten Tasks You Actually Do
Public benchmarks of large language models are a fine way to compare models in the abstract, but they’re close to useless for answering questions about things that actually matter. Generally, it’s helpful to know which model is the best in general at a specific benchmark, but it doesn’t answer the question of which model is the best at that specific thing you ask it to do all day long.
You need to build a golden set. Ten tasks where you understand the input and the output.
Scoring the code is not the hard part. Picking the ten tasks is going to be the hardest thing. How do you pick something that has a true pass or fail, but also applies to your specific problem?
The temptation is to sit down and try to come up with representative tasks. Chances are you’re going to waste a ton of time and not produce any better results.
Instead, you should be harvesting your tasks from a variety of sources.
What happens when the benchmark that matters is your last ten pull requests?
- Git history. What have you actually been changing? A month of commits will show you the shape of your work faster than introspection will.
- Issues and TODOs. These are tasks somebody already wrote down in task-shaped language.
- Prior agent sessions. If you have logs, this is the best source, because it’s literally the distribution you’re trying to measure.
- The things you retry. Anything you’ve asked an agent twice because the first answer was wrong is a high-value task. It’s already demonstrated it can discriminate.
A good eval task is one where you already know what the failure is and that it’s possible, because you’ve seen it fail.
Scoreable Means Checkable
So now you have your tasks, and it’s a different problem. You need to decide how to score the task and whether or not the agent got it right. This doesn’t necessarily mean you have to build automation from day one. Some good starting points that would qualify are the following:
- Tests pass or don’t
- Output parses or doesn’t
- The right files changed and no others
- A required field is present and well-formed
- The result matches a known-good output you saved earlier
Keep the amount of comparisons small. Don’t expand and keep evaluating. You can keep your old evaluations, but they shouldn’t impact future decisions forever.
Some amount of change in passing or failing over time is representative of a healthy set.
Here’s how you can get started.
Open your git log. Find things that the agent did well and things the agent could have done better. Write a definition of done.
You now have a small golden set that’s going to be more relevant and useful than any leaderboard online, because it was built from the results of your actual work.
The leaderboard tells you which model wins on average. You are not the average.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
The Decision Log: A Lightweight Artifact for Agentic Coding
Coding agents are remarkably good at reopening decisions you already made.
Imagine a content pipeline where posts live as local Markdown files and a database holds the scheduling metadata. You open a fresh session. Which one does the agent think is authoritative?
It has to guess. And the database looks like the better answer, because databases usually are. So it proposes the obvious cleanup: make the database the source of truth and treat the files as an export format.
It’s wrong, for a reason the code never states. You can rebuild the database from the files. You cannot rebuild the files from the database. Drop the database and you re-index from markdown, losing some operational history on the way. Lose the markdown and the posts are gone.
Some questions are still better answered by the database, scheduling among them. But the authoritative copy is whichever one you could regenerate the other from, and that asymmetry is the entire argument. It appears nowhere in the schema, nowhere in the file layout, and nowhere in the diff.
So you explain it. The agent takes the point immediately, drops the idea, and gets on with the actual work. Good outcome.
Then the session ends, and the next one arrives with the same instincts and the same blank slate. Nothing’s wrong with its reasoning. It’s missing the one piece of context the repository doesn’t contain: that road has already been walked.
That’s what a decision log is for.
Issues and Skills Leave a Gap
Agentic projects tend to accumulate two useful artifacts.
Issues describe work. Add a field, fix the retry logic, move the cache, update the API client. They tell an agent what needs to change and, if the issue is any good, what done looks like.
Skills and project instructions describe process. Use this package manager. Run these tests. Never call this destructive command. They tell an agent how work should happen, repeatedly.
Both matter. Neither is a natural home for, “We considered Redis, rejected it because this service has to run without another dependency, and we’ll reconsider if the process moves to multiple instances.”
Artifact The question it answers Issue What needs to change? Skill or instruction How should work be done? Decision log Why this path instead of another one? The code records the result. Git records the diff. A closed issue might contain the discussion, if somebody thinks to go looking for it. The decision log keeps the conclusion and the rejected alternatives somewhere an agent can find them before it starts planning.
The Rejected Option Is the Important Part
I wrote recently about decision debt, the gap created when code gets produced faster than anyone records what it means or why it exists. A decision log is one small way to pay that debt as you go.
The useful part isn’t “We chose SQLite.” The repository already contains a SQLite database. The useful part is why SQLite won, which alternatives lost, and what would have to change before the decision should be reopened.
Without that negative history, an agent sees an absence and treats it as an oversight. No Redis? Add Redis. No abstraction around this HTTP client? Generate one. No Kubernetes deployment? Surely the project just hasn’t matured enough yet.
Sometimes the missing thing is missing on purpose.
Humans do this too, of course. We reopen old arguments when the people who remember them leave, or when the conclusion is buried in a meeting recording. Agents just compress the cycle. Every fresh session is a new developer joining the project with excellent technical instincts and absolutely no institutional memory.
Keep the Entry Small
Architecture Decision Records have been around since Michael Nygard described the pattern in 2011. They preserve the status, context, decision, and consequences of an important architectural choice. Lightweight templates like MADR also capture the options considered, the decision drivers, and why one option won.
That’s the right idea. I just don’t need a formal architecture record for every consequential choice in a personal project.
I’d start with some sort of log file, holding entries like this:
## 2026-07-26: Keep Markdown as the publishing source of truth **Context:** Posts exist as local files and as database records. **Decision:** Frontmatter controls publish state. The database owns scheduling metadata and is not authoritative for publishing. **Rejected:** Making the database authoritative, or newest-write-wins. **Why:** The database can be rebuilt from the files. The files cannot be rebuilt from the database. **Revisit when:** Editing moves to a multi-user hosted application. **Links:** the repository module, the publishing documentation.That’s the whole artifact.
An agent that reads that before it starts planning knows the database-as-source-of-truth idea isn’t a fresh insight. It also knows exactly what would have to change before it becomes one again.
I’ll be honest that I haven’t started doing this yet, but it sounds like a good idea right?
The entry doesn’t need a transcript of the debate. It needs enough context for a future person or agent to understand that the alternative was considered, why it lost, and which changed condition would make it worth discussing again.
What Belongs in the Log
If every choice becomes an entry, the log turns into another file nobody reads. I’d record a decision when at least one of these is true:
- Two or more reasonable approaches existed
- A future agent is likely to propose the rejected option again
- The choice establishes a source of truth, security boundary, schema, dependency, or workflow
- Reversing it later would be expensive or dangerous
- The reason isn’t obvious from the code
Don’t log naming arguments, routine implementation details, or every library function you picked. “Used a dictionary here” isn’t institutional knowledge. “Kept provider integrations on raw HTTP because the SDK doesn’t support the endpoint we need” might be.
Major decisions can still become full ADRs in
docs/decisions/or whatever. The lightweight log is for the big middle ground between an architectural record and a comment somebody vaguely remembers leaving on a pull request.
Make Agents Read It at the Right Time
Creating the file isn’t enough. The agent needs a retrieval rule. One sentence in the project instructions:
Before proposing changes to architecture, dependencies, data ownership, security boundaries, or core workflows, search `docs/decisions.md` for related decisions and revisit conditions.Then the other half:
After a consequential decision is approved, propose a short decision-log entry. Do not record a new project policy without human confirmation.The instruction defines the recurring behavior. The log supplies the project-specific facts. That keeps settled choices out of a giant instruction file while still putting them in the agent’s path when they matter.
When a decision changes, don’t quietly rewrite history. Add a new entry that supersedes the old one and say which revisit condition actually showed up. Version control preserves the edit, but the document should make the change legible without requiring repository archaeology.
Not Another Memory System
A chat transcript contains every false start, tool result, and half-formed idea. It’s too noisy to act as a project constitution. Agent memory can help retrieve past context, but it might be private to one tool, unavailable to a collaborator, or hard to review in a pull request.
A decision log needs to be deliberately boring. Plain text. Searchable. Reviewable. Stored next to the code, or accessible by it. Any human or agent can read the same entry and argue with it in the open.
None of this guarantees an agent will make the right call. It removes one wasteful failure mode: spending another session rediscovering a settled tradeoff and confidently proposing the option the project already rejected.
Issues tell the agent where to go. Skills tell it how to move. The decision log marks the roads we already closed, why we closed them, and when they might be worth opening again.
That feels like context worth keeping.
Sources
- Documenting Architecture Decisions — Michael Nygard’s original 2011 ADR proposal.
- MADR — a lightweight decision-record format covering options, rationale, consequences, and revisit conditions.
- The GDS Way: Documenting architecture decisions — keeping decision rationale in the repository while using issues to track implementation.
- How Claude Code works — fresh session context, compaction, project instructions, and persistent memory.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Run Your Whole Agent Stack on a $5 Box
I SSH’d into my home server this afternoon and ran
docker statson the memory layer that every one of my coding agent sessions talks to. Here’s what came back:mem0-qdrant 28.09MiB / 60.75GiB 2.13% mem0-neo4j 612.7MiB / 60.75GiB 0.77%640 megabytes. Vector store and graph store, both up for three weeks straight, serving every
rememberandrecallcall my agents make. The entire persistent memory for my AI tooling uses less RAM than one Chrome tab with Figma open.So let’s talk about why you’re paying a monthly subscription for this.
Local-First, Not Local-Only
I want to be precise here, because “self-hosted AI” has become a phrase people use to mean nine different things.
My setup is local-first, not local-only. The state lives on my hardware. The memories, the embeddings, the graph relationships, everything my agents have learned about my projects, all of it sits on a box I own, in a Docker volume I can
tarand carry away. Nobody can deprecate it, price-hike it, or sunset it.The inference does not. My embedder points at Mistral’s managed API. I’ll get to why, and how to swap it, but I’m not going to pretend otherwise in a post about self-hosting.
That embedder is the only thing that leaves my network, and only when something actually gets embedded, so when writing a memory and searching for one. Listing, deleting, and every graph operation are local with zero API calls.
State is what you can’t get back. Compute is a commodity you rent by the token. Losing access to an API means switching providers. Losing two years of accumulated project context means starting over.
The Four Pieces
Qdrant is semantic search. When I ask what it remembers about my package manager preferences, Qdrant turns that into a similarity query and hands back the relevant memories. It’s Rust, it’s fast, and at 28MB resident it’s essentially free to run. One gotcha: vector dimensions are fixed when the collection is created. Swap embedding models and you need a new collection, not a migration. I learned that the annoying way.
Neo4j is the graph store. Vectors are great at “find me things that sound like this” and bad at “what depends on what.” The graph holds explicit subject-predicate-object facts, so
project Xbuilt_withPython 3.13is a traversable edge instead of a fuzzy match. It’s the heavy one at 613MB, but it’s a JVM, so that’s mostly heap floor rather than working set. If you’re squeezing onto the smallest possible VPS, interrogate this one first.mem0 is the orchestration on top: what gets extracted from a conversation, what gets deduped against existing memories, what gets written where. That’s the difference between a database and a memory system.
The MCP server is what makes any of it useful. A small Go binary that speaks Model Context Protocol over stdio to Claude Code, exposing eight tools:
remember,recall,list_memories,forget,memory_stats,add_relation,recall_related,forget_relation.The topology is deliberately boring:
Mac Home server ┌──────────────┐ ┌──────────┐ ┌─────────────────┐ │ Claude Code │◄─►│ mem0-mcp │ LAN │ Qdrant + Neo4j │ │ │ │ (Go) │───────►│ │ └──────────────┘ └──────────┘ └─────────────────┘ stdio HTTP + boltClient binary on my laptop, containers on a box. No cloud in the middle, no account, no dashboard, no seat license.
The Ansible Role Is the Whole Argument
Anyone can
docker compose upa stack once. That’s a weekend, not infrastructure. What makes this real is that it’s a role in a repo, and rebuilding it on a fresh box is one command:ansible-playbook -i common_hosts home.yml --tags mem0That role does the unglamorous work:
- Installs a read-only deploy key scoped to exactly one repo, with an SSH
Hostalias so it can’t collide with my personal GitHub key - Clones and updates the source at a pinned branch
- Templates a
.envwith secrets pulled from Ansible Vault,no_log: trueso nothing leaks into terminal output on a--diffrun - Brings up the compose stack with
remove_orphans: true, so when I dropped a service upstream, the stale container went with it instead of lingering forever
Be careful with your secrets and how you are creating your .env files!
The Honest Part About the API Key
I self-host the state and rent the inference. Two reasons.
The first reason is speed. I ran embeddings locally before this, on CPU, and it was miserable: roughly 87 seconds to embed 32 memories, against about 2 seconds through a hosted API. That is a 45x difference on an operation sitting directly in the path of every
rememberandrecall. A good model on a CPU is still a slow model, and this was never a quality problem.The second is that embeddings have gotten cheap enough that not worth the time to setup your own embedding service. Mistral charges $0.10 per million tokens for
mistral-embed. Google’sgemini-embedding-001is $0.15 per million, halved on their batch API. Both are good models. Both bill you.Cloudflare is the worth knowing about if you’d rather not pay at all. Workers AI includes 10,000 neurons per day free, on the free plan as well as the paid one. Neurons are their normalized compute unit, and
bge-m3costs 1,075 of them per million input tokens — so that daily allowance is roughly nine million tokens a day, at no cost. Past it you’re at $0.012 per million, which is an order of magnitude under the paid competition. For a personal memory layer, nine million tokens a day is not a trial. It’s just free.One detail if you’re swapping:
bge-m3emits 1024-dimension vectors, the same asmistral-embed. Go back to that Qdrant gotcha — matching dimensions means your existing collection still works. Mismatched ones mean starting over.And the escape hatch is already built. The env vars in my role are
TEI_BASE_URL,TEI_MODEL,TEI_DIMENSIONS. Generic OpenAI-compatible embedder knobs, named after Text Embeddings Inference for historical reasons and pointed at Mistral today. Aim them at a self-hosted TEI container, at Ollama, at anything speaking that shape, and the rest of the stack doesn’t notice.That’s what local-first buys you. Not purity. Optionality.
So, the $5 Box
My server has 60GB of RAM, which is absurd overkill and exists because it does a dozen other things. The stack itself measured 640MB with three weeks of uptime, essentially zero CPU at idle.
That fits comfortably on a small cloud VPS in the few-dollars-a-month range. Check current pricing yourself rather than trusting a number in a blog post, but the shape is: a 2 vCPU / 4GB instance from Hetzner or similar costs less per month than one seat of most AI memory SaaS products, and you get to run everything else on it too.
Your real constraint is RAM, specifically Neo4j’s JVM floor. On a 1GB instance you’d be fighting it. At 2GB you’re fine. At 4GB you’ll forget it’s running.
Why I Care
The indie web ethos is about noticing that renting your identity from a platform means the platform decides what happens to it.
We’re about to make the same mistake with agent memory, except worse, because the thing being accumulated this time is a working model of how you think and what you’re building. Every “our AI remembers you across sessions” product is a proposal that you deposit that into someone else’s database and hope the pricing page stays reasonable.
Qdrant is Apache 2.0. Neo4j Community is GPL. Docker Compose is a YAML file. Ansible is idempotent YAML. Nothing in this stack is exotic. The barrier to owning your agent memory is an afternoon and 640 megabytes.
Not everyone needs this, and I’m not going to pretend a solo dev with three side projects is being exploited by a $20 subscription. But if you’re accumulating context you’d be genuinely sad to lose, the math changes. Own the state, rent the compute, and keep the role in version control so the whole thing is reproducible on a box you haven’t bought yet.
Moving the embedder onto Cloudflare’s free tier is next on my list, what’s on yours?
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
- Installs a read-only deploy key scoped to exactly one repo, with an SSH
-
If I Worked at Apple Again, I'd Build iApps
I worked as a contractor at Apple more than a decade ago. The specific work doesn’t matter, but if I walked back through the doors today, I know what I’d pitch.
Call them iApps, just as a thought experiment.
An iApp would be a small interactive tool generated for one specific problem. You ask Apple Intelligence for help, it gives you the research, and when the answer needs a calculator, form, checklist, map, or dashboard, it builds that too.
Then you can send the whole thing to somebody else, and it still works.
The Garden Calculator That Got Stuck in an HTML File
My wife is interested in turning part of our front yard into a stone-edged garden with more wildflowers. So I asked Claude to help me think through it.
It pulled together the HOA rules we should check, possible city permits, who to call, and some flower ideas. Then it built a calculator for the border. Give it the dimensions, choose chopped limestone, stacked flagstone, or river boulders, and it gives you a rough material estimate.
This was useful. It was also an HTML file with a little JavaScript inside it.
I can send my wife the research as text. I can send her the HTML file and explain how the JavaScript doesn’t work when it’s just a file. I am not hosting the html file somewhere just for her to open it. I don’t even know if its useful yet.
No-ThnX.
What I want to send is the answer, including the interactive part. She should tap it in Messages, see where the information came from, adjust the garden dimensions, switch the border material, and get a new estimate. No hosting. No App Store listing. No explanation about Safari sandbox for iMessage sucks.
That is an iApp.
An App That Behaves More Like a Document
An iApp would sit somewhere between a static AI answer and a full application.
It would not have a bundle identifier, a developer account, three subscription tiers, or a product manager asking about quarterly engagement. It might live for ten minutes, get saved until next spring, or be shared once and never opened again.
Think about the small tools we make and discard all the time:
- A calculator for comparing other garden ideas
- A packing checklist that changes with the weather and trip length
- A scorekeeper for a game your family plays twice a year
- A trip comparison with filters for price and travel time
None of those ideas deserves a conventional software project. They do deserve something better than a wall of generated text.
The iApp should be portable, more like a document than an installed app. It contains the interface, data, sources, and a declaration of what it can do. Send it through iMessage or AirDrop, save it in Files, or share an iCloud link. The receiving device validates and renders it.
The AI Should Generate a Recipe, Not Code
The obvious implementation is to let the model write HTML and JavaScript, then open it in a web view.
That is also how you turn every shared iApp into a tiny supply-chain attack.
I would not let the model generate Swift, JavaScript, WebAssembly, or anything else the device executes directly. It should generate a versioned document against a strict schema: text, fields, pickers, tables, charts, buttons, and a deliberately boring set of formulas and actions.
Perhaps something like this:
{ "type": "calculator", "fields": ["length", "width", "border_material"], "result": "estimated_material", "capabilities": [] }Apple would then own the renderer. A fixed SwiftUI component library turns that document into the interface. A formula engine handles arithmetic without
eval. The model supplies the recipe, and the operating system decides how the recipe is allowed to run.Pieces of this already exist. Apple’s Foundation Models can or should be able to produce Swift with very little guidance. With App Intents you can expose pre-declared system actions to enable built-in device functionality like sharing a link.
The parts exist. The product does not.
Make the Permissions Painfully Obvious
Most iApps would need no permissions at all. The garden calculator needs its bundled research, a few numeric fields, and some arithmetic. It does not need my contacts, photos, location, calendar, microphone, or the ability to phone a server in the background.
When an iApp does need more permissions, it should declare that capability before it opens… probably?
I could see the permission screen being a barrier to most normal people. who aren’t really going to care or understand the implications of granting it access to your calendar I think we Apple would need to build some sort of like end CP like layer on top of their native APIs and then build like a d a detection system into like you know is this Application requesting one event or all events. And uh, you know, denying access based on behavior, basically.
The research and the generated information would need citations. So you yeah, you would need like a Web researcher remote agent with sandbox rules that is able to Properly identify claims made by The intent or the output and verify the claims.
The Category Already Exists
The idea is not new. Claude already builds interactive artifacts for you. Google’s Opal makes mini apps inside the Gemini web app. Microsoft’s App Builder makes them inside Microsoft 365. You describe the tool, the model builds it, and you share it.
Some of them share better than others. Publish a Claude artifact and anyone can open the link with no account at all. Opal wants a Google sign-in. App Builder stays inside your Microsoft tenant.
But they all share the same way: as a URL pointing at somebody else’s server. However, a link is not a document. It needs the vendor to still be running, the page to still be published, and my wife to have the right app to run it. Publishing it means putting a calculator about our flower bed into the “cloud” somewhere.
What I want to send is the thing itself, sitting in the message thread on her phone. Not a pointer to a copy that lives somewhere else.
Apple Is Uniquely Positioned to Do This
Apple controls the full stack. People also trust an iPhone to open strange things their family sends them without turning every message into a security review.
iApps would make the operating system the host. Apple could define the component catalog, broker every capability, revoke a dangerous document, and keep generated behavior inside a box designed for it from the beginning.
This does not replace the App Store. A real product with customers, accounts, payments, and years of maintenance will continue to exist. iApps are for the space below that threshold, where a problem is worth solving but not worth turning into a company.
I would build that.
Sources
- Guided generation — validated Swift data structures.
- App Intents — defined actions exposed to system experiences.
- SwiftUI — declarative Apple interfaces.
- ShareLink — system sharing for custom data.
- App Clips — lightweight parts of existing apps.
- Anthropic Artifacts — interactive apps made and shared through Claude.
- Google Opal — shareable AI mini apps.
- Microsoft 365 Copilot App Builder — interactive apps made inside Copilot.
Disclosure: I have no inside knowledge and no idea whether Apple is working on anything like this. If it eventually announces something similar, that is a happy coincidence, not a leak.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Your Agent Needs a Dollar Limit, Not a Token Budget
If you let an autonomous coding agent run in an unbounded loop, I have bad news for you, or rather, your wallet.
It happens easily. An agent gets handed a task, runs into an unhandled error or a failing test, and gets stuck in a retry loop. It re-reads the same files, attempts the same broken patch, and streams tokens the whole time while nobody is watching.
We solved this problem in cloud infrastructure ten years ago. Nobody deploys a Lambda function without an execution timeout. Nobody configures a Horizontal Pod Autoscaler without setting
maxReplicas. Let a container run wild without bounds and your infra team will revoke your deployment credentials before breakfast.Yet here we are, handing autonomous agents full access to our terminal and our API keys with a carrot “go do the task, good luck!”
If we want agents to be production-grade tools, we need to treat token spend like compute spend. The most important feature an agent harness can ship is a first-class, per-task spend ceiling.
Claude Code Shows the Path
Anthropic is paving the way here, and it’s worth talking about.
In Claude Code, you can set a USD ceiling on a single invocation using the
--max-budget-usdflag:claude -p --max-budget-usd 2.00 "refactor auth module"This flag only works in print mode — that’s the
-pabove, which is short for--print.This cap is apparently aware of any fan-out that might occur from subagents, so spend on subagents counts against the same ceiling. Claude Code will then kill the background subagents that are still running if it hits the budget limit. For this feature to work, you need to be running Claude Code v2.1.217 or later.
So if you’re building CLI harnesses on top of Claude Code, this will be a huge quality-of-life feature for you to implement. This way you can kick off a background task and rest assured that the harness will not result in a big surprise on your API bill.
The other one is not a cap at all
Anthropic also has something on the raw API side called the task budget.
However, it is not the same and it will not protect you. Task budgets are in beta, and they hand the model a token allowance for it to run its full agentic loop. It tries to wrap things up gracefully rather than being cut off in the middle of a tool call.
The task budget on the API side is in tokens, not dollars. So it’s fine for getting an idea of whether something is possible within a given token budget, but it’s not going to save you from any surprises on the API bill side of things.
The Gap in Codex and OpenCode
The rest of the CLI agent ecosystem hasn’t caught up.
OpenAI Codex CLI (
codex exec) has no native--max-budget-usdflag or budget setting. You can pin a cheaper model profile, but you cannot set a hard dollar limit on a per-task basis.OpenCode (
opencode run) is in a similar spot, which is strange, given that OpenCode has done a great job of adding features to their CLI harness. Unfortunately, there’s no way to pass a pre-execution cap to OpenCode before you launch a task. It kind of feels like a missed opportunity, or one that they will add soon, given that OpenCode already tracks consumption inside the CLI if you’re using it directly.
How We Hack Around It Today
So how do you enforce a dollar cap on non-Anthropic models right now? You push the problem down a layer and let a gateway handle it — which is one more reason your AI stack probably wants a gateway anyway.
The infrastructure side of things has solved this already with the proxies that are available. They expose the functionality that you need in order to set hard per-key budgets. LiteLLM Proxy will start rejecting calls after that budget has been exceeded, with a
400and abudget_exceedederror type. If you’re using Cloudflare AI Gateway, they shipped a dollar-denominated spend limit in June of this year, which returns a429once you cross the line. You can scope it by model, provider, or other custom metadata, and you can configure it to fail over to a cheaper model instead of blocking, which is a nice to have.
Shift-Left till you get to FinOps
FinOps is what happens when you keep shifting left.
Eventually, we’re going to get tired of the bill.
The gateway vendors have come prepared, and the CLI harnesses have yet to fully adopt a decent token/thinking/dollar budget flag system.
We’ll get this figured out one of these days.
Sources
- Claude Code CLI reference —
--max-budget-usd, its print-mode constraint, subagent spend counting toward the cap, and the v2.1.217 enforcement requirement; verified locally againstclaude --helpon v2.1.220, 2026-07-26. - Anthropic: Task budgets — the advisory, token-denominated API feature; source of the “soft hint, not a hard cap” language and the note that task budgets are unsupported on Claude Code.
- OpenAI Codex CLI: local inspection of
codex exec --help, 2026-07-26 — no budget or spend-cap flag. - OpenCode CLI: local inspection of
opencode run --helpandopencode stats, 2026-07-26 — post-run cost reporting, no pre-run cap. - LiteLLM: Budgets, Rate Limits — virtual key
max_budget,duration, and thebudget_exceededrejection. - Cloudflare AI Gateway: Spend limits — dollar budgets scoped by model, provider, or metadata,
429on block, optional cheaper-model fallback. - Your AI bill is out of control. Cloudflare can fix it now. — the June 2026 launch announcement.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
- Claude Code CLI reference —
-
How I Would Build Observability for an Autonomous Agent
I have not built a full production observability stack for an autonomous agent.
I’ve built lots of small wrappers around existing coding harnesses. I have a pretty good idea how quickly their output can turn into a wall of model responses, tool calls, and subprocess logs. But I have not run LangChain across a Kubernetes cluster or operated an LLM router at scale.
So this is not a postmortem. It’s a design exercise.
If one of my wrappers became an always-on agent service tomorrow, what would I need to see when it failed? Where would I put that telemetry on Azure, Railway, or Cloudflare? And which parts of the design should stay the same no matter where it runs?
Start With the Trace, Not the Platform
An agent run is a distributed trace hiding inside a loop.
There is a request that starts the work. The agent calls a model, the model requests a tool, the tool talks to another service, and the result goes back into the model. Repeat that enough times and a normal application log becomes hard to follow because the interesting question is not just, “What failed?” It is, “What sequence of decisions got us here?”
I would use OpenTelemetry and give every run one root trace. Each model call and tool call becomes a child span. Structured logs carry the same trace and span IDs, so I can move from a failed run to the exact log event without searching timestamps and hoping I found the right one.
The minimum useful event would look something like this:
{ "event": "agent.tool.completed", "agent_run_id": "run_01K0...", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7", "step": 7, "tool": "github.get_issue", "duration_ms": 842, "status": "error", "error_type": "rate_limit" }That gives me four things I can debug:
- The run: which request or scheduled job started the work?
- The model: which provider and model answered, how long did it take, and how many tokens did it use?
- The tool: what operation ran, how long did it take, and did it succeed?
- The sequence: what happened immediately before the failure?
I would keep an
agent_run_idbecause it is useful in a UI and support ticket, but I would not use it as a replacement for a trace ID. The trace context needs to travel across HTTP calls, queues, model gateways, and anything else the agent touches.Do Not Log Everything
The tempting version of agent observability records every prompt, response, tool argument, environment variable, and file body. It is also a convenient way to copy credentials and private data into the system with the broadest internal access.
My default would be structural telemetry first. Model name, token counts, latency, tool name, result status, retry count, and safe error categories are useful without storing the full content.
Prompt and tool payload capture would be an explicit policy, not a debug switch someone leaves enabled for six months. I would use an allowlist, redact before export, encrypt whatever remains, and give raw payloads a shorter retention period than ordinary metrics.
This also answers where an AI gateway fits. The gateway is the natural place to record model latency, provider errors, and token usage. It cannot see the agent’s local decisions or tool calls, so it is one part of the trace, not the entire observability system.
The Same Design in Three Places
The trace model stays the same. The deployment choice changes how much infrastructure I have to own.
Azure: The Integrated Option
If the agent already lived in Azure, I would use the Azure Monitor OpenTelemetry distribution and send telemetry to Application Insights, backed by a Log Analytics workspace. That gives the system a managed place for traces, logs, metrics, exceptions, queries, and retention policies.
Microsoft Foundry can also trace model calls, tool calls, intermediate steps, tokens, latency, and errors into Application Insights. I would treat that as an accelerator, not an excuse to skip the application-level design. Some of its agent tracing paths are still in preview, and content tracing can include sensitive prompts and outputs.
Azure is the option I would choose for an organization already paying the Azure complexity tax. It has the most integrated telemetry path of these three and the governance controls a larger company is likely to ask for. For a personal agent prototype, it is probably more platform than I need.
Railway: The Straightforward Container
Railway is where I would start for a small agent that needs a normal process, a Docker image, background work, and the freedom to use ordinary libraries.
I would instrument the application with OpenTelemetry and export traces over OTLP to an external observability backend. Railway’s built-in logs and container metrics are useful for deployment health, CPU, memory, disk, and network usage. They are not a distributed tracing backend for the agent itself.
That split is fine. Railway runs the service, while the application owns its telemetry schema and exports it somewhere designed to query traces. I would also make sure the exporter flushes on
SIGTERM, because a clean deployment is not helpful if the final spans disappear during shutdown.This is the least ceremonial option. It is also the one where I would have to make a separate decision about the telemetry backend and its retention cost.
Cloudflare: The API-Oriented Agent
Cloudflare gets interesting when the agent mostly calls models and HTTP APIs instead of running shell commands against a workspace.
Workers can collect logs and traces automatically, and Cloudflare can export both in OpenTelemetry format to an external destination. I would still add application spans for the model and tool semantics because automatic request tracing cannot know which prompt, tool, or agent step matters to me. The OTLP export is also currently beta and does not export Worker metrics.
For state, I would use a Durable Object when one run needs a single coordination point, D1 for relational records across runs, and R2 for larger artifacts or archived transcripts. If the agent needs retries, sleeps, or a human callback, Cloudflare Workflows is a better fit than trying to keep one HTTP request alive.
I would not put a coding agent that needs arbitrary processes, a writable repository, and long CPU-heavy tool calls into a Worker just because the edge sounds nice. Cloudflare’s runtime has real CPU, memory, connection, and logging limits. For that workload, I would run the agent in a container and consider Cloudflare for the gateway or API edge instead.
What I Would Pick
For my first version, probably Railway. It matches the small wrappers I already build, lets the agent behave like a normal application, and leaves me free to try different tracing backends without moving the workload.
I would pick Azure when the surrounding organization already uses Azure Monitor and needs one managed governance story. I would pick Cloudflare when the agent is an API orchestrator and its tools already live on the network.
A mistake would be choosing a platform or dashboard first and calling that observability solved. I would start small with a trace per run or tool call with structured logging turned on, but no prompt content until there is a redaction or sanitization policy feature in place.
I haven’t built this stack yet. But that is how I would start building it.
Sources & References
- OpenTelemetry Logs specification — trace and span correlation in structured logs.
- Azure Monitor OpenTelemetry — supported telemetry signals and Application Insights setup.
- Microsoft Foundry tracing — agent trace contents, storage, and data-handling considerations.
- Railway third-party observability — application-side OpenTelemetry export and shutdown guidance.
- Railway metrics — native container and service metrics.
- Cloudflare Workers traces — automatic tracing and native retention.
- Cloudflare OpenTelemetry export — OTLP trace and log export capabilities and current limitations.
- Cloudflare Durable Objects — per-instance coordination and transactional storage.
- Cloudflare Workflows — durable multi-step execution, retries, and external events.
- Cloudflare D1 — relational storage for records shared across runs.
- Cloudflare R2 — object storage for larger artifacts and archived transcripts.
- Cloudflare Workers limits — CPU, memory, connection, and log limits.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Your Justfile Is Your Repo's API
Ask your agent to run the tests in a repo it has never seen before and watch what happens. It’s gonna load so many things into the context to try to figure it out, digging through the repo to find the right command and what framework you’re using.
What if you could standardize on a way to run the tests no matter the language or the framework?
just testIt’s a statement. It’s an interface. Sure, it’s a shortcut.
I wrote about Just as a command runner back in March.
After months of using it on a daily basis with agent-driven work, I’m here to tell you that it’s an incredibly useful cog in the wheels of complexity.
Your justfile can become what is essentially an operational API for your repository. You use it, your agents use it, the CLI calls it. Everything underneath it you can change as much as you want, as long as the command keeps its promise.
The Command Is the Contract
An API lets a caller ask a system to do something without understanding every detail inside it. A justfile does exactly that for application operations.
When I run
just test, I’m not asking Just to test the software. I’m asking the repository to perform its official test operation. Today that recipe might be:test: lsm exec -- uv run pytestThose few words hide several decisions. The project uses
uv, tests run throughpytest, secrets come throughlsm.Whoever calls that doesn’t need to rebuild or reconstitute any past decisions that went into that command. And the recipe can change as long as what happens when I run the command doesn’t. This is what makes adding Just to your application a proper interface and not just an alias.
Well, actually, if I were in charge of making a motto, I think the tongue-in-cheek version should be
just an alias.The Justfile Connects the Tooling Layers
Modern repos can have several tools doing different jobs, and that’s fine.
- Mise can pin your Python version, and it can operate as your task runner.
- uv manages the Python environment and runs commands inside it.
- LSM is more of a me thing, but it provides secrets when an operation needs them.
- The application CLI holds the actual business behavior.
- Just If you’re already using Mise as your task runner, you may not need Just at all.
Modules Turn Commands Into Namespaces
Once a project has more than a handful of operations, one flat list becomes a junk drawer. Just supports modules, so the root justfile can declare:
mod blog mod links mod db mod booksEach module owns its related recipes, and the result reads like a small command-line app:
just blog status just db backup just books unresolvedThe manual describes module recipes as subcommands. The hierarchy helps humans discover the interface, and it gives agents a predictable way to narrow down the operation they want. It also kills the naming nonsense of flattening every domain and action into one giant alphabetical list. The namespace carries the context.
Hide Plumbing, Not Consequences
An API should make a system easier to use. It should not disguise what the system does.
A recipe can absolutely hide the auth wrapper, the environment setup, and the CLI invocation. It should never make a destructive production operation look like an innocent local check. Good recipe names describe intent, so the caller knows which commands read, write, publish, restore, or preview. Dangerous workflows still need real validation, permissions, confirmations, and backups.
A justfile is an interface. It is not a place to dump everything.
A Few Rules to Follow
Sometimes it’s obvious.
Name recipes after intent.
test,backup,publish,status. Nottest-pytest-with-lsm— it’s hard to tell what that actually does.Keep one canonical path. If
just test, Mise, a shell script, and the README all run different tests, you have four contracts and no interface.Pass through useful arguments.
just books fetch --dry-runpreserves application options without a new recipe for every flag.Make discovery useful. Comments plus modules turn
just --listinto documentation.Keep recipes thin. Branching rules, error handling, and database logic belong in tested application code.
The Boring Boundary Wins
Just still does the simple thing I liked back in March. It saves project-specific commands and runs them as recipes. The manual calls it a command runner, not a build system, and that’s correct.
The bigger value shows up when everyone agrees to call the same recipes. Humans stop memorizing setup. Agents stop reconstructing commands. CI and local development share an entry point. The tools underneath can churn without dragging things down.
Congratulations, that’s your application as an API.
Sources
- Just Programmer’s Manual — defines Just as a project-specific command runner and documents recipes, arguments, listing, and multi-file organization.
- Just modules documentation — documents
modstatements and invoking module recipes as subcommands.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
The Human Job Is Choosing What Not to Build
Coding agents have made one word much harder to say.
No.
You describe a feature, and the agent can already see the model, the migration, the command, the tests, and the docs it needs to touch. The whole thing sounds like an afternoon instead of a week.
So why not build it?
That question used to contain its own answer. We didn’t have the time or the people, and the feature wasn’t worth interrupting everything else for two weeks. Now the implementation might take twenty minutes. The old constraint is gone, but the need to choose is not. If anything, choosing matters more, because we can say yes faster than we can understand what all those yeses are doing to the product.
The agent’s job is to make the thing we asked for. The human job is deciding whether the thing should exist.
Yes Produces Better Receipts
Building something leaves evidence. There’s a diff. Tests pass. A new command shows up in the help output. You can take a screenshot, close an issue, and point at the feature. Work happened, and the artifacts prove it.
Choosing not to build produces almost nothing. Maybe you leave the idea in a notebook. There’s no demo for the dependency you avoided or the interface you never had to support.
That makes restraint feel less like engineering.
It can look like indecision, or lack of ambition, or an empty afternoon of planning.
A good “no” can preserve more value than a clean implementation.
You protected the shape of the product. You kept it understandable. You left room for the features and the users that actually matter.
Agents Make Local Ideas Look Great
Coding agents are strongest when the task is concrete. Add this flag. Support this file format. Create an adapter for this provider. Cache this response. Put a dashboard on top of these logs.
Given one of those, the agent inspects the local code and produces a reasonable path forward. It sees where the new feature fits.
What it doesn’t automatically carry is the opportunity cost.
You can put that context in the prompt, but somebody still has to decide how much product is enough.
A human has to protect the product.
Cheap Is Not the Same as Free
I recently argued that code is cheap now, but decisions are not. The mistake is pricing a feature only by how long the first implementation takes.
A twenty-minute feature can create a permanent interface, another concept every future agent has to understand, or a second way to do a task when the first way was already fine.
The implementation estimate answers: how quickly can we make this work?
The product decision asks: is this worth changing what the system is?
Four Ways to Say No
Not building something doesn’t always mean rejecting it forever. Here are four versions of no.
No, this is not the product
Some ideas are useful and still don’t belong. A small command-line tool doesn’t automatically need a web dashboard. A personal publishing pipeline doesn’t need multi-tenant permissions because it could theoretically serve a team. A library doesn’t need a plugin system before a second plugin exists.
These features may solve real problems. But they solve somebody else’s version of the product.
This is the cleanest no.
Write down the boundary and move on.
Not yet, we don’t have the evidence
Sometimes the problem is plausible but unproven. The database might need caching. The API might need another abstraction. Users might want a second export format. “Might” is doing all the work.
Wait for the system to produce evidence. Measure the slow query. See a second provider show up. Hear the same request from someone who actually has the problem. The agent will still be there when the need is real.
Deferral is only useful when it has a condition: “Revisit imports when we have enough data to support the decision.”
All the “maybe laters” are not backlogs; they are fossils that deserve to be buried.
Yes, but smaller
Plenty of ideas contain one valuable piece surrounded by a feature-shaped cloud.
Build something that you think is useful. Not everything needs a dashboard.
You don’t need a generic provider framework until you have a reason for the generic provider to exist.
You don’t need a rules engine if you’ve only got three rules.
Ask for the smallest change that provides the most value.
Narrow the solution before it becomes a broad one.
No longer
The hardest no is the one aimed at code that already exists. How do you determine whether it’s actually being used or not? This is how we get features that outlive the reason they were built. You added an experiment that became a supported path, and now everybody’s afraid to remove it.
Prune the branches of the product tree. Before you decide what to build next, make sure you’re not letting the system grow forever.
You should be asking yourself: what choices are you deciding against? Sometimes it’s important to know what already exists before deciding what should exist.
My Filter Before I Say Yes
Product committees suck, especially for small changes. Before I accept an idea as a feature, I ask myself the following questions. Or I should ask myself. Or I hope I ask myself. Whatever version of that makes sense for the day.
- What problem gets easier?
- Why now?
- What new promise does this create?
- What gets harder after this exists?
- Can a smaller change prove the value?
- What would make us remove it?
These help you tell the difference between what’s easy to generate and what’s worth owning. Sometimes they help you solve for a hypothetical problem. It’s a good thing to try to figure out whether the tests will pass before you build them.
Taste Is the Remaining Bottleneck
As implementation gets cheaper, the scarce skill becomes taste.
Not taste as in fonts and rounded corners. Taste is recognizing when a product has enough concepts. It’s choosing the boring interface people can understand. It’s seeing that a flexible abstraction makes the current problem worse. It’s knowing which rough edge gives the tool character and which one just wastes time.
Then decide.
The future of software isn’t a world where we finally build every idea in the backlog.
That sounds exhausting.
The future of software is a world where we get to decide, and be honest about which ideas were actually good to begin with.
Don’t forget you’re allowed to say no.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].