Evals
-
Cost and Latency Belong in the Score
Congratulations, the best model available passed your eval. That’s not the question you should be answering. What other models could have achieved equivalent results? How much did it cost to run those other models? How long did it take for those other models to achieve equivalent results? All of these are the questions that you should be asking yourself.
If the only thing you’re measuring is if the output is correct or not, then we already know the ranking. The bigger models are going to score higher on the majority of tasks. You don’t need an eval harness to answer that question. Quality scoring is only part of the answer that you actually care about. On every eval run, you should be recording at least three things per task.
- Quality. Whatever your pass or fail criteria are. This is the part you already have.
- Cost. What the run consumed. Tokens in, tokens out, and the price attached to them.
- Latency. Wall clock, start to finish.
These aren’t particularly hard to capture. Cost and latency are usually right there in the response and you’re probably throwing them away. Latency is a hill to climb in its own right, given that it can vary depending on the reliability of the models. Latency at different times of day also matters more than at others. If the human operator is asleep, then latency matters less.
Once you start recording these three dimensions, the promise is you’ll be able to more accurately answer the question: which model is best?
I mean that’s the whole point of all this, right?
What you’re really building is a system, or an attempt at a system, for predicting the output of a non-deterministic system.
Good luck.
A global spending cap for the month doesn’t answer the question: are you using the wrong model for particular tasks?
Task routing is complicated. I can see it being a complex problem to try to solve.
You’re going to find that cheap models win more often than you’d think. There’s a whole bunch of types of work where you don’t need a frontier model, where you’re just using the wrong tool for the task at hand. Think about reformatting, or extracting data from structured text, or classifying something into a bucket, or summarizing a document, or renaming things. All of these are narrow tasks that don’t require a lot of reasoning between the input and the output.
One thing to try is to do it with the cheap one first and then escalate to a larger model if it fails. You’ll often find that scaling up is a lot easier than scaling down in terms of model intelligence.
The takeaway here is that cost and latency belong in or alongside quality in an eval harness.
Once you start recording all three, then you can start measuring and deciding between models.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Your Vibes Are Not an Agent Eval
Do you really know what you’re doing? You swapped a model, you tuned an agentic workflow, you had the agent rewrite a chunk of a system prompt. You added a skill. Now the output feels sharper. Is that feeling a measurement? No, it’s an impression. This is when the vibes start seeping into your agentic engineering world view.
How Impressions Fail
A subjective assessment isn’t necessarily useless, since it’s how you can notice something is wrong in the first place, and it can be a good signal, or an early signal that leads to a corrective action.
A few things here conspire to work against you.
Recency. You remember the last three runs vividly and the forty before them not at all. If the last three happened to be easy tasks, the model got better. If they were gnarly, it got worse.
Confirmation. You just spent an hour rewriting a prompt. You are not a neutral judge of whether that hour helped. Nobody is.
Prompt drift. This one is sneakier. You’re not asking the same thing you asked last month. Your prompts got better because you got better at prompting, and that improvement gets silently credited to the model.
Task drift. The work changed. You were doing greenfield scaffolding in June and you’re doing debugging in August. Those exercise completely different capabilities, and comparing across them tells you nothing.
All of these will sneak up and bite you in the ass. A decent working knowledge of the system is not a measurement.
The actual risk with an agentic workflow isn’t a sharp and dramatic decline in quality. It’s a slow regression over time as you start missing things that slip through the cracks when you’re not paying attention as closely as you should on that day.
I talked about evals that are worth building in previous posts. You should go have a look at some examples there on how to get started.
How do you test a harness? You need to separate the model from the harness. It turns out the harness changes frequently along with the model. Is it even worth testing the harness?
A model swap tripwire is a good place to get started. A tripwire asks whether this specific change made things worse. It’s a binary operation. You run it before the swap, save the results, and run it again after the swap, and compare the results. Same task, same prompt, only the model changed.
If you keep going on vibes, they will keep telling you things. That may or may not matter. At the end of the day, vibes are a decent smoke alarm, but make for a terrible way to measure quality.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Start With Ten Tasks You Actually Do
Public benchmarks of large language models are a fine way to compare models in the abstract, but they’re close to useless for answering questions about things that actually matter. Generally, it’s helpful to know which model is the best in general at a specific benchmark, but it doesn’t answer the question of which model is the best at that specific thing you ask it to do all day long.
You need to build a golden set. Ten tasks where you understand the input and the output.
Scoring the code is not the hard part. Picking the ten tasks is going to be the hardest thing. How do you pick something that has a true pass or fail, but also applies to your specific problem?
The temptation is to sit down and try to come up with representative tasks. Chances are you’re going to waste a ton of time and not produce any better results.
Instead, you should be harvesting your tasks from a variety of sources.
What happens when the benchmark that matters is your last ten pull requests?
- Git history. What have you actually been changing? A month of commits will show you the shape of your work faster than introspection will.
- Issues and TODOs. These are tasks somebody already wrote down in task-shaped language.
- Prior agent sessions. If you have logs, this is the best source, because it’s literally the distribution you’re trying to measure.
- The things you retry. Anything you’ve asked an agent twice because the first answer was wrong is a high-value task. It’s already demonstrated it can discriminate.
A good eval task is one where you already know what the failure is and that it’s possible, because you’ve seen it fail.
Scoreable Means Checkable
So now you have your tasks, and it’s a different problem. You need to decide how to score the task and whether or not the agent got it right. This doesn’t necessarily mean you have to build automation from day one. Some good starting points that would qualify are the following:
- Tests pass or don’t
- Output parses or doesn’t
- The right files changed and no others
- A required field is present and well-formed
- The result matches a known-good output you saved earlier
Keep the amount of comparisons small. Don’t expand and keep evaluating. You can keep your old evaluations, but they shouldn’t impact future decisions forever.
Some amount of change in passing or failing over time is representative of a healthy set.
Here’s how you can get started.
Open your git log. Find things that the agent did well and things the agent could have done better. Write a definition of done.
You now have a small golden set that’s going to be more relevant and useful than any leaderboard online, because it was built from the results of your actual work.
The leaderboard tells you which model wins on average. You are not the average.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].