Congratulations, the best model available passed your eval. That’s not the question you should be answering. What other models could have achieved equivalent results? How much did it cost to run those other models? How long did it take for those other models to achieve equivalent results? All of these are the questions that you should be asking yourself.

If the only thing you’re measuring is if the output is correct or not, then we already know the ranking. The bigger models are going to score higher on the majority of tasks. You don’t need an eval harness to answer that question. Quality scoring is only part of the answer that you actually care about. On every eval run, you should be recording at least three things per task.

• Quality. Whatever your pass or fail criteria are. This is the part you already have.
• Cost. What the run consumed. Tokens in, tokens out, and the price attached to them.
• Latency. Wall clock, start to finish.

These aren’t particularly hard to capture. Cost and latency are usually right there in the response and you’re probably throwing them away. Latency is a hill to climb in its own right, given that it can vary depending on the reliability of the models. Latency at different times of day also matters more than at others. If the human operator is asleep, then latency matters less.

Once you start recording these three dimensions, the promise is you’ll be able to more accurately answer the question: which model is best?

I mean that’s the whole point of all this, right?

What you’re really building is a system, or an attempt at a system, for predicting the output of a non-deterministic system.

Good luck.

A global spending cap for the month doesn’t answer the question: are you using the wrong model for particular tasks?

Task routing is complicated. I can see it being a complex problem to try to solve.

You’re going to find that cheap models win more often than you’d think. There’s a whole bunch of types of work where you don’t need a frontier model, where you’re just using the wrong tool for the task at hand. Think about reformatting, or extracting data from structured text, or classifying something into a bucket, or summarizing a document, or renaming things. All of these are narrow tasks that don’t require a lot of reasoning between the input and the output.

One thing to try is to do it with the cheap one first and then escalate to a larger model if it fails. You’ll often find that scaling up is a lot easier than scaling down in terms of model intelligence.

The takeaway here is that cost and latency belong in or alongside quality in an eval harness.

Once you start recording all three, then you can start measuring and deciding between models.

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].