Public benchmarks of large language models are a fine way to compare models in the abstract, but they’re close to useless for answering questions about things that actually matter. Generally, it’s helpful to know which model is the best in general at a specific benchmark, but it doesn’t answer the question of which model is the best at that specific thing you ask it to do all day long.

You need to build a golden set. Ten tasks where you understand the input and the output.

Scoring the code is not the hard part. Picking the ten tasks is going to be the hardest thing. How do you pick something that has a true pass or fail, but also applies to your specific problem?

The temptation is to sit down and try to come up with representative tasks. Chances are you’re going to waste a ton of time and not produce any better results.

Instead, you should be harvesting your tasks from a variety of sources.

What happens when the benchmark that matters is your last ten pull requests?

• Git history. What have you actually been changing? A month of commits will show you the shape of your work faster than introspection will.
• Issues and TODOs. These are tasks somebody already wrote down in task-shaped language.
• Prior agent sessions. If you have logs, this is the best source, because it’s literally the distribution you’re trying to measure.
• The things you retry. Anything you’ve asked an agent twice because the first answer was wrong is a high-value task. It’s already demonstrated it can discriminate.

A good eval task is one where you already know what the failure is and that it’s possible, because you’ve seen it fail.

Scoreable Means Checkable

So now you have your tasks, and it’s a different problem. You need to decide how to score the task and whether or not the agent got it right. This doesn’t necessarily mean you have to build automation from day one. Some good starting points that would qualify are the following:

• Tests pass or don’t
• Output parses or doesn’t
• The right files changed and no others
• A required field is present and well-formed
• The result matches a known-good output you saved earlier

Keep the amount of comparisons small. Don’t expand and keep evaluating. You can keep your old evaluations, but they shouldn’t impact future decisions forever.

Some amount of change in passing or failing over time is representative of a healthy set.

Here’s how you can get started.

Open your git log. Find things that the agent did well and things the agent could have done better. Write a definition of done.

You now have a small golden set that’s going to be more relevant and useful than any leaderboard online, because it was built from the results of your actual work.

The leaderboard tells you which model wins on average. You are not the average.

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].