Public benchmarks of large language models are a fine way to compare models in the abstract, but they’re close to useless for answering questions about things that actually matter. Generally, it’s helpful to know which model is the best in general at a specific benchmark, but it doesn’t answer the question of which model is the best at that specific thing you ask it to do all day long.
\nYou need to build a golden set. Ten tasks where you understand the input and the output.
\nScoring the code is not the hard part. Picking the ten tasks is going to be the hardest thing. How do you pick something that has a true pass or fail, but also applies to your specific problem?
\nThe temptation is to sit down and try to come up with representative tasks. Chances are you’re going to waste a ton of time and not produce any better results.
\nInstead, you should be harvesting your tasks from a variety of sources.
\nWhat happens when the benchmark that matters is your last ten pull requests?
\nA good eval task is one where you already know what the failure is and that it’s possible, because you’ve seen it fail.
\nSo now you have your tasks, and it’s a different problem. You need to decide how to score the task and whether or not the agent got it right. This doesn’t necessarily mean you have to build automation from day one. Some good starting points that would qualify are the following:
\nKeep the amount of comparisons small. Don’t expand and keep evaluating. You can keep your old evaluations, but they shouldn’t impact future decisions forever.
\nSome amount of change in passing or failing over time is representative of a healthy set.
\nHere’s how you can get started.
\nOpen your git log. Find things that the agent did well and things the agent could have done better. Write a definition of done.
\nYou now have a small golden set that’s going to be more relevant and useful than any leaderboard online, because it was built from the results of your actual work.
\nThe leaderboard tells you which model wins on average. You are not the average.
\n\n\n", "date_published": "2026-08-06T10:00:00-05:00", "url": "https://llbbl.blog/2026/08/06/start-with-ten-tasks-you.html", "tags": ["AI","Testing","Developer-tools","Agentic-coding","Evals"] } ] }I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @logan@llbbl.blog.
\n