I’ve watched thousands of tests run over the years.

When they pass it has felt good. A huge screen of green checks in the terminal is incredibly rewarding as a software developer.

But the work is never finished.

The tests are always valuable. They are evidence, but they were never the definition of done.

Tests Prove the Claims We Give Them

I’ve written before that testing is how we show our work, and I still believe it. If software is meant to run more than once, test it. No excuses, just testing.

The mistake is treating a passing suite as proof of more than it covers.

A test makes one specific claim about a system:

given this setup
when this action happens
then I expect this result

Run enough of those claims and you build confidence. You know the parser handles a missing field. You know the repository preserves an identifier. You know the command returns a failure code when authentication breaks.

What you do not know is whether you wrote the right claims.

If the requirement was misunderstood, the implementation and its tests can agree perfectly. If your fixture contains a field the real API never sends, every parser test can pass while production fails. If your mock accepts a request the provider rejects, you’ve proven compatibility with the mock.

That’s useful, but let’s not get carried away.

The Agent Can Test Its Own Misunderstanding

Coding agents make this distinction more important, because they generate code and tests together.

You ask for a feature. The agent interprets the request, implements that interpretation, then writes tests asserting the implementation behaves exactly as it wrote it. Everything passes.

Sometimes that’s a clean, efficient workflow. Sometimes it’s one misunderstanding with excellent test coverage.

Picture asking an agent to add a published_at field from an external service. It sees another timestamp called created_at, assumes they mean roughly the same thing, and uses one as a fallback for the other. Then it writes tests proving the fallback works.

The suite is green. The code is wrong.

The issue isn’t that an agent wrote the tests. Humans have spent decades writing tests around their own bad assumptions. The difference is speed. An agent can turn a vague requirement into a thoroughly tested wrong answer before you notice the semantic choice it made.

More tests don’t rescue a bad premise. They preserve it.

Mocks Prove You Understand the Mock

Mocks are one of the best tools we have for keeping tests fast and deterministic. I use them a lot. I don’t want every local run hitting a real provider, burning rate limits, changing remote data, or failing because somebody’s service is having a bad Tuesday.

Still, a mocked integration is a model of reality, not reality.

Your fixture may be stale. The provider may omit fields you marked required. Auth may use a header you never modeled. Pagination may stop differently than the docs imply. Error responses may arrive as HTML because a proxy had opinions.

Unit tests prove your code handles the world you described. A live, read-only check tells you whether the world still resembles that description.

That doesn’t mean turning the whole suite into live integration tests. It means picking a small amount of extra evidence proportional to the risk:

• Fetch one real response and inspect the fields you depend on.
• Exercise authentication without modifying remote state.
• Run a dry run through the production code path.
• Validate a migration against a realistic database copy.
• Confirm the deployed service reports the expected version.

Tests stay fast. Reality gets a vote.

Passing Is Not the Same as Shipped

There’s another gap between tested and done that has nothing to do with correctness.

Code can pass every check and still exist only in a working tree. A migration can be valid but unapplied. A config change can be committed but missing from the deployment environment. A feature can reach production without the logs you’d need to understand its first failure.

This sounds operational because it is. Software isn’t finished when the implementation works in the place where it was written. It’s finished when the intended system has the change, and you can tell whether that change is healthy.

For a small personal script, that might mean committing it and running it once with real input. For a web service, deployment, a health check, logs, metrics, and a rollback path. For a database change, verifying both the schema and the application behavior after migration.

The evidence changes with the risk. The principle doesn’t.

pytest cannot tell you whether you forgot to push the commit.

Done Is a Decision Built From Evidence

A definition of done should answer a broader question than “did the tests pass?” It should answer: what evidence would make us comfortable owning this change?

For most work, I look in a few categories:

1. Intent: The behavior matches the actual requirement, not the first interpretation of it.
2. Implementation: Automated tests cover the important paths and failure cases.
3. Quality: Static analysis, formatting, types, and review caught what they’re designed to catch.
4. Integration: Real boundaries behave the way our fixtures and mocks claim.
5. Operations: The change is delivered, observable, and recoverable in proportion to its risk.
6. Durability: Code, migration, docs, and task state are saved where the next person or agent can find them.

Not every change needs all of them. Fixing a typo doesn’t require a rollback drill. Changing how customer data is stored deserves more than one unit test and a thumbs-up from the agent that wrote it.

Good engineering is choosing the right amount of evidence, not applying the largest checklist to everything.

Green means the evidence looks good. Done means you have enough of it.

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].