My Test Agreed With My Bug

The fix was one character. I appended "s" to a string. That’s the entire diff in the file that mattered:

func printTimeoutValue(timeout time.Duration) string {
	seconds := int64(math.Ceil(timeout.Seconds()))
	if seconds < 1 {
		seconds = 1
	}
	return strconv.FormatInt(seconds, 10) + "s"
}

Before that, it returned "120". After, "120s". The service had been deployed and broken for every job that used this path, and the test suite was green the whole time. That last part is the reason I’m writing this down.

The setup

I’d built a small HTTP gateway in Go that wraps a CLI tool. The whole point of it is that callers get almost no control. They can submit a prompt and an optional bounded timeout. They cannot pick the executable, the agent, the model, the output format, the working directory, any additional flags, or anything resembling shell syntax. The gateway assembles a fixed argv and runs it with exec.Command, no shell in between:

args := []string{
	"-p", prompt,
	"--agent", r.agent,
	"--output-format", "json",
	"--print-timeout", printTimeoutValue(timeout),
	"--mode", "plan",
}

I like this shape. It’s boring and it’s hard to abuse. The client-controlled surface is two fields, both bounded, and everything else is server-owned config.

And it was completely non-functional, because printTimeoutValue was producing a value the CLI refuses to parse.

The reproduction is two lines

I didn’t want to take my own commit message’s word for it, so I ran the actual binary:

$ agy -p "hi" --print-timeout 120 --output-format json
invalid value "120" for flag -print-timeout: time: missing unit in duration "120"

There it is. The flag is a time.Duration, registered through Go’s flag package, which parses with time.ParseDuration. That function wants a unit. A bare integer isn’t a duration, it’s just a number, and ParseDuration says so.

The tell was in the help text the whole time, and I’d read past it:

--print-timeout    Timeout for print mode wait (default 5m0s)

5m0s is what time.Duration.String() produces. If a CLI renders its default that way, the flag is a duration and it’s going to be strict about units. I looked at that line, saw “timeout”, thought “seconds”, and moved on.

The failure mode is worth noting too. This isn’t a runtime error partway through the work. flag rejects it during parsing, so the process died before it did anything at all. Every one of these jobs failed before inference started.

The part that actually bothers me

Here’s the assertion that was sitting in the test file:

want := []string{"-p", "ok", "--agent", "WebResearcher", "--output-format", "json", "--print-timeout", "5", "--mode", "plan"}

Read that carefully. The test checks that the gateway builds exactly the argv I intended. It’s a good test. It’s the right kind of test for this code, because the whole security story is “the argv is fixed and the client can’t touch it,” and that deserves a lock. It passed. It had always passed.

It passed because I wrote the assertion from the same wrong assumption that produced the bug. I believed the flag took an integer. So the code emitted "5" and the test demanded "5", and they agreed with each other perfectly, all the way into production.

A test that encodes your assumption doesn’t verify your assumption. It just makes it harder to notice. This is the failure mode that unit tests are structurally bad at: anything where the contract lives outside your process. My test knew what I meant to send. It had no idea what the other side would accept, because nothing in that test ever went near the real CLI.

And then the boundary hid the evidence

The second thing that went wrong is that this took longer to diagnose than a one-character bug has any right to take.

The gateway captures the subprocess’s stdout and stderr into size-limited buffers and, on a non-zero exit, throws stderr away:

if err != nil {
	return runnerResult{}, fmt.Errorf("antigravity exited unsuccessfully: %w", err)
}

That’s deliberate. The service’s own docs say public errors and structured logs must not include prompts, bearer tokens, CLI auth state, raw stdout, raw stderr, or raw diagnostics. I wrote that rule, I still think it’s correct, and it meant the operator-visible error was antigravity exited unsuccessfully: exit status 1.

The string time: missing unit in duration "120" was captured into a buffer and discarded, by design, every single time. The one line that explains the whole failure was right there and I’d built a machine to make sure nobody ever saw it.

I don’t have a clean resolution for that tension. Redacting subprocess stderr from public errors is the right default when the subprocess handles credentials. But “we never log it anywhere, at any level, for anyone” and “we redact it from the public error” are different policies, and I’d conflated them.

What I took from it

Five things, concretely:

  1. When a CLI prints a default like 5m0s, that flag is a duration. Go renders time.Duration that way. Read the default, not just the flag name.
  2. An argv assertion locks in your intent, not the callee’s contract. It’s still worth having. It just isn’t evidence that the command works.
  3. Non-zero exit before any output is a parsing failure, not a logic failure. Empty stdout plus instant exit means you never got started.
  4. Redaction policy and logging policy are separate decisions. Keep the subprocess’s stderr somewhere internal even when the public error says nothing.
  5. Regression coverage for a formatter should include the boring edges. The three cases I added were exact seconds (120s), a fractional value rounding up (1500ms becomes 2s), and the floor (zero becomes 1s).

The fix took a second. Finding it did not. If you’re wrapping someone else’s CLI, at least once, run the exact argv you’re generating and paste the output into the test as a comment. Your assertion can only ever be as right as you were when you wrote it.

Sources

  • time.ParseDuration — the parser behind duration flags, and the source of the “missing unit” error
  • flag.Duration — how a duration flag gets registered, and why it fails at parse time
  • os/exec — running a subprocess with a fixed argv and no shell

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

Testing Debugging Go