Ci

• Pinned Fixtures Need a Separate Freshness Check

  The logger’s conformance suite passed against a pinned fixture release that didn’t include the new cases its changes were meant to satisfy.

  The pin made the fixture selection explicit. It didn’t tell CI whether the selected fixtures were current.

  The fix kept the pin and added a separate check for upstream changes. It also added a guard against running the job without its fixtures.

  Keep the test input identifiable

  The logging spec and its fixtures live in the separate treering repository. The logger’s workflow checks out a named fixture release into .treering and runs the conformance suite against it.

  That is useful when investigating an old result. The workflow identifies the intended fixture release rather than asking whatever happens to be on the upstream branch at the time.

  A tag can still move, so a commit SHA is a stronger pin. Recording the resolved revision also helps establish exactly what ran. Pinning fixtures is one part of reproducibility, not a guarantee about the entire build environment.

  The problem was coverage, not the existence of the pin. A suite can pass all of its selected cases without including the case an implementation change was supposed to address.

  The older fixtures hadn’t ceased to exist or become invalid. They answered a narrower question than the review needed.

  Check freshness separately

  The added step fetches treering’s default branch and compares its fixtures/ directory with the pinned checkout. If the contents differ, the job fails and reports the difference along with the newest available fixture tag.

  That makes the mismatch visible without silently replacing the test input.

  It also introduces a deliberate dependency on current upstream state. Re-running an old commit can now produce a different freshness result after treering changes, even if the conformance result against the pinned fixtures remains the same.

  I want those results understood separately: the selected cases may still pass while the pin needs review. An upstream difference isn’t, by itself, proof that the implementation fails a new case.

  The workflow limits the comparison to fixtures/. That keeps unrelated repository changes from triggering this particular guard. But it doesn’t establish that specification prose is irrelevant. A prose change can alter or clarify the contract before a fixture captures it.

  The directory comparison detects fixture-content drift. It isn’t a complete check for every change in the specification.

  The failure message helps with the next step, but the newest tag is only a candidate to review. It might not contain every change on the default branch, and adopting it may require implementation work. Automatically changing the pin would skip that decision.

  Missing fixtures are another failure

  The runner allows a local contributor to run the library’s ordinary tests without checking out treering. In that case, it skips conformance tests.

  That can be reasonable locally. In a CI job whose purpose is conformance, missing fixtures must be an error.

  The workflow added this guard:

  test -f .treering/fixtures/README.md
  

  It catches a missing checkout at the expected path before the runner can skip the suite. The skipped tests don’t become passing tests; the risk is that the overall job can succeed without exercising them.

  The existence check is intentionally limited. A README doesn’t prove the fixture cases are present, parse correctly, or run. The runner still needs to validate its inputs and make the executed coverage visible.

  Both fixes preserve information the final green check couldn’t explain on its own. One says whether the fixture checkout is present. The other says whether its contents differ from upstream. The conformance run says whether the implementation passes the selected cases.

  Keeping those questions separate makes a failure easier to act on. It also makes a pass easier to describe without claiming more than the job checked.

  Sources

  • logan-logger-ts conformance workflow — the fixture pin, checkout guard, and freshness comparison.
  • actions/checkout — selecting a revision and checkout path.

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  Sep 14, 2026 DevOps Testing Github Ci

  Enjoyed this?