Programming
-
Where Should This Agent Knowledge Live?
Every agent has a junk drawer.
It usually starts with project instructions. Then I added build commands, personal preferences, database warnings, old architecture decisions, and things to fix later.
The agent could see everything if I wanted it to, but then it would have to read a small novel before touching the code, recurring workflows were buried between random facts, and completed work kept hanging around like it was still relevant. I had given the agent more context and somehow made it less informed.
The problem was not missing knowledge. The problem was putting every kind of knowledge in the same place. An instruction, a skill, a memory, and an issue can all be written in Markdown.
They do four completely different jobs.
Four Places, Four Jobs
A clean version looks like this:
What the agent needs Where it belongs A rule that must apply during every relevant session Instructions A reusable procedure for a particular kind of work Skill A durable fact that may become relevant later Memory A commitment that remains open until completed Issue tracker In practice, this is messier than a table… That’s where the engineering and attention to detail really matter.
So the useful questions are Does the agent need to know about this? and What bucket does this knowledge belong in?
Instructions Are Guaranteed and Expensive
Project instruction files are the things your agent loads automatically. Depending on the tool, that might be
AGENTS.md,CLAUDE.md, or another repository-level file.This is your guaranteed layer. The agent (or harness) doesn’t have to remember to search for it. If a session starts in the project, the rules are sitting in context.
Use that guarantee for knowledge that must shape nearly every relevant action:
- the preferred package manager and command runner,
- where the main source and tests live,
- dangerous commands that require explicit approval,
- the authoritative source for important data,
- mandatory validation before work counts as complete,
- a pointer telling the agent when to load a skill or recall a memory.
The guarantees come with a cost. Every line added to the context is loaded on every session, even when all you really need is a lightweight session where that context doesn’t matter.
Be diligent about cleaning up and maintaining your guaranteed context window, especially if you don’t have a memory layer in place.
Skills Are Procedures With Judgment
A skill answers a different question: how should the agent perform this kind of work?
Publishing a blog post, reviewing a pull request, applying a database migration, preparing a release, updating dependencies. Those are workflows. They have an entry condition, a sequence, safety rules, and a way to verify the result. That’s more than a fact. It’s operational judgment packaged for reuse.
Before we had skills, we had playbooks. Now we can make playbooks out of anything.
A good skill tells the agent when the workflow applies, what to inspect before acting, which steps and tools are appropriate, what must never happen silently, and what evidence proves the work succeeded.
Maybe the deployment instructions can now stay short; when doing a deployment, load the deployment skill.
Instructions are guaranteed. Skills are conditional.
Memory Is Context, Not Policy
Memory is where durable facts live without being injected into every session.
I prefer pnpm for JavaScript and most TypeScript projects. I prefer uv, and sometimes Poetry, for Python. These are facts that shouldn’t have to be repeated.
What about that time you had to troubleshoot an integration and observed some strange behavior? What about when you changed the database design and it broke the support layer? None of this deserves to be injected into every prompt, but it deserves a place where the details can be accessed later.
A semantic memory system can store and retrieve the relevant durable facts and give them to the agent when it asks. I described this earlier.
Memory can be large, and it can be flexible. But it’s also not guaranteed. The agent might not use the right keyword. You might have a problem with the vendor. A critical dependency could go down and take the memory system offline.
Don’t put safety-critical policies in memory. It’s good to have backups. If preferences get lost, they can be recreated, but absolutes like never print secret values belong in several places. If it has anything to do with security, cover your ass.
Memory is best for facts, preferences, relationships, explanations, and past decisions.
Issues Are Promises, Not Storage
An issue tracker tells the agent what needs to be done.
Issues have always been little documentation vaults. We write the history of the bug as it travels through the system. We link back to the issue as it maintains relevance.
An issue should preserve the context for a decision. It should act as a durable property of the project, recording the circumstances around a decision point.
Don’t make it a container for everything the agent did along the way, but I think it’s totally fine if you use it to publish an implementation plan.
Just, you know, you gotta read it.
Our job now is reading about software. The issue trackers are our corpus.
Route the Knowledge With Four Questions
When I don’t know where something belongs, these four questions can help.
1. Must the agent know this before it acts?
Instruction
2. Is this about performing a recurring kind of work?
Skill
3. Is this a durable fact that may help later?
Memory
4. Is this unfinished work or a commitment?
Issue
Sometimes the answer can be more than one place. But don’t copy the content blindly between locations.
All the files may be Markdown, but maintaining the architecture now means knowing where to put the information.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Every Feature Opens a Maintenance Account
Coding agents have developed a dangerous little habit. You ask for one change, and somewhere near the end they offer to add three more.
Would you like a configuration flag? A JSON output mode? A retry option? Maybe a dashboard while we’re here?
The agent can build it. The tests and relevant files are already in the context. So why not?
Just one more feature, one more prompt. You know what I’m talking about.
Then six months later the flag is still there. Somebody relies on the JSON shape. The retry behavior conflicts with a new timeout. The dashboard has a dependency that needs a security update. The agent that offered to build all of it has moved on with its life, mostly because it never had one.
You still own the feature.
That’s the part we need to consider. Every feature is another record in a maintenance ledger.
A Feature Is a Standing Promise
We tend to think of a feature as the code that implements it. Add the function, connect the interface, write the tests, merge the change. Feature complete.
That’s the construction phase. The feature itself is a promise that begins after the merge.
If you add a CLI flag, you’re promising that scripts can keep using it. If you expose a JSON response, you’re promising something about its shape. If you store a new piece of data, you’re promising to preserve, migrate, export, and eventually delete it correctly. If you connect another service, you’re promising to notice when its API changes.
The code might be thirty lines. The promise has no line count.
This is why tiny features get expensive. The implementation fits in one file, but the behavior crosses documentation, tests, support, deployment, security, and every future change near that boundary. Agents are very good at showing us the thirty lines. They’re much less likely to show us the next three years.
The Opening Balance Looks Great
Before coding agents, implementation cost acted as a filter. Not always a good one, but a filter.
Someone had to estimate the work, find time in a sprint, write the code, and get it reviewed. A small convenience feature might lose simply because nobody wanted to spend two days on it. Frustrating, sure, but it forced the question: is this worth building?
Now the estimate is twenty minutes. The agent has already inspected the codebase. It can update the model, add the command, generate the tests, and fix the type errors without needing another meeting. The old cost-benefit calculation collapses, because almost everything looks worth building when you only count the first implementation.
So we say yes more often.
That’s not automatically bad. Plenty of useful software never existed because construction cost too much. Cheaper implementation lets small teams solve problems that used to require a real budget. Good for us, but the maintenance math didn’t collapse along with it.
The feature still adds another path through the system. It still creates behavior that can regress. It still has users, even if the only user is you on a Sunday afternoon six months from now.
The opening balance is cheap. The account stays open.
What Accumulates
Maintenance is easy to wave away because no single piece sounds overwhelming. It’s just one more test. One more paragraph in the docs. One more migration. Then the interest starts adding up:
- Compatibility: Existing callers depend on behavior you considered an implementation detail.
- Testing: Every supported path needs coverage, fixtures, and updates when neighboring code changes.
- Documentation: The feature needs to be discoverable, accurate, and removed from the docs if it goes away.
- Dependencies: A tiny feature can introduce a library that now participates in every upgrade and security review.
- Operations: New jobs, tables, queues, or API calls need logs, failure handling, and a recovery story.
- Support: Someone has to answer why it behaved differently on another machine.
- Removal: Deleting it later means finding its users, migrating their data, and deciding how long compatibility lasts.
None of these costs are unique to generated code. We’ve always paid them. The difference is volume. Agents let us open maintenance accounts much faster than we close them.
A Ten-Minute Flag Is Still an Interface
Let me give you an example.
You have a command that prints a human-readable table. An agent offers to add
--json. That sounds great. It probably is great. The code serializes the existing records, the tests compare a sample payload, and the whole change lands before lunch.Then someone pipes that output into another script.
Now field names matter. Null behavior matters. Ordering might matter even though you never promised it. A renamed internal property breaks an external workflow. Adding a timestamp creates noisy diffs. Removing a field requires a compatibility decision.
The flag didn’t add another display format. It created an API.
Would you still build it? Probably. I like useful CLI tools, and machine-readable output is usually worth supporting. The point isn’t to reject the feature. The point is to recognize the account you’re opening. Once you see it as an interface instead of a ten-minute patch, you define the schema deliberately, document what’s stable, avoid exposing fields that should stay internal, and decide whether versioning matters before somebody’s automation answers that question for you.
Same code. Better ownership.
Backlogs Hide the Statements
One reason maintenance gets away from us is that backlogs are organized around changes, not promises.
The issue says “add export support.” It rarely says:
Maintain this export format for as long as anyone depends on it, update it whenever the underlying model changes, keep its documentation accurate, and provide a safe way to retire it later.
That would look ridiculous in an issue title. It’s still what the issue means.
Agents make backlogs disappear quickly, which feels fantastic. I’ve watched them knock out work that would have sat around for months. But a closed issue can become an open obligation. A project with fifty completed features isn’t necessarily healthier than one with twenty. It might just have thirty more things that can break.
Price the Account Before You Open It
I don’t want a meeting for every CLI flag. The whole advantage of these tools is that we can move fast.
We can still take thirty seconds to ask better questions before accepting the extra code:
- Who will depend on this? A person clicking a button creates a different promise than a script parsing output.
- What new state or interface does it introduce? Stored data and public schemas are much harder to remove than local calculations.
- What has to stay compatible? Name the stable boundary instead of letting users guess.
- How will we know it broke? Tests help, but logs, validation, and recovery may matter more.
- What ongoing work does it create? Dependencies, docs, migrations, provider changes.
- What would cause us to close the account? Decide now whether it’s experimental, permanent, or removable.
If the answers are cheap too, build it. If the feature creates a permanent public contract for a minor convenience, nope, not going in.
Cheap Construction Needs Better Restraint
I’m not interested in making software expensive again. Faster implementation is good. More people turning an idea into a working tool is good. Small teams getting leverage that used to belong to large companies is very good.
We just need to stop treating features as free.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
AI Programming Software-development Coding-agents Maintenance
-
Pi and Hermes Are Trying to Solve Different Problems
I went looking for a talk from Mario Zechner, the creator of Pi, because I wanted to understand why someone would build another coding agent when we already have a pile of them. I found: a talk called “Building pi in a World of Slop.”
Zechner described Pi as a minimal, extensible coding agent that should fit your workflows instead of forcing you into its workflow. He also made a point that should be printed on the box of every AI coding tool: code is not free. The model can produce it quickly, sure. You still own the review, the maintenance, the weird edge cases, and the next person trying to understand it six months later.
That framing explains Pi better than any feature list does.
I’ve also been reading about Hermes Agent, from Nous Research. Hermes is a useful comparison because it’s also an open, provider-flexible agent harness. But it isn’t trying to be Pi with a few extra switches turned on.
Pi and Hermes are trying to solve different problems.
Pi Gives You a Small Place to Start
Pi is a terminal coding harness with a deliberately small default: read files, write files, edit files, run shell commands. Underneath that CLI is a set of TypeScript packages for model access, the agent loop, sessions, and the terminal UI. You can use the CLI, run it through JSON/RPC, or embed the SDK in something else.
That last part is the point.
Pi deliberately leaves out things a lot of agent products treat as table stakes: MCP in the core, subagents, plan mode, permission popups, to-do lists, background shell work. This can look like a missing-feature list if you evaluate it like Claude Code or another finished product.
I don’t think that’s the right test.
Those omissions are Pi’s design. It’s saying: a harness should give you a stable loop, a tool boundary, sessions, and enough extension points to build the workflow you actually need. Then it should get out of the way.
Want MCP? Add it. Want a planning workflow? Make one. Want agents that coordinate over a message bus, work in separate git worktrees, or run in a weird internal deployment? You own the composition. Pi has extensions and packages for that, and now an explicitly experimental orchestration package, but none of it is presented as the one true way to work.
That’s a compelling idea if you’re building a specialized system. It’s also work. Both things can be true.
Hermes Starts With the System
Hermes starts from almost the opposite direction. It’s an integrated autonomous-agent platform with persistent memory, learned skills, built-in delegation, MCP support, scheduling, multiple execution environments, and surfaces that extend beyond the terminal into messaging and desktop interfaces.
Hermes is asking a larger question: what does an agent need to keep working over time, across channels, with memory of what it has already learned?
That’s not just a bigger Pi configuration.
When Hermes includes persistent memory and skill creation, it’s making those things part of the product contract. When it includes subagents and scheduling, it’s giving you an operating model for delegation and recurring work. You get more out of the box, and you inherit more of the system’s assumptions.
For a lot of people, that’s exactly right. If you want an agent to run continuously, show up in Slack or Telegram, remember prior work, and execute recurring workflows, building all of that from Pi primitives would be a very committed hobby.
Good for you, but I think most teams shouldn’t volunteer for that job unless the control model is part of what they’re building.
The Comparison That Matters
Here’s the version I keep coming back to:
Pi Hermes Default posture Minimal programmable harness Integrated autonomous-agent platform Core workflow You compose the pieces The product ships an opinionated system Multi-agent work Extensions, packages, or your own topology Built-in delegation and parallel work Memory Session primitives and JSONL history Persistent memory and skill-learning features Best fit A workflow or control plane you need to own A capable agent system you want to operate This isn’t a scorecard. Hermes isn’t “better” because it has more rows filled in, and Pi isn’t “purer” because it has fewer.
The question is where you want the complexity to live.
With Pi, much of it lives in the system you build around the harness. You have to decide how agents coordinate, what gets remembered, which tools are safe, and how approval works. In exchange, the result can fit your environment instead of being a very configurable version of someone else’s environment.
With Hermes, more of that complexity is already in the platform. You spend less time assembling basic capabilities, but you should understand its memory model, delegation model, security posture, and operational boundaries before you give it real work.
Neither choice removes responsibility. It just changes the shape of it.
Don’t Build a Harness Because It Sounds Fun
Agent harnesses are one of those things that sound like a great weekend project. You wire up a model, give it a few tools, add memory, spawn a couple subagents, and suddenly you have a tiny digital organization running in your terminal.
Then Monday happens.
The agent needs a permission model. It needs observability. It needs a way to recover from bad state. It needs sensible defaults for credentials and logs. It needs evaluation. It needs someone to own the changes when a provider API shifts or an extension becomes a security problem.
That’s why I like the Pi and Hermes comparison. It makes the tradeoff visible.
Use Hermes when you want an agent platform. It already has an opinion about the features an always-on, multi-surface agent needs.
Use Pi when the workflow itself is the product, or when the product assumptions are exactly what you need to escape. Pi’s small core is valuable because it leaves room for a different control plane.
And if all you need is a better code-review prompt or a way to query one internal system, build that inside the harness you already use. A skill, extension, or MCP server is usually a better answer than inventing an agent platform because you wanted one new capability.
This is the same point I landed on in a recent post: you think you want to build your own harness, but what you usually want is a wrapper around the one you already have.
Code is not free. Neither is a harness.
Sources & References
- “Building pi in a World of Slop” — Mario Zechner (talk) — Pi’s design philosophy, workflow fit, and the cost of generated code.
- Pi documentation — current product scope, installation, extensions, and operating modes.
- Pi usage documentation — default tool surface and deliberate core omissions.
- Pi monorepo — TypeScript package architecture and experimental orchestrator package.
- Hermes Agent documentation — persistent memory, skills, delegation, MCP, execution environments, and surfaces.
- Hermes Agent repository — open-source project and implementation reference.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Graphify Turns Your Repos Into a Map You Can Query
Navigating code dependencies inside a single repository is already hard enough. But if you’re on a microservice setup, or a split frontend and backend, tracking what depends on what across multiple repos is a special kind of misery. A backend API route changes. Which frontend components just broke? Good luck. You’re grepping three workspaces and hoping you didn’t miss one.
So when I ran across Graphify, an open-source project from Graphify Labs (YC S26), it caught my attention. It maps your code directories into queryable knowledge graphs. Not fuzzy text search. Not an expensive vector RAG lookup that burns tokens every time you ask it a question. A deterministic index of your codebase.
Let me walk through how it works, why it’s useful for AI coding agents, and the part I wanted to figure out: how to stitch multiple repos into one unified map.
What Graphify Does
Instead of guessing at relationships, Graphify parses your source and builds a real graph out of it. Three pieces make it tick:
- Deterministic AST parsing. It uses
tree-sittergrammars locally to parse roughly 40 languages, pulling out classes, functions, calls, and imports. No LLM tokens, no API rate limits. Just parsing. - Explicit vs. inferred edges. Every relationship gets a confidence tag.
EXTRACTEDmeans it’s right there in the syntax, like an import or a direct function call.INFERREDmeans it deduced the connection from context. You always know how much to trust an edge. - Leiden community clustering. It automatically segments your code into logical domain boundaries, which makes it easy to spot the “god nodes”, the files with way too many dependencies hanging off them. Those are usually the first thing you want to refactor.
Merging Multiple Repos Into One Graph
This is the part I cared about. Graphify supports it natively through the CLI, and here’s the flow straight from the docs (I haven’t run it on my own repos yet). Say you’ve got a frontend repo and a backend repo. Three steps.
Step 1: Scan each repo on its own. Run the scan inside each folder. Results land in a
graphify-out/directory.# In your frontend repo cd ~/Work/frontend graphify . # In your backend repo cd ~/Work/backend graphify .Step 2: Merge the graphs. The
merge-graphssubcommand joins the JSON outputs into one combined map of nodes and relationships.graphify merge-graphs \ ~/Work/frontend/graphify-out/graph.json \ ~/Work/backend/graphify-out/graph.json \ --out ~/Work/combined_graph.jsonStep 3: Traverse it, or hand it to your agent. Now you can trace a call path straight across the service boundary, or serve the combined graph to a coding agent over MCP.
# Trace a path across the frontend/backend boundary graphify path "login_component.ts" "auth_controller.py" --graph ~/Work/combined_graph.json # Or expose the combined graph to your coding agent over MCP python -m graphify.serve --graph ~/Work/combined_graph.jsonThat
pathcommand is the whole pitch, honestly. You point it at a frontend file and a backend file and it tells you how they’re connected. No manual grep archaeology.Why This Matters for AI Coding Agents
If you use Claude Code, Cursor, or Antigravity, you already know the problem. Feed the agent raw files and you torch the context window in about four prompts. Point it at Graphify’s output instead, the
GRAPH_REPORT.mdor thegraph.jsonover MCP, and the agent can do a few things it otherwise can’t:- Figure out exactly which files a refactor will touch before it edits anything.
- Trace dependency lineage across code boundaries deterministically, not by vibes.
- Describe your architecture based on the actual shape of the code, not a hallucinated version of it.
That last one is underrated. Half of “the AI got confused” moments happen because the AI never saw the whole picture.
Two Gotchas Before You Install
A couple of things will trip you up, so here they are up front.
The package name has a typo built in.
graphifywas already taken on PyPI, so the official package is registered asgraphifyy. Two y’s. You install it like this:pip install graphifyyWatch your Python version. The Leiden community detection library has C-extension limits, so Graphify currently runs best on Python under 3.13. Worth checking or switching to a compatible version (like 3.12) using mise.
The honest appeal here isn’t the visualization, pretty as the HTML map is. It’s that cross-repo dependency tracing has been a manual, error-prone chore for as long as I’ve worked on split codebases, and this makes it a single command.
Sources
- Graphify Labs on GitHub: setup requirements, supported parsers, and CLI options.
- Auriga IT’s Graphify introduction: explains the three-pass architecture and Leiden clustering optimization.
- Graphify on PyPI: package installation details and version compatibility.
- Aider’s Repository Map: on using tree-sitter to parse AST-based codebase maps for token-efficient coding context.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
- Deterministic AST parsing. It uses
-
Julia: High-Performance Computing Without the Two-Language Tax
I hadn’t heard of Julia until someone posted about it on Bluesky. The language was new to me, so I went reading, and this post is a summary of what stood out. Fair warning: I haven’t written any Julia myself yet.
The thing Julia is built to solve is what people call the two-language problem: you prototype in something pleasant like Python or R, hit a performance wall, then rewrite the slow parts in C or Fortran and glue them back together. Now you maintain two codebases and a wall between the people who know the domain and the people who know the fast code. Julia’s pitch, out of MIT in 2012, is to collapse that into one language: Python-like expressiveness with C-like speed, and no rewrite step.
The Three Things That Make It Different
1. Multiple dispatch
This part reorganizes how you think.
In Python or C++, methods belong to classes. You write
object.method(arg), and the language picks which method to run based on one thing: the type ofobject. That’s single dispatch.Julia flips it. Functions are the top-level concept, and methods are implementations attached to them. When you call a function, Julia picks the exact method based on the concrete types of all the arguments, not just the first one. Same function name, a different method chosen by the full type signature. This replaces the
if isinstance(...)ladders you’d write in Python and lets the type system route the call for you.2. JIT compilation, not interpretation
Julia is dynamically typed, but it does not interpret your code. The first time you call a function with a specific combination of argument types, the JIT compiler (built on LLVM) infers the types across the whole call graph and compiles specialized native machine code. Every call after that, with those same types, runs at C speed.
That’s the trick behind the whole “no rewrite” promise. You didn’t add type annotations everywhere and you didn’t drop into another language. The compiler just specialized for you.
3. Math that looks like math
Julia treats mathematical notation as first-class. Unicode variable names (
α,β,θ), literal coefficients like2x + 3y, and native linear algebra operators. If you’re translating a paper into code, the code ends up looking like the paper:α, β = 3, 4 α^2 + β^2 # => 25 2α + 3β # => 18 (2α means 2 * α, no asterisk needed)The strange thing about programming is how often the math matters. First-class support for mathematical notation sounds minor, but for the tasks Julia is built for, it turns out to be surprisingly useful.
Another Quick Example
One Julia feature that stood out to me is broadcasting. In NumPy, chaining array operations like
sin(x) + cos(y)quietly allocates temporary arrays for each step. In Julia, one dot fuses the whole expression into a single loop with zero intermediate allocations:x = [1.0, 2.0, 3.0, 4.0] y = [10.0, 20.0, 30.0, 40.0] result = [@](https://micro.blog/). sin(x) + sqrt(y) * 2.5 # fused, no temporaries
Tooling and Two Rules
Setup is one installer.
juliaupmanages versions (curl -fsSL https://install.julialang.org | sh, thenjuliaup add release), and the built-in package manager lives one keystroke away: hit]in the REPL to enter package mode, thenadd DataFrames DifferentialEquations Makie CUDA. The ecosystem worth knowing: DifferentialEquations.jl for solvers, JuMP.jl for optimization modeling, Makie.jl for GPU-accelerated plotting, and CUDA.jl for writing GPU kernels without touching CUDA C++.Two rules keep your code at full speed. First, avoid untyped global variables. A non-const global forces a dynamic type lookup on every access, and it will silently gut your performance. Use
const, or pass values in as parameters. Second, keep your functions type-stable, meaning the return type depends on the types of the arguments, not their runtime values. When something feels slow, run@code_warntype my_function(1.0, 2.0)and look for the red. That’s the compiler telling you it couldn’t infer a type.
So When Should You Actually Use It?
Reach for Julia when you’re building numerical simulations, differential equation solvers, actuarial or financial risk models, or custom ML architectures, basically anywhere NumPy or PyTorch hits a wall and you can feel the rewrite coming.
Don’t reach for it to build a web scraper or a CRUD service. Go or Node boots faster and the JIT warmup isn’t worth it there.
The pitch is simple: the language that lets you prototype is the same language that ships. No second codebase, no translation layer, no wall between your domain experts and your fast path. If you’ve ever paid the two-language tax, that’s the whole reason to give Julia a look.
Sources
- Bezanson, Edelman, Karpinski & Shah, Julia: A Fresh Approach to Numerical Computing, SIAM Review 59(1), 2017.
- Julia Documentation, Performance Tips.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Adding Types to JSON with Dhall
A few months ago I wrote a post asking whether there’s something better than JSON. Two configuration languages that sit above JSON kept coming up: CUE and Dhall. Both give you the things JSON lacks when you author config by hand, and both compile down to plain JSON, YAML, or whatever your services actually read. I spent more time with CUE back then and never gave Dhall a real look. This post is me going back for that second look, because the one feature I kept wanting was a type system over my config.
JSON is the universal language of API payloads and config files, and I don’t want that to change. But as a format for authoring configuration by hand, it’s rough:
- No comments.
- No variables or functions, so you copy-paste the same block ten times.
- No type system, so
"8080"and8080look equally valid. - No imports, which is how you end up with a 2,000-line monolith nobody wants to touch.
The usual escape hatch is a templating engine like Jinja or Helm, or a real programming language like Python or TypeScript that spits out JSON. That works, but you’ve traded one problem for a scarier one: your config generator is now Turing-complete. It can crash, hang in an infinite loop, or reach out and read some local environment variable, and it’ll do it at 2 AM when the pipeline runs.
This is where Dhall comes in.
What is Dhall?
The short version: Dhall is JSON plus types, plus functions, plus imports. It’s a programmable, strongly-typed configuration language.
The part I actually care about is what it doesn’t have. Dhall is not Turing-complete. No arbitrary recursion, no side effects. Every Dhall program is guaranteed to terminate. You get the abstraction power of a functional language like Haskell or Elm, with the guarantee that it will never hang your build. That’s a different trade than “just write a Python script.”
The problem, in JSON
Here’s a normal
config.jsonfor a microservice:{ "serviceName": "payment-api", "port": 8080, "environment": "production", "database": { "host": "db.internal.net", "maxConnections": 50 } }Three ways can be a problem in production: someone writes
"port": "8080"and the service won’t boot, someone typos"prodution"and it silently runs in debug mode, or someone forgetsmaxConnectionsentirely and you get a null blowup at runtime. Nothing catches any of it until it’s live.The same thing, typed
In Dhall you define the shape up front. Enums, record types, default values:
-- schema.dhall let Environment = < Local | Staging | Production > let Database = { Type = { host : Text, maxConnections : Natural } , default = { maxConnections = 20 } } let Config = { Type = { serviceName : Text , port : Natural , environment : Environment , database : Database.Type } , default = { port = 8080, environment = Environment.Local } } in { Environment, Database, Config }Now you author against that schema, and you get defaults and composition for free:
-- config.dhall let Schema = ./schema.dhall let myConfig : Schema.Config.Type = Schema.Config.default // { serviceName = "payment-api" , environment = Schema.Environment.Production , database = Schema.Database.default // { host = "db.internal.net", maxConnections = 50 } } in myConfigMisspell
Production, or pass"8080"as a string, and Dhall throws a type error before a single line of JSON is generated. Hopefully the benfit is now clear; adding a type safety layer to your config files.Compiling down to JSON
You don’t ship Dhall to your services. You ship the JSON they already understand:
brew install dhall-json dhall-to-json --file config.dhallOut comes clean, boring, standard JSON. Your services never know Dhall was involved. The part that I like is the safety lives at authoring time, and the runtime artifact stays dumb.
Two features worth knowing about
Hermetic imports with hash pinning. Dhall can import from a URL, so shared utilities live in one place instead of being copy-pasted across five repos. To keep someone from swapping the file out from under you, you pin the import to a SHA-256 hash of its normalized form:
let Prelude = https://prelude.dhall-lang.org/v22.0.0/package.dhall sha256:10db4c919c25e4d262db3ed0d1d6120da3e3906673f00e3012c1d14e1963976aIf the remote content changes, the hash won’t match and the build fails. The hash above is just an example, and each Prelude version has its own, so don’t copy it by hand.
dhall freeze --inplace config.dhallcomputes the correct hashes for whatever you’ve imported and pins them automatically.Exhaustive matching with
merge. When you map a union type to output, Dhall makes you handle every variant:let getLogPrefix = \(env : Environment) -> merge { Local = "[DEV] ", Staging = "[STAGE] ", Production = "[PROD] " } envAdd a
QAvariant later, and everymergeblock that touchedEnvironmentfails to compile until you deal with it. No forgottenswitchcase slipping into production. The compiler keeps a running list of everything you now owe it.Is it worth it?
Raw JSON Dhall Type safety None, fails at runtime Static, at compile time Comments & logic No Yes Termination N/A Guaranteed Dependency pinning No SHA-256 Output Consumed directly Compiles to JSON/YAML/TOML For a two-key config file, it doesn’t make sense, but once you’re staring down Kubernetes manifests, a pile of near-identical microservice configs, or anything where a typo takes down a service, the calculus changes. You keep clean static JSON as the thing your services actually read, and you move all the ways-to-get-it-wrong to a place where a compiler catches them first.
Sources
- Dhall Language Tutorial & Cheatsheet: records, union types, default overrides,
dhall-to-json, anddhall freeze. - Dhall language standard on GitHub: the non-Turing-complete design and the semantic integrity hash spec.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Programming Development links code software engineering language
-
Build Your Own Skills Repo
If you’ve been working with AI coding agents for a while, you’ve probably started collecting workflows. You might not call them that yet, but they’re there. Another name for workflows is Skills.
Some are tiny: run these checks before shipping. Some are project-specific: when you touch this library, preserve this API contract. Some are operational: never print secrets, always summarize the logs. Right now they probably live in chat history, a README note, a shell alias, or your own memory. That works until it doesn’t.
A skills repo is a better shape for this. It’s one place to collect, version, review, and share the workflows that make agents useful in your projects. I built my own yesterday, so let me walk through how I think about it.
A skill is judgment, not a command list
The first mistake is treating a skill like a list of commands. Commands matter, but they’re the easy part. The real value is judgment: when to run the command, what to inspect first, what not to do, how to validate the result, and what risks are specific to this tool.
A good skill makes an agent more careful. It narrows the space of bad decisions. So before you write a single instruction, figure out who the skill is for.
Separate adoption from maintenance
Most serious projects have two audiences: people using the project, and people maintaining it. Those should usually be two different skills.
An adoption skill helps an outside developer get value from your project. Install the package, configure it correctly, use the right import path, migrate existing code, avoid the common mistakes, run the right validation.
A maintainer skill helps contributors work inside the source repo. Understand the layout, run the local quality gate, preserve compatibility promises, follow the release conventions.
Here’s why you need two. If you only write maintainer skills, your repo becomes a private automation folder. If you write adoption skills too, it becomes onboarding infrastructure.
Keep each skill focused
A skill should have a job. Not “everything about this project.” Not a duplicate README. Good skill names are verbs:
integrate,audit,upgrade,migrate,debug,ship. That keeps the trigger obvious, so when someone asks the agent to do that kind of work, the skill has a clear reason to load.If a project needs multiple workflows, split them. A library might have
integrateanddevelop. A deployment system might havedeployandrollback. Don’t cram them into one file.Put the safety rules near the top
The most important part of many skills is the “do not” section:
- Do not print secret values.
- Do not delete or archive anything without confirmation.
- Do not add failing CI enforcement unless asked.
- Do not do broad rewrites before previewing a diff.
- Do not commit local registry URLs.
Agents are good at momentum. Safety rules are how you make that momentum usable. The more destructive the workflow, the more explicit the guardrails should be.
Include validation, not just execution
Every skill should answer one question: how do we know this worked? That might be
pnpm test && pnpm build, orcargo test && cargo clippy -- -D warnings, orgo test ./.... For non-code workflows it might be “export the review list” or “verify the generated config is ignored by git.”This matters because agents can complete every step without completing the work. Validation closes the loop.
Write for the agent inside the repo
Skills should assume the agent is operating in a real project with real files and existing conventions. So the useful instructions look like:
- Inspect
package.jsonbefore choosing a package manager. - Read the existing test scripts before adding new ones.
- Prefer the local task-runner commands when they exist.
- Check the framework boundary before picking an import path.
That’s the context generic model knowledge won’t reliably infer. And it’s why you shouldn’t just copy the README into the skill. A README is for a human browsing the project. A skill is for an agent doing work. They overlap, but they aren’t the same artifact. Keep the skill short enough that loading it is cheap.
Use a marketplace repo as the index
Your skills repo doesn’t need to be the canonical home for every skill. Some projects should own their own plugin metadata, especially if they already have a CLI, release process, and docs. Your marketplace can just point at them remotely. Other skills can live directly in the marketplace repo. One structure that works:
skills/ .claude-plugin/ marketplace.json plugins/ esm/ .claude-plugin/plugin.json skills/develop/SKILL.md upkeep-rs/ .claude-plugin/plugin.json skills/audit/SKILL.mdThe marketplace becomes the thing people add once. Individual plugins stay free to live locally or point at their canonical upstream.
Scan third-party skills before you import them
The moment your marketplace points at someone else’s plugin, you’ve inherited their security posture. And skills are a soft target. The dangerous payload usually isn’t code, it’s prose: an attacker buries instructions inside a
SKILL.md, gated behind an innocent-sounding trigger, that tell the agent to read your.envand send it somewhere. A normal code scanner walks right past that. There’s no malware signature to match. It’s just English.This isn’t hypothetical. Snyk’s ToxicSkills research found prompt injection in 36% of the skills they tested, across more than a thousand malicious payloads. If you’re pulling skills from a public index, some fraction of them are trying to do something you didn’t ask for.
So run a scanner before you add anything you didn’t write. A few worth knowing:
- Snyk agent-scan inventories your installed agents, MCP servers, and skills, then checks them for prompt injection and data-handling problems.
- NVIDIA SkillSpector scans repos, URLs, or single files against a big catalog of patterns: injection, exfiltration, privilege escalation, tool poisoning.
- claude-skill-antivirus is purpose-built for Claude Code skills and runs several detection engines at once.
One caveat worth internalizing: scanning an MCP config can execute it, because starting a stdio server means running the command in the file. Do that in a sandbox, a container or a throwaway VM, not on your main machine. The tool you run to check for danger shouldn’t be the thing that sets it off.
This cuts both ways. If you publish a plugin others will install, a clear “do not” block and an honest description of what the skill touches is part of being a good citizen of the marketplace.
Start with your serious projects
You don’t need a skill for everything. Start where better agent behavior would matter: public libraries people might adopt, CLIs with safety-sensitive workflows, tools with tricky setup, projects with recurring maintenance, systems where mistakes are expensive.
For each one, ask yourself:
- Who is this for: user, maintainer, operator, contributor?
- What’s the concrete task?
- What should the agent inspect first?
- What commands are preferred, and which are dangerous?
- What should never happen silently?
- What validation proves the work succeeded?
Answer those and you have enough to write a useful first skill.
Why it’s worth doing
A skills repo turns scattered project knowledge into reusable operational guidance. But it also forces a better product question. If this project is meant to help people, what would it look like for an AI agent to help them use it well?
That’s a higher bar than “can the agent run the command?” The point isn’t to automate everything. It’s to package the judgment around your tools so the next agent starts from a better place.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Web Development Trending Bluesky Feed
Trending web dev topics on bsky.
View the algorithm definition: www.graze.social/feeds/170…
Want to improve it? Post at @hipstsersmoothie.com
-
A Model Context Protocol (MCP) server for the Internet Archive’s Open Library API that enables AI assistants to search for book and author information. - 8enSmith/mcp-open-library
-
How Do You Actually Review Code an Agent Wrote?
We’ve all had the magic moment by now. You fire up an agent in Claude Code, hand it a bunch of tasks, then let it do its thing. It feels like the future.
Right up until you have to hit
git commit.That’s when the magic curdles into a very specific kind of anxiety: how do I actually review this? I didn’t write it. I barely watched it happen. And now I’m supposed to vouch for it.
The AI Reviewer Trap
The obvious move is to fight fire with fire. An AI wrote it, so pipe the diff into an AI PR review tool and let the machines sort it out, right?
I wrote about this back in May: AI code reviewers won’t save you. Having one LLM grade another LLM’s homework can potentially lead to disaster. They share the same blind spots. They’re trained on the same patterns, so they tend to nod along at the same plausible-looking mistakes, and they’re notoriously bad at catching the subtle, systemic logic flaws that span an entire architecture. The bug that matters is rarely on one line. It’s the interaction between four files, and that’s exactly the kind of thing a second LLM waves through.
So you swing the other way and read it yourself, line by line. Every variable assignment, every branch. And that’s exhausting. Worse, it defeats the entire point of using an agent. If I have to mentally re-type every line the agent produced, I might as well have typed it for real.
So we’re stuck between a reviewer we can’t trust and a review process that erases the speedup. Neither one is the answer.
The Bottleneck Just Moved
Agentic development didn’t make software engineering easier. It moved the hard part.
Writing the implementation used to be the bottleneck. That’s the part the agent is genuinely good at now. What it can’t do for you is tell you the behavior is correct. Verifying the behavior is the whole job now, and that’s a different skill than writing code.
This is why migrating my test suite to Vitest earlier this year has paid off more than I expected at the time. When you’re driving autonomous agents, automated testing stops being a chore you do to keep a coverage badge green. It becomes the only safety net you actually have.
Trust the Spec, Not the Code
In an agentic workflow, my job as the human isn’t to write the function anymore. It’s to write the tests, or at least to rigorously verify the ones the agent proposes.
Think about what that buys you. If I have a comprehensive, fast test suite, I don’t ALWAYS need to read all 400 lines Claude Code just generated. I need to watch the runner light up green. If the tests pass, and the tests are good, the implementation details matter a lot less than they used to. The tests are the contract. The code is just one way to satisfy it.
That second condition is doing a lot of work, though, so I want to be honest about it. “If the tests are good” is the entire game. A passing suite that doesn’t cover the edge cases is worse than no suite, because it hands you false confidence at the exact moment you’ve stopped reading the code. So the scrutiny doesn’t disappear. It relocates. Instead of reviewing the implementation, you review the spec. Are the right behaviors tested? Are the failure modes tested? Did the agent quietly write a test that asserts its own bug?
That’s a much smaller surface to review than 400 lines of implementation, and it’s a far more durable thing to spend your attention on. The tests outlive any single refactor.
TDD Didn’t Die, It Got Promoted
For years people treated test-driven development as a discipline you adopted if you were virtuous and skipped if you were busy. Agentic coding flipped that. It made testing the load-bearing skill, because tests are now the interface between what you want and what the machine builds.
So the answer to “how do I review code an agent wrote” turns out to be: mostly, you don’t. You review what it’s supposed to do, you encode that in tests you trust, and you let the green checkmark tell you whether the agent got there.
I’m curious whether this matches your experience. I’ve found I write more tests now than I did before the agents showed up, not fewer. The implementation got cheap. Being sure it works did not.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
The Agentic Dev Space Is Moving Fast, and I'm Having a Blast
Step back and look at the last six months of software development. It’s almost hard to believe how much has shifted. Back in January we were doing “vibe coding” with basic autocomplete. By June we’re handing fully autonomous agents a feature, walking off to grab a coffee, and coming back to find it architected, written, and tested.
It’s easy to look at the volume of new tools dropping every single day and feel a little fatigued. But honestly? I’m just excited. This is one of the most fun eras of programming I’ve worked through in decades.
Here’s a quick look at the tools that have already changed my workflow, and the ones I want to dig into next.
The Tools I’ve Adopted
If you read my post earlier this week, you know Supacode has become my daily driver. It bridged the gap between raw terminal access and a UI that understands how context-heavy agentic work is. That’s a harder problem than it sounds, and it’s really feeling good so far.
I also have to shout out the tools that paved the way earlier this year. Claude Code and Antigravity proved out the model of a CLI-native agent that could navigate your file system and do the work. Running Claude Code daily—and turning to Antigravity occasionally when I have access—has completely retrained my habits. I stopped typing every line of code and started acting more like a technical lead reviewing pull requests from a tireless junior developer. That shift in posture is the real unlock, and these were the tools that taught me it.
The Tools on My Radar
Because the space moves so fast, my “stuff I want to learn” backlog keeps growing faster than I can clear it. Here are the three I’m most eager to explore when I carve out the time:
- Hermes. I keep hearing great things about how it handles multi-step reasoning and huge context windows. I haven’t had the right project to throw at it yet, but it’s at the top of the list.
- Pi Agents. The concept here is highly specialized, networked agents working in tandem. Instead of one monolithic agent doing everything, you’d have a “frontend agent” talking to a “database agent.” That feels like a different way to structure the work, and I want to see if it holds up in practice.
- Evaluating Memory Systems. I’ve been diving into how agents remember things over time. I wrote a Mem0 MCP server that runs locally, and I’ve added hooks for Mem0 right into my CLI agents. It’s been a great way to slowly improve my own local agentic memory system. I’m really eager to see how other memory systems (like LangChain’s implementations) work under the hood, and I want to spend more time comparing and contrasting them.
You Don’t Have to Learn It All Today
The best part about this explosion of tooling is that you don’t need to know all of it. You don’t.
Find a tool that solves an immediate problem in your workflow, master that one, and let the rest sit on your radar until you need them. The ecosystem will keep evolving whether or not you’re watching every release. The tools will keep getting better. Chasing every launch is a great way to learn nothing well.
So pick one. Get good at it. Let the backlog wait.
What’s the one agentic tool sitting on your radar that you just haven’t had time to dig into yet?
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Why Supacode Became My Daily Driver for Claude Code
The number of ways to get an AI to write your code right now is … great? Claude Code, Codex, Opencode, and all the others. I think that’s a good thing. The easier we make it to hand real work to a capable coding agent, the better off we all are in the long run. I want to try most of them because that’s half the fun, figuring things out.
My cool tool for the week is Supacode.
What Supacode Is
Supacode is a native macOS app, written in Swift and built on libghostty, with no Electron and no web wrapper anywhere in sight. It’s fast the way native software is fast, the kind of speed you feel in every keystroke.
The website calls it a “command center for coding agents,” and well ok I get it now. Supacode is the nicest way I’ve found to run Claude Code. It’s open source too, so you can poke at the internals on GitHub if you’re curious.
What It Does for Me
This is the part that I like.
Supacode keeps all my projects organized down the left side, so jumping between them is one click instead of a mental map of which terminal tab is which. When I start a Claude Code session, its chat window moves up into the active area, and I can see every live session I have going at a glance.
The big one: it hooks into Claude Code, so Claude tells Supacode when it’s finished a task or when it needs something from me. Instead of babysitting a terminal waiting for the next prompt, I just get told when Claude is waiting on me. I tried for a long time to get something like that working in a plain terminal and never really nailed it. Here it just works.
And when I need more than one CLI open on the same project, I can group those tabs together instead of squinting at a wall of identical-looking shells.
None of that is glamorous. It’s just the difference between running Claude Code and pleasantly running Claude Code.
I Still Love Ghostty
I’m not leaving Ghostty behind, to be clear. I love Ghostty. Mitchell Hashimoto and everyone who’s worked on it has built something special. When I need a clean shell to compile a binary or grep through some logs, Ghostty is where I want to live.
Supacode is built on libghostty, the same engine that powers Ghostty. So the speed and feel I love about Ghostty is sitting right underneath Supacode too. They’re cousins. I get Ghostty for raw terminal work and Supacode for running Claude Code, and both of them are quietly standing on the same excellent foundation.
So Here We Are
We’re spoiled for choice with AI coding tools right now, and I love that. Most of what I try is interesting, and then I move on. Every so often something just slots into how I already work and earns a permanent spot. For me, right now, that’s Supacode. If you’re running Claude Code all day on a Mac, it’s well worth a look.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Native terminal coding agents command center. Run 50+ coding agents in parallel.
-
Day 19: Leap Seconds or When a Minute Has 61
It’s week four and this week we’re talking about the inconsistencies, the problems, the strange things about time and how we measure it. If week three was mostly about software and computer systems, week four is more about the properties or the features or the details of recording time and our time systems.
The first crack in our concept of time is the quite irregular leap second.
UTC works?
How can a minute ever have 61 seconds? If you start at zero, you got to end at 59. You start at zero and you end at 60, now you have 61 seconds. That doesn’t make sense.
If you have ever seen
:60Zin a log file, that is the leap second. That minute had sixty-one seconds in it.It exists because we are trying to do the impossible: keep two completely different definitions of the “second” in alignment, forever. There are, it turns out, two definitions.
But wait, it gets worse.
There are three time scales running in the background of our civilization right now:
- TAI, or International Atomic Time. The average of about 400 atomic clocks at standards labs around the world, all ticking off the cesium hyperfine transition we covered on Day 8. TAI is uniform. Every second is the same length as every other second. TAI does not care about the Earth.
- UT1, or Universal Time 1. Defined by the Earth’s actual rotation, measured by tracking distant quasars with radio telescopes. UT1 is wobbly. The Earth speeds up and slows down by milliseconds per day, mostly because of tidal friction (slowing it down) and core-mantle coupling (anyone’s guess on any given decade).
- UTC, or Coordinated Universal Time. The civil time on your phone. UTC is TAI minus an integer number of leap seconds, kept within 0.9 seconds of UT1.
It’s time to stop pretending like the current version of UTC isn’t a compromise. It certainly is not the best we could come up with. It’s just what everyone could agree on.
The first of its problems is the dang leap second.
So how did we get to this mess?
The IERS, the International Earth Rotation and Reference Systems Service in Paris, watches the gap between UT1 and UTC. They announce the leap second six months in advance in an actual notice called Bulletin C. They also have their own weekly and monthly newsletters called Bulletin A and Bulletin B. I don’t know what is going on at IERS and I’m sorry to anyone working there, but this leap second thing is kinda crazy.
When the leap happens, the clock reads:
23:59:58 23:59:59 23:59:60 ← this is the leap second 00:00:00That
:60is the part that breaks software. Most date/time libraries do not believe:60is a real value. POSIX, the standard governing Unix systems, explicitly defines Unix time to pretend leap seconds don’t exist.Since the system was introduced in 1972, 27 leap seconds have been inserted, although none since 2016. Also, there has never been a negative leap second. We’ve only ever needed to slow UTC down to match the Earth.
But, that may be about to change. More on that tomorrow.
Because I can’t help myself.
Here’s more stuff about… Computers.
There are three ways a computer can handle the leap second arriving.
- Step. At midnight, the clock just jumps back one second. From the OS’s perspective, time briefly moves backward. Anything assuming time is monotonic, meaning it only goes forward, sees its assumption violated and may explode.
- Stall. Hold
23:59:59for two seconds. Time doesn’t go backward, but two events get the same timestamp. Anything depending on timestamp uniqueness gets confused. - Smear. Spread the extra second over a long window (Google originally used 20 hours centered on the leap, then standardized at 24 hours) by ticking slightly slow for the whole period. No
:60ever appears. No backward step. Just a clock that runs 1.0000116× slow for a day.
Google introduced smearing in 2008. By the late 2010s most cloud providers (Amazon, Microsoft, Facebook) had adopted some flavor. It is now the de facto practice.
Before smearing, leap seconds were MORE dangerous.
A second is a second not two
A leap second is like when Pluto was a Planet. It has to be a singular definition. A known amount. A standard. Software written at some of the most capable engineering organizations still took down important infrastructure, internet infrastructure. Clearly it should not be this hard to define a unit of time.
But anyways, onto the next post.
Tomorrow will cover the historic 2022 vote to abolish the leap second, the fight that produced it, and what we all agreed to.
Sources
- International Earth Rotation and Reference Systems Service (IERS) — The body responsible for monitoring Earth’s rotation and issuing Bulletin C to announce leap seconds.
- A global timekeeping problem postponed by global warming — Nature (2024). The Duncan Agnew paper detailing how melting polar ice has counteracted the Earth’s acceleration, delaying the unprecedented “negative leap second” until roughly 2029.
- POSIX.1-2017 Base Definitions: Seconds Since the Epoch — The Open Group. The formal specification demonstrating that Unix time legally ignores leap seconds.
- Time, Technology and Leaping Seconds — Google’s original 2011 blog post introducing the concept of the “leap smear.”
- The Leap Second Glitch Explained — Wired. Detailed breakdown of the 2012 Linux
hrtimerbug that took down Reddit and Qantas. - How and why the leap second affected Cloudflare DNS — Cloudflare’s excellent, candid post-mortem of their 2017 New Year’s RRDNS outage.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Programming Infrastructure Time 30daysoftime Leap-second UTC
-
Day 18: DST and The Related Software Bugs
This is one of two or three DST posts in the 30 Days of Time series. Today’s angle: the software bugs.
Twice a year, in most of the developed world, the clocks jump forward in spring and back in fall. The hour that doesn’t exist in spring materializes and then disappears in the fall. This is the source of more shipped bugs than any other single phenomenon in software.
The Two Impossible Hours
The mechanics, if you’ve never thought hard about them.
Spring forward. On the transition Sunday in March (in the US), the clock reads
01:59:59and then immediately reads03:00:00. The hour from 2:00 to 2:59 AM does not exist. 2:30 AM on that day is not a time. If you tell a computer to do something at 2:30 AM on that day, you have asked it to do something at a time that doesn’t exist.What it does is up to the library:
- It might silently skip.
- It might silently run at 3:30 instead.
- It might throw an exception.
- It might run at the wrong time and silently throw your reports off.
The classic landmine is a daily task scheduled in local time, for example a job set to run at
1:30 AM. If you’re using the standard Linuxcrondaemon, it has battle-tested, built-in logic to detect DST transitions and prevent duplicates.The problems are usually at the application layer. If you are using an application-level scheduler or Cron library that hasn’t been configured properly and blindly trusts the system clock, you can get into a situation where that 1:30 a.m. doesn’t exist or runs twice.
A Short Tour of Named Disasters
March 2007, United States. Congress passed the Energy Policy Act of 2005, which moved DST to begin three weeks earlier and end one week later. The change took effect in March 2007. Every system in the country running on a tz database older than mid-2006 spent three weeks in March, and one week in November, off by an hour. Banks ran payroll at the wrong time. BlackBerry calendars showed every meeting an hour off. Federal agencies had to issue advisories. The DOE later estimated the extension saved about 0.5% of electricity per day of extended DST, or roughly 1.3 TWh annually. The remediation cost across every affected piece of software in the country dwarfed that figure. (More on that later.)
New Year 2011, iOS. Non-recurring alarms set for January 1 or 2, 2011 did not fire, in any time zone. People slept through work. Apple’s official advice was to set one-time alarms as recurring until January 3. This was on top of an iOS DST bug from a few months earlier, when the fall 2010 transition shifted alarms by an hour in countries that had already changed clocks. Two months after the New Year’s bug, iOS again mis-handled the US spring DST transition. Apple released an apologetic fix and quietly rewrote the alarm subsystem.
Brazil, April 2019. Brazil canceled DST after decades of observing it, via Decree 9,764. This is fine for clocks going forward, but the cancellation was announced only a few months in advance, and the IANA tz database had to ship updates fast. Every Brazilian server running on a stale cache spent the next year an hour off, in particular for any future-scheduled event saved as “local time.”
Palestine. For about a decade running, Google Calendar shipped wrong DST data for Palestine, because the Palestinian Authority changes DST rules with short notice and the IANA volunteers don’t always learn in time. Meetings between Israeli and Palestinian colleagues would silently shift by an hour twice a year.
The Shape of the Failure
The DST bugs usually go like this:
- A piece of software was written with the assumption that local time is well-defined and monotonic.
- Local time is neither.
- The author never hit edge cases. It only happens twice a year, in certain regions, under certain settings.
- The bug ships. It runs fine for six months. Then it doesn’t.
The mitigations are well-known and this is why we do what we do.
- Store UTC. Always. The IANA zone ID goes in a separate column. Never, ever store a naked local timestamp.
- Recompute the local display every time. Treat local-time as a view, not data.
- Never schedule anything between 2 and 3 AM local. That hour does not exist in your country half the time.
- Use libraries that surface the ambiguity. The older Python
pytzlibrary would throw when you constructed an impossible local time. The modernzoneinfohandles it silently via afoldattribute, meaning you have to manually check for ambiguity. JavaScript’sDateproduces inconsistent results across engines. TheTemporalAPI, which reached Stage 4 in March 2026 and ships in Chrome 144, Firefox 139, and Node 26, lets you explicitly reject ambiguous times. Use it the soonest you can. - Keep tzdata current. This is a system-package problem and most teams forget about it until something breaks.
The tricky part of software has always been that we think that the wall clock or the wall time, the number you see on a daily basis is the same as the actual physical passage of time when in reality they are not. Daylight savings time is a really great example of the absurdity of our timekeeping.
Week 3 Recap
If this is the first time you are reading this series I figured a recap is order. Week 3 has been about the infrastructure of practical timekeeping, the layer where computers, calendars, and humans actually have to agree on what time it is.
- Day 13: Unix Time, 1,780,620,532 — The 10-digit integer counting up from 1970 that runs every computer on Earth, and the weird properties hiding behind the name.
- Day 14: The Bug That Didn’t End the World, and the One That Still Might — Y2K was a save, not a hoax, and Y2038 is the one nobody is preparing for.
- Day 15: The Man Who Synchronized the World — David Mills, NTP, and the forty-year project that keeps every networked clock on Earth within a few milliseconds of UTC.
- Day 16: How the World Agreed on a Date Format (Except the US) — The century-long campaign that produced
2026-06-08T14:30:00Z, and why a bare05/06/26is still an act of faith. - Day 17: Time Zones Are a Nightmare — 38 named offsets in active use, half-hour zones, and why “what time is it there?” is the wrong question.
- Day 18 (today): DST, and the bugs that ride along with it twice a year.
The picture I want to leave you with is that every problem we’ve covered in Week 3 is a downstream consequence of a deeper one. The system isn’t fragile because of bad programmers. It’s fragile because the underlying thing, “what time is it, here, right now,” was never a single answer, and we’ve been pretending it was.
What’s Coming
Week 4 is about the cracks. What if a minute had 61 seconds? What if October had only 21 days? What if every meeting on every calendar landed on the same weekday, forever? Each of those has actually happened, or is being voted on, or was almost adopted. Week 4 covers leap seconds and their abolition, the DST fight nobody can win, and the calendars we use, almost used, and may yet use.
Sources
- Energy Policy Act of 2005 (Wikipedia). Details the US DST schedule change that took effect in 2007.
- Impact of Extended Daylight Saving Time on National Energy Consumption (US DOE). The 0.5%-per-day savings estimate from the post-2007 study.
- Apple confirms New Year’s alarm bug (AppleInsider). Coverage of the iOS 2011 non-recurring alarm bug and Apple’s workaround.
- Daylight saving time in Brazil (Wikipedia). History of DST in Brazil, including the 2019 abolition via Decree 9,764.
- Zune 30GB leap year bug (Wikipedia). The firmware loop that bricked Zunes on New Year’s Eve 2008.
- 2012 Reddit leap second outage (Wired). Write-up on the Linux
hrtimerbug that took down Reddit, LinkedIn, and Qantas. - TC39 Advances Temporal to Stage 4 (Socket). Current status of the JavaScript
TemporalAPI.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Day 17 — Time Zones Are a Nightmare
Yesterday I wrote about how
2026-05-24T14:30:00Zwon the format wars. ThatZat the end, it turns out, is quite important. It says Zulu, it says UTC, it says: I am refusing to participate in the nightmare that is Timezones.However, today we participate.
The lie you were told in school
There are 24 time zones, one for every hour, neat 15° slices of the globe.
There are not. There are at least 38 named offsets in active use right now, and the list changes a few times a year.
Some of them are not on the hour:
- India is at UTC+5:30. The whole country, one zone, half-hour offset.
- Nepal is at UTC+5:45. Forty-five minutes. Because Nepal decided in 1986 that it wanted its civil time anchored to the meridian passing through Gauri Shankar, not Delhi.
- Newfoundland is at UTC−3:30.
- The Chatham Islands are at UTC+12:45.
The range isn’t 24 hours either. It runs from UTC−12 to UTC+14, a 26-hour spread, because Kiribati got tired of being split by the international date line in 1995 and just moved the line. One day the Line Islands woke up and it was tomorrow.
Then there’s China, which geographically spans five time zones and politically uses one (UTC+8), so in the far west of Xinjiang the sun rises at what the clock insists is 10 AM. North Korea changed its offset twice in the 2010s, from UTC+9 to UTC+8:30 in 2015 to mark liberation from Japan, then back to UTC+9 in 2018 to align with Seoul during a diplomatic thaw.
Time zones are not geography. Time zones are politics with a clock face glued to the front.
How we got here
Before about 1850, every town in the world ran on its own clock. Noon was when the sun was overhead here, which meant noon in Boston was several minutes off from noon in New York, which was off from Philadelphia, which was off from everywhere.
Nobody cared, because nobody was traveling fast enough for it to matter.
Then the railroads showed up.
When your train leaves at “noon” and arrives at “3 PM” and every station defines noon differently, you can’t print a schedule. Britain rolled out Railway Time (GMT everywhere) in the 1840s. The American railroads, bless them, didn’t wait for permission. On November 18, 1883, they unilaterally divided the United States into four zones. Newspapers called it “the day of two noons” because clocks across the country jumped, sometimes forward, sometimes back, to land on the new shared time.
The following year, in October 1884, twenty-five countries met in Washington for the International Meridian Conference and made it official. Greenwich is 0°, the universal day starts at midnight in Greenwich, every other place is some offset from that.
France abstained. France wanted Paris. France held out until 1911.
The “database” holding the world together
When your phone shows you the right time after you land in Tokyo, when your calendar correctly reschedules a meeting because Mexico canceled DST a few years ago, when your server logs all line up across a deploy in three continents, that all happens because of a single, voluntarily maintained text file.
It’s called the IANA Time Zone Database, also known as the Olson Database, after Arthur David Olson, an NIH employee who started maintaining it in 1986 as a side project. Today it lives under IANA stewardship and is primarily maintained by Paul Eggert, a UCLA computer scientist who has been doing this, mostly alone, for decades.
Every Unix system, every Linux distro, every Mac, every iPhone, every Android phone, Java, Python, Go, Rust, Postgres, browsers, every piece of software that knows what time it is, gets its time zone rules from this database. The release cadence is multiple updates per year, almost always triggered by some country’s parliament deciding to change DST rules with three months' notice.
The format is something like
America/New_York,Europe/Berlin,Asia/Kolkata,Pacific/Kiritimati. Area, slash, location. NotEST, notGMT+5, because those are offsets and offsets aren’t enough. The rules are what you need, because the rules change with politics.The whole arrangement is held together by a small group of volunteers, a mailing list, and Paul Eggert’s continued willingness to keep doing this. If he ever stops, somebody else will have to start.
Why this is one of the hardest problems in working programmer software
A few categories of pain, none of them solvable, all of them shipped to production daily:
1. Ambiguous local times. When the clocks fall back in November, the hour from 1:00 to 2:00 AM happens twice. If a user schedules a meeting at “01:30 local time” on the wrong day, which 01:30 do they mean? There is no correct answer. Your software picks one and someone shows up an hour off.
2. Nonexistent local times. When the clocks spring forward in March, 2:30 AM doesn’t exist. If somebody’s medication-reminder app is set for 02:30, what does it do that morning? Skip? Run at 03:30? Run at 01:30 the previous hour? There is no correct answer.
3. Future timestamps are mutable. If you store a meeting as “October 15, 2027 at 3 PM in Mexico City,” and Mexico cancels DST between now and then, which it did, in 2022, the meeting moves. The number of hours from now until that meeting changes after you saved it. The cardinal rule, the only thing that saves you, is this: store UTC and the IANA zone name separately, never store a local timestamp alone, and recompute on display.
4. JavaScript’s
Dateobject. It’s not great but I worte more about it here, There is a fix called the JavaScript Temporal API. It’s technically here now, though browser support is still rolling out, and we will all be happier when it’s fully supported everywhere.
The list of lies (about time)
There’s a famous post called Falsehoods Programmers Believe About Time, and a partial sample from the time-zone section gives you the texture:
- “There are 24 time zones.” (38+, give or take, depending on the week.)
- “A day is 24 hours.” (DST transitions make some days 23 or 25.)
- “Time zones don’t change.” (They change several times a year.)
- “If I store the UTC offset I don’t need the zone ID.” (You do, for any future date.)
- “UTC is a time zone.” (UTC is a time scale. Zones are offsets from it. This distinction is going to matter more than it sounds like it should.)
All are gotchas that programmers encounter when working with time zones.
The ultimate example of technical debt
The time zone system isn’t broken, per say… it’s working exactly as designed.
It was designed by railroad executives in 1883, ratified by diplomats in 1884, and then handed off to every country on Earth to amend at will. Every president who has ever moved a DST date for political reasons, every dictator who has ever changed the national offset to flatter a neighbor, every parliament that has voted to abolish daylight saving without specifying when, has added their fingerprint to the IANA database.
It is a working international system. It is also a Rube Goldberg machine running on a tar pit, held aloft by Paul Eggert and a mailing list.
If you ever thought Timezones were bad, tomorrow it gets worse. We’re going to talk about Daylight Saving Time, and the specific, named software disasters it has caused.
Sources
- Nepal Standard Time — Wikipedia. Details Nepal’s 1986 decision to anchor civil time to the Gauri Shankar meridian (
UTC+5:45). - Time in Kiribati — Wikipedia. Covers the 1995 shift of the International Date Line, creating the 26-hour global spread.
- Time in North Korea — Wikipedia. Documents the geopolitical shifts between
UTC+8:30andUTC+9. - Day of Two Noons — Wikipedia. History of American railroads unilaterally standardizing time on November 18, 1883.
- International Meridian Conference — Wikipedia. The 1884 agreement that established Greenwich as 0° (and France’s holdout until 1911).
- IANA Time Zone Database — Wikipedia. History of the Olson database, its maintenance by Arthur David Olson and Paul Eggert, and its fundamental role in modern computing.
- Daylight saving time in Mexico — Wikipedia. Details the national abolishment of DST in October 2022.
- JavaScript Temporal API — TC39 Documentation. The modern fix for JavaScript’s notoriously broken
Dateobject (currently rolling out to browsers). - Falsehoods Programmers Believe About Time — Noah Sussman’s canonical post detailing the myriad ways developers misunderstand time.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Day 16: How the World Agreed on a Date Format (Except the US)
There is a war that has been quietly raging for about a century, and it is fought over six little characters.
05/06/26In the United States, that is May 6, 2026. In most of Europe, it is June 5, 2026. In Japan, which uses a year-month-day order and frequently uses the imperial calendar, the
05might be read as year 5 of the Reiwa era (2023). In Iran, the entire premise is wrong, because Iran’s official calendar is the Solar Hijri and the current year is 1405.So here we are. A date written by one person and parsed by another is, in the general case, an act of faith.
The most consequential standards effort of the late 20th century was an attempt to end this. It succeeded, sort of. The format it produced,
2026-06-08T14:30:00Z, looks unremarkable now, but it represents a multi-decade campaign to drag the world’s date conventions into a single, unambiguous, machine-parseable shape.That standard is ISO 8601, and the story of how it won is the story of why your API logs look the way they do.
What’s actually wrong with
05/06/26Let me give you the tick of the tock (lay of the land). Every culture has a different intuition about which number comes first in a date, and none of them is wrong.
In the United States, the convention is month-day-year. This descends from spoken American English, “May sixth, twenty-twenty-six,” where the month comes first in speech.
In most of Europe, Latin America, and much of Asia, the convention is day-month-year. “The fifth of June, twenty-twenty-six.” The day is the first specific element.
In East Asia, the convention is year-month-day, written largest-unit-first. This reflects a linguistic preference for going general-to-specific that runs the opposite direction of the English phrasing.
Who is to say which is more correct than another? The problem is that a single string of digits separated by slashes can mean three different things depending on who wrote it, and there is no way other way to tell.
When the ambiguity bites
It’s not just an annoyance.
International travel figured this out the hard way. Across global passport documentation, the convention settled on a three-letter month abbreviation:
08 JUN 2026. It’s unambiguous because no month is named06. The international passport standard (ICAO Doc 9303) mandatesJANthroughDECfor exactly this purpose.In healthcare, patient safety organizations have flagged date ambiguity as a documented source of medication error: a chart that says
7/8/09can be read as July 8 by one clinician and August 7 by another.The shape of the problem is the same across medicine, aviation, logistics, contracts, and customs declarations. Different conventions lead to confusion and errors.
The standard
In 1988, ISO published
ISO 8601:1988. Pick one format, make it unambiguous, make it sort lexicographically, make it machine-parseable, and standardize the world on it.The format they picked:
2026-06-08T14:30:00ZThe choice of
YYYY-MM-DDwas deliberate. Year-month-day is the East Asian convention, but it has a technical property that the other two don’t: it sorts correctly as a string.2025-12-31comes before2026-01-01whether you sort by character or by number.12/31/25and01/01/26do not. For the emerging computing industry of the late 1980s, databases, log files, file systems, this was a decisive advantage.The capital
Tseparates the date from the time. Not pretty, but unambiguous. The trailingZ(informally pronounced “Zulu”) means UTC. This timestamp has no timezone offset, it is anchored directly to UTC.What actually use: RFC 3339
ISO 8601 is too permissive for engineering use.
It allows fractional seconds. It allows omitting components. It allows the basic form (
20260608T143000Z) without separators. It allows week dates and ordinal dates. It allows24:00:00as midnight (this was removed in 2019, then reinstated by amendment, in one of those standards-committee compromises that satisfies no one).So in 2002, the IETF published
RFC 3339. RFC 3339 is a profile of ISO 8601, a strict subset that picks one form and forbids the rest. The basic form is disallowed. Week dates are disallowed. The time component is mandatory. The timezone designator is mandatory.This is what every modern internet API actually uses. GitHub, AWS, Stripe, Cloudflare, OpenAI. They accept RFC 3339, not full ISO 8601. They reject
20260608T143000Zeven though it’s legal ISO 8601.What everyone calls “ISO 8601” in casual conversation is, almost always, RFC 3339.
What ISO 8601 isn’t
A few things worth being clear about:
- ISO 8601 is not UTC. UTC is a timescale. ISO 8601 is a format.
- ISO 8601 is not Unix time. Unix time is the integer
1781055000. ISO 8601 is the string2026-06-08T14:30:00Z. They can represent the same instant. They are not the same thing. - ISO 8601 does not solve leap seconds. The format permits
:60in the seconds field, but what to do with such a value is implementation-defined. - ISO 8601 does not include the calendar system. It assumes the Gregorian calendar. No provision for Islamic, Hebrew, or Buddhist calendars.
The civilizational payoff
There is a sense in which
2026-06-08T14:30:00Zis the most consequential string format in modern computing.While legacy systems still cling to their own formats—HTTP headers use RFC 1123, Git and JWTs use integer Unix timestamps, and X.509 certificates use ASN.1—RFC 3339 has conquered the modern web. It is the default serialization for datetime objects in modern programming languages. It appears in the JSON payloads of almost every modern API (GitHub, Stripe, AWS, OpenAI). It is the standard format for XML’s
xs:dateTime. It is written into millions of cloud infrastructure log lines every second.It is the closest thing modern technical infrastructure has to a universal vocabulary for the question “when did this happen?"
It won because it was unambiguous and sortable, and a single committee was willing to pick one of three equally valid cultural conventions and tell the other two cultures to deal with it. Most international standards die in the negotiation. ISO 8601 survived because the technical advantages of
YYYY-MM-DDwere strong enough to overwhelm the political cost.Us Americans haven’t adopted it (yet). We still write
06/08/2026on bank checks, forms and filings, but the machines we all use are on 8601 and they are doing most of the talking.
Sources
- Japanese era name — Wikipedia — Reiwa began 1 May 2019; Reiwa 5 = 2023.
- Solar Hijri calendar — Wikipedia — year 1405 began 21 March 2026, ends 21 March 2027.
- ISO 8601 — Wikipedia — first published 1988; ISO 8601-1:2019 removed
24:00; the 2022 amendment reinstated it. - ISO 8601-1:2019/Amd 1:2022 — the amendment that put
24:00:00back. - RFC 3339 — Date and Time on the Internet: Timestamps — IETF, July 2002. Profile of ISO 8601 used by most modern APIs.
- RFC 3339 vs ISO 8601 — visual map of which forms each standard accepts; basic form (
20260608T143000Z) is valid ISO 8601 but not RFC 3339. - Machine-readable passport (ICAO Doc 9303) — Wikipedia — ICAO standard requiring three-letter month abbreviations (
DD MMM YYYY) in the visual inspection zone of all passports. - ISMP List of Error-Prone Abbreviations — highlights the risk of ambiguous documentation and dates in medical records.
- RFC 1123 — Requirements for Internet Hosts — specifies the required date format for HTTP Date headers.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Tomorrow: Unix time, the second-counting system that runs under every timestamp you’ve ever seen, and the rollover problem that hits in 2038.
Programming Software development 30daysoftime Standards Iso8601
-
Day 15: The Man Who Synchronized the World
David Mills, the father of internet time, wrote the protocol that synchronizes every computer on Earth. He did it as a professor at the University of Delaware, on a project he started in the early 1980s and never stopped working on.
The code lives in your laptop, your phone, your router, every cloud server you’ve ever touched, and every satellite in low Earth orbit. The protocol it implements is called NTP. The reason your computer’s clock is correct, right now, within a few milliseconds of UTC, is that Mills spent forty years of his life making sure it would be.
He once described the early ARPANET days as a “sandbox” where researchers were simply told to “do good deeds.” Part of the allure of the time-synchronization work, he told The New Yorker in 2022, was that he was just about the only one doing it. He had his own “little fief.”
For forty years, that is exactly what it was.
The problem
The early internet had a clock problem. As soon as there were enough machines on the network that “what time is it?” didn’t have a single answer, somebody was going to have to write a protocol. Each computer had its own oscillator. Each oscillator drifted at its own rate. Two machines that agreed at noon could be tens of seconds apart by midnight.
Why did this matter? For most things, it didn’t. For some things, it mattered a lot. A file saved on one machine and copied to another could look older than the version it overwrote, confusing every backup tool that assumed time moves forward. Cryptographic handshakes that expire after a few seconds could fail because the two ends disagreed on what “a few seconds ago” meant. Database replicas could apply writes in the wrong order and corrupt their own state. Email between two servers could arrive timestamped before it was sent. Debugging a multi-machine bug meant correlating log entries across clocks that didn’t agree about which event came first.
Mills decided the actual problem was that there was no protocol for negotiating the truth (in this case, time) across multiple systems. The clock on his desk was wrong. Every other clock was also wrong. The question wasn’t “who has the right time?", it was “given that nobody has the right time and the network adds an unknown delay to every measurement, how does the system converge on a consensus that is closer to UTC than any individual node could achieve alone?"
His first NTP RFC,
RFC 958, was published in September 1985. We now call that protocol NTPv0, or the prototype. In it, Mills nailed down the four-timestamp packet format and the offset/delay math that has been in every revision since. The packet format and the core algorithm haven’t meaningfully changed in forty years. That kind of staying power is rare in any field. In internet infrastructure, where the half-life of a protocol can be measured in single-digit years, it is quite commendable.The four timestamps
NTP’s core insight is that the network delay between client and server can be measured, not just guessed, as long as both sides record their own timestamps for both legs of the conversation. Four timestamps are exchanged in a single round trip:
Client Server ────── ────── T₁ ──── request ───────────────► T₂ T₃ T₄ ◄────────────── response ─────- T₁ — the client sends the request (client clock)
- T₂ — the server receives it (server clock)
- T₃ — the server sends the response (server clock)
- T₄ — the client receives it (client clock)
Now the client has four numbers. T₁ and T₄ are in the client’s reference frame, T₂ and T₃ are in the server’s. From those four numbers, two things fall out: the round-trip delay (how long the conversation took, minus the time the server spent thinking) and the clock offset (how far the client’s clock is from the server’s). The client now knows how wrong it is, and by how much.
The math depends on one critical assumption: the network is symmetric. The packet takes the same time to travel in both directions.
If you’ve been following along in the series, you know there are a lot of ways to measure time. Atomic clocks. GPS receivers. The quartz crystal in your laptop. Radio signals broadcast from government antennas. They don’t all tick at the same rate, and they don’t all agree on what the current time is. How does NTP reconcile across that much varity in time sources?
The stratum hierarchy
NTP organizes the world’s clocks into a tree, with depth measured in strata.
Stratum 0 is the reference. Cesium atomic clocks. Hydrogen masers. GPS receivers. Radio receivers tuned to WWV, DCF77, or MSF. These are not on the network, they’re physical devices wired directly to a small number of computers via PPS pulses on serial ports.
Stratum 1 is the small group of servers wired directly to Stratum 0. There are perhaps a few thousand of these globally. NIST runs some. Major universities run some. The big internet exchanges run some.
Stratum 2 servers sync with Stratum 1, Stratum 3 with Stratum 2, and so on down to Stratum 15. Stratum 16 means “unsynchronized, do not trust.”
A typical Linux laptop syncs against Stratum 2 or 3 servers. A typical cloud VM syncs against its provider’s internal Stratum 1 fleet. Your phone syncs against whatever its carrier provides. The whole tree is held together by NTP itself, recursively.
The genius of the design is that there is no central authority. Mills did not own the protocol. There is no “official NTP server.” Anyone can run a Stratum 1 with the right hardware, and anyone can run a Stratum 2+ by syncing with a few Stratum 1s of their choice. The largest public pool,
pool.ntp.org, is a volunteer effort started in 2003 by Adrian von Bidder. It currently aggregates a few thousand donated stratum-2 servers worldwide and serves several billion requests per day. Nobody is in charge of it. It just works.The slew, not the step
There are three different times to keep track of on every synced computer. The reference time is what UTC says, the truth NTP is chasing. The tick rate is how fast the computer’s oscillator pulses. It’s supposed to produce one second of clock time per real second, but always drifts a little. The system clock is what gets reported when an application asks for the current time. Synchronizing means closing the gap between the system clock and the reference time without breaking anything that depends on the system clock being well-behaved.
NTP’s primary tool for that is the slew: it adjusts the tick rate, making each tick slightly longer or shorter than nominal, so the system clock drifts into alignment on its own. The alternative would be to jump the clock forward or backward by the full offset (a step), which is fast but can produce duplicate keys in a database, expire valid TLS sessions, or cause a logging system to mis-order events.
Mills designed
ntpdto slew conservatively. A 200ms gap might take several minutes to close, and corrections larger than about 128ms would get stepped because slewing them gradually was prohibitively slow. That trade-off worked for the always-on Unix workstations of the 1980s and 90s. It works less well for the modern reality of laptops that suspend for hours and resume with a clock that hasn’t been touched since last Tuesday, or cloud VMs that get migrated between hosts. Modern variants likechronyslew more aggressively for exactly that reason. When you open your laptop lid, you want the clock right now, not after fifteen minutes of imperceptible easing.The legacy
In a sense, NTP is the thing that made the modern internet possible.
Without well-synchronized clocks, you cannot have SSL certs. The browser needs to know when the cert expires, and if its clock is off by more than a few minutes, the encryption breaks. The same goes for databases. No matter the type, NoSQL or otherwise, they all depend on a clock to record when an operation took place.
Without NTP, cell towers wouldn’t agree on when to hand off a call. Financial transactions wouldn’t be enforceable. And all those log files you’ll totally read one day wouldn’t make any sense. NTP is foundational to all of it. It runs as a daemon on every machine, the ones you stare at all day, the ones you don’t see, and the ones you don’t care about.
We remember Mills as the internet’s “Father Time” and the man who synchronized the world. Neither is a metaphor.
Sources
- In Memoriam: David Mills (UDaily, March 2024) — University of Delaware’s obituary; biographical detail, career timeline.
- David L. Mills — Wikipedia — congenital glaucoma from birth, vision worsening from ~2012, fully blind by 2022; UDel professor 1986–2008.
- David Mills, the internet’s Father Time, dies at 85 — The Register — death date (Jan 17, 2024), age 85.
- RFC 958 — Network Time Protocol (September 1985) — the original NTPv0 specification.
- Network Time Protocol — Wikipedia — version lineage: RFC 958 (v0, 1985), RFC 1059 (v1, 1988), RFC 1119 (v2, 1989), RFC 1305 (v3, 1992), RFC 5905 (v4, 2010), RFC 8915 (NTS, 2020).
- NTP pool — Wikipedia — Adrian von Bidder started the pool in January 2003; Ask Bjørn Hansen has run it since 2005.
- MiFID II RTS 25 clock synchronization (Meinberg) — 100µs requirement for high-frequency trading at sub-1ms gateway latency.
- A Brief History of NTP Time: Confessions of an Internet Timekeeper (Mills, PDF) — Mills' own history of NTP.
- The Thorny Problem of Keeping the Internet’s Time (The New Yorker, September 2022) — Nate Hopper’s profile of David Mills and the fragile state of NTP maintenance.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Tomorrow: ISO 8601, the format wars, the carnage of MM/DD vs DD/MM, and why
2026-06-07T14:30:00Zwon. -
Day 14: The Bug That Didn't End the World, and the One That Still Might
On December 31, 1999, a measurable percentage of the developed world stockpiled bottled water, withdrew cash from ATMs, and stayed up to see if the lights would go out at midnight.
They didn’t. Planes did not fall from the sky. Power grids did not collapse. Bank balances did not reset. The new millennium arrived, the champagne was opened, and by January 3rd everyone agreed it had been a hoax.
It was not a hoax. It was a save.
Y2K was a global $300+ billion engineering effort spread across roughly five years and almost every government and Fortune 500 IT department on Earth. The reason nothing happened on January 1, 2000 is that for half a decade, an enormous number of people worked very hard so that nothing would happen. The bug was real. The fix worked. Most people forgot it was ever a problem.
Twelve years from now, a structurally identical bug detonates again, but first let’s understand what happened in ‘99.
Y2K: the bug
The Y2K bug is almost embarrassingly simple. From the 1960s through the 1980s, computer storage was expensive enough that programmers had a habit of representing years with two digits,
99instead of1999,73instead of1973. It saved two bytes per date. Across a payroll system tracking millions of employees, that mattered.The assumption baked into that decision was: we’ll have rewritten this system long before the century rolls over.
This is the most consistently wrong assumption in software engineering. Code outlives its authors’ confidence. By the late 1990s, vast amounts of critical infrastructure, bank ledgers, airline reservation systems, hospital records, utility billing, social security disbursement, military logistics, nuclear plant monitoring, were running on COBOL programs from the 60s and 70s that had been patched but never rewritten. The language is unfamiliar to many but the fix it later approach is relable to everyone. The developers at the time all quietly assumed that the year
99was less than the year00.When the rollover hit,
99-12-31 + 1 day = 00-01-01looked, mathematically, like jumping back to 1900. Interest calculations would compute negative ages. Pensioners would suddenly be billed for a century of debt. Reservation systems would mark every upcoming flight as having departed in the past. Insurance policies would expire en masse.The reason planes did not fall is that, starting roughly in 1995, every major airline, manufacturer, FAA system, and air traffic controller began an exhaustive audit-and-fix campaign.
The reason the power grid did not collapse is that every utility company in North America and Europe ran the same campaign on their SCADA systems.
The reason your bank balance was still correct on January 1, 2000 is that someone, somewhere, spent late nights in 1997 reading printouts of code written before they were born.
The estimated total global cost: $300 to $600 billion. The amount of measurable damage on January 1, 2000: small enough that people argued for the next decade about whether the spend had been justified.
It was. The bug was real. The fix worked. The result of a successful preventive engineering campaign is that it looks, in retrospect, like the problem was never there.
Y2038: the same bug, different number
Twelve years from now, specifically, January 19, 2038, at 03:14:07 UTC, a structurally identical bug fires for a different reason.
Unix time is stored, on a huge amount of legacy infrastructure, as a signed 32-bit integer. That gives you about 2.1 billion seconds of positive range from the 1970 epoch. 2.1 billion seconds is 68 years. 1970 + 68 = 2038.
At
03:14:07 UTCon that date, the counter hits its maximum value,2,147,483,647. The next tick overflows. In two’s-complement signed integer arithmetic, the value rolls over to its most negative possible value:-2,147,483,648. Interpreted as a Unix timestamp, that’s December 13, 1901.Every 32-bit Unix-derived system that hasn’t been patched will, in the span of one tick, conclude that it is now the early 20th century. The effects are the same family of effects as Y2K, but applied to a much wider deployment surface.
File modification times become nonsensical. SSL certificates appear expired, or worse, not-yet-valid. NTP synchronization fails. Filesystems with 32-bit inode timestamps lose ordering. Embedded device firmware that schedules tasks based on wall-clock time begins executing at random intervals. Industrial control systems that latch state machines on “time since last event” calculations latch on negative durations and either freeze or behave unpredictably.
Modern desktop and server operating systems are mostly fine. Linux finished migrating to 64-bit
time_ton all architectures by kernel 5.6 (2020) and glibc 2.32. macOS and Windows have been 64-bit-clean for over a decade. AWS, GCP, and Azure all run 64-bit kernels.The problem is not where you are reading this. The problem is in the physical world that keeps everything running.
The long tail is enormous
Estimates of the number of currently deployed 32-bit embedded devices that interact with
time_tin some way range from a few hundred million to several billion.Industrial controllers, automotive ECUs, network routers, smart-meter firmware, point-of-sale terminals, medical imaging devices, GPS units, cable boxes, elevator controllers, traffic light systems, ATM internals, payment terminals, building HVAC, water-treatment SCADA, satellite firmware, oil rig control systems, and the embedded computer in your refrigerator.
Each one, depending on vintage and vendor, may or may not have been patched.
Many of these devices are not internet-connected and cannot be patched remotely. Many are running firmware whose source code has been lost. Many are running firmware whose vendor no longer exists. Many are in places where physical access is hard, a deep-sea oil platform, a satellite in geostationary orbit, a controller welded inside an industrial machine.
The Y2K fix worked because the affected systems were largely centralized: mainframes in data centers, software at named companies, code with active maintainers. You could audit it. You could rewrite it. You could ship a patch.
Y2038 is decentralized. The affected systems are everywhere.
The Buff Must Flow
In 2022, Microsoft Exchange Server stopped delivering email worldwide. The cause was a 32-bit signed integer in Exchange’s anti-malware scanner that stored the date as a long-form number. On New Year’s Day, the value tipped over the limit and the scanner refused to load. Mail queues backed up everywhere. Microsoft shipped an emergency script the next day. They called it Y2K22.
On April 6, 2019, the GPS week number counter rolled over. The failure mode was familiar, an integer designed when the engineers thought it was going to be big enough turned out, decades later, not to be. NYC’s municipal wireless network went down. KLM grounded a flight. Older car and marine GPS units showed dates in 1999.
Two examples of overflows hitting production and breaking real things. Y2038 will be every one of those at once, in places nobody is thinking about.
Y2038 is foreseeable. We know about it. We know what needs to be done. We have twelve years. We should get started sooner rather than later. A lot of important systems need to be replaced, and the fewer that fall through the cracks, the better.
There’s no checklist for the devices we’ve already forgotten about, but maybe there should be.
Tomorrow: The Smear, how Google, Amazon, and Meta quietly decided to stop telling the truth about leap seconds, and why everyone else followed.
Sources
- Year 2000 problem — Wikipedia
- Year 2038 problem — Wikipedia
- Microsoft Exchange year 2022 bug in FIP-FS breaks email delivery — BleepingComputer
- Microsoft Exchange Fixes Disruptive ‘Y2K22’ Bug — BankInfoSecurity
- GPS week number rollover — Wikipedia
- GPS Week Number Rollover — GPS.gov
- The impact and resolution of the GPS week number rollover of April 2019 — Geoscientific Instrumentation (Copernicus)
- Linux kernel 5.6 — 64-bit time_t support for 32-bit architectures (KernelNewbies)
- The Open Group Base Specifications: time.h
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Day 13: Unix Time, 1,780,620,532
That’s roughly what time it is, right now, as I type this.
Not 8:48 PM. Not “Thursday.” Not “June 4th, 2026.” None of those are what your computer thinks “now” is. To your laptop, your phone, your car’s infotainment system, the streaming server pushing this page to your browser, and the ATM in the corner store, now is a number. A 10-digit integer. Counting up, one tick per second, since a fixed moment in 1970.
That number runs the world. It’s the closest thing the global computing infrastructure has to a heartbeat. And it has some weird properties, almost none of which are explained by the name it goes by.
Unix time.
The clock under every clock
Open a terminal. Type
date +%s. You’ll see something like1780620532come back. That’s Unix time. Seconds since the Unix epoch,1970-01-01T00:00:00 UTC.Every modern operating system tracks time this way internally, even if it dresses up the output for you. The pretty “8:48 PM” on your menu bar is a calculation: take the current Unix timestamp, apply your timezone offset, run it through the calendar rules, format it for display. The underlying number is just
1,780,620,532-and-change, counting up.JavaScript’s
Date.now()? Unix time in milliseconds. Java’sSystem.currentTimeMillis()? Unix time in milliseconds. Python’stime.time()? Unix time as a float. Go’stime.Now().Unix()? Unix time. PostgreSQL’sEXTRACT(epoch FROM ...)? Unix time. SQLite’sstrftime('%s', 'now')? Unix time.It’s the lingua franca of computing. Two systems written in different languages, on different continents, with different calendars in their UIs, agree about what now means because they both agree about this one number.
Why 1970?
The honest answer is: convenience.
In the early 1970s, Ken Thompson and Dennis Ritchie were building Unix at Bell Labs. They needed a way to represent time on a 32-bit machine. Their first attempt counted 1/60 of a second per tick in a 32-bit integer, and overflowed in about two and a half years. So they switched to 1 tick per second, which gave them roughly 136 years of range in a signed 32-bit integer.
Then they needed a zero. They picked
1970-01-01because:- It was recent enough that the historical calendar mess (Julian vs. Gregorian, the dropped days in 1582, the year that started in March) was someone else’s problem.
- It was round.
- It predated every Unix system anyone cared to represent.
- It was conveniently close to UTC’s formalization a couple of years later.
That’s it. There’s no cosmological significance to 1970-01-01. It’s not aligned with any astronomical event. It’s the timestamp equivalent of
git init. We’ll start counting from here, and we’ll figure the rest out later.The “later” turned out to mean everywhere.
The thing that isn’t there: leap seconds
The computer’s time problem mostly comes from UTC.
Unix time is defined as the number of seconds since the Unix epoch. You might reasonably assume that if I have two timestamps, the difference between them is the actual number of physical seconds that elapsed between those two moments.
It is not.
Unix time does not count leap seconds. Since 1972, the IERS has inserted 27 leap seconds into UTC, extra seconds added to keep civil time aligned with Earth’s slowing rotation. Unix time pretends they never happened. The Unix clock has, over its 56-year lifetime, “lost” almost half a minute relative to reality.
Even weirder: during the actual leap second, when UTC ticks
23:59:59 → 23:59:60 → 00:00:00, Unix time has to do something. POSIX doesn’t specify what. So implementations have invented three different answers:- Repeat the second. The clock shows
23:59:59for two real seconds and then jumps to00:00:00. Two distinct physical moments share the same timestamp. File mtimes can collide, log entries can appear out of order. - Insert the second. The clock briefly shows
23:59:60, which is a valid UTC string but breaks every parser that assumes seconds run 00–59. Linux kernels do this. Hilarity ensues at midnight. - Smear it. Don’t insert the second at all. Slow every clock down by a tiny fraction over a 24-hour window so it absorbs the missing second smoothly. Google does this. Amazon does it. Facebook does it.
So “Unix time” in 2026 means three subtly different things depending on whether your server is running stock Linux, smeared Google time, or one of the dozens of variants in between. Two timestamps from two providers may disagree by a second, and both are correct under their own definitions.
That’s what the spec authors call “implementation-defined behavior” and what the rest of us call “why distributed-system logs don’t line up.”
The number is also a string
Integers are easy for computers but humans expect a string. Unix time is the easiest timestamp format to compare, sort, and store because it’s an integer, but as soon as we convert to human-readable format, all that changes.
To find out which one is earlier, subtract. To sort a million events, sort the integers. To store one efficiently, write 8 bytes. To send one over the network, send 8 bytes.
Compare this to a full ISO 8601 timestamp like
2026-06-04T16:47:23.512847+00:00. That’s a 32-character string that needs to be parsed, validated, normalized for timezone, and converted to a comparable representation before you can do anything with it. Every comparison is a parsing pass. Every storage is 4× the bytes. Every sort is a string sort with calendar rules.Unix time is fast. It’s so fast that even formats designed to replace it (Google’s Spanner, AWS’s KSUIDs, Twitter’s Snowflake) embed Unix-like millisecond counts at their core and just append entropy bytes around them.
The ubiquity isn’t an accident. It’s the natural result of picking the representation that’s cheapest at every step.
The Untimes
Unix time is a convention that has eaten the world.
It’s anchored to UTC, which means it inherits UTC’s quirks. It’s embedded controllers in cars, industrial equipment, network gear, satellite firmware, gas pumps, so pretty much every piece of modern infrastructure.
1,780,620,532is just a number, a timestamp. It’s used by your bank for transactions, used by your file system for its files, but also it’s a hack. A 56-year-old dart in the board of of time, that ignores leap seconds, depends on UTC, has three different definitions during the same physical second, and we built the entire internet on top of it.Tomorrow will be on what happens when the bill comes due. Y2K and Y2038, the bug that didn’t end the world, and the bug that still might.
Sources
- Unix time — Wikipedia
- Leap second — Wikipedia
- Coordinated Universal Time — Wikipedia
- International Earth Rotation and Reference Systems Service — Wikipedia
- Leap Smear — Google Developers
- Look Before You Leap — The Coming Leap Second and AWS
- It’s time to leave the leap second in the past — Engineering at Meta
- How Precision Time Protocol handles leap seconds — Engineering at Meta
- Leap second bug cripples Linux servers at airlines, Reddit, LinkedIn — The Register
- Resolve Leap Second Issues in Red Hat Enterprise Linux
- History of Unix — Wikipedia
- Snowflake ID — Wikipedia
- ksuid — segmentio (GitHub)
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Declutter your JavaScript & TypeScript projects
Project linter to find unused dependencies, exports and files
Programming Tools Dev links code javascript software engineering
-
Day 10: The Zero Point
Three epochs quietly run the world:
- The Unix epoch. Midnight, January 1, 1970. Almost every computer measures time as seconds since this instant.
- The GPS epoch. Midnight, January 6, 1980. Every GPS satellite, every navigation chip in every phone, measures time as seconds since this instant.
- The astronomical epoch (J2000.0). Noon, January 1, 2000, Terrestrial Time. Almost every star catalog, planetary orbit calculation, and space mission uses this instant.
Three different zeros. Three different conventions. None of them line up with anything you’d find on a calendar. Here is why.
You Can’t Have a Clock Without a Zero
A clock counts intervals. To tell you what time it is right now, it needs to know how many intervals have passed since something. The “since something” is the epoch: a fixed, agreed-upon instant from which all measurement runs.
Most timekeeping systems hide their epoch behind a calendar facade. “April 14, 2026” is meaningful to humans, but underneath, the computer is doing arithmetic on a single integer counted from a particular zero.
The calendar is the friendly mask.
The epoch is the actual machinery.
The Three Big Epochs
Unix Epoch: January 1, 1970, 00:00:00 UTC
Picked in the early 1970s by the engineers building Unix. They needed a zero point for the system’s internal
time_tinteger. 1970 was recent enough to feel current, far enough away to leave room for negative numbers (events before 1970), and round enough to remember.I think that they probably thought, like, well, if time is all relative, then let’s just pick some arbitrary time and it doesn’t matter.
It was an choice, not an astronomical one, just relative to some arbitrary point they decided.
So let me say that again.
The Unix epoch has no relationship to any natural event. It is a convention that, through the pervasive nature of Unix, became the default for all modern computing.
GPS Epoch: January 6, 1980, 00:00:00 UTC
The GPS satellite constellation started broadcasting on January 6, 1980. The epoch was just the moment the system turned on.
Why January 6? Because that’s a Sunday, and the GPS week-counting system uses weeks, and weeks start on Sunday.
The first GPS week is week zero.
GPS time has run continuously from that instant and has never had a leap second adjustment, so it is currently 18 seconds ahead of UTC, a gap that keeps growing.
But more on that in a tomorrow’s post.
J2000.0: January 1, 2000, 12:00:00 Terrestrial Time
This is the astronomers' epoch, and it’s the most carefully chosen of the three. Notice two things:
- It’s noon, not midnight.
- It’s in Terrestrial Time, not UTC.
Both choices have reasons.
Why noon? Astronomers observe at night. A “day” for an astronomer historically started at noon and ran through the following noon, so a single night’s observation session never straddled a date boundary.
If you started a date at midnight, half the stars you saw last night would log on one date and half on the next.
Annoying for astronomers, so they decided to reduce their suffering by redefining the epoch.
The Julian Date system, introduced by Joseph Scaliger in 1583, runs from noon to noon for this reason.
Noon TT on January 1, 2000 was Julian Date 2,451,545.0 exactly, a perfectly round Julian-Date integer.
Why such a huge number?
Because Julian Dates count days from noon on January 1, 4713 BC, the start of Scaliger’s count.
He picked that year because three big calendar cycles (solar, lunar, and the Roman indiction) all aligned there, and because it sat well before any recorded astronomical observation, so every date in history would be a positive integer.
By noon on January 1, 2000, exactly 2,451,545 days had elapsed.
The “0” at the end of “J2000.0” is a flag for that round number, a clean integer in a counting system older than telescopes.
Why Terrestrial Time and not UTC? Because UTC has leap seconds and Terrestrial Time doesn’t.
TT is the smooth atomic timescale we built two days ago (TAI + 32.184 seconds). Anchor your epoch to UTC and every leap second shifts your historical observations sideways. Anchor it to TT and it stays put. That’s why the canonical zero is in TT.
TAI: 2000-01-01 11:59:27.816 UTC: 2000-01-01 11:58:55.816 TT: 2000-01-01 12:00:00.000 ← this is J2000.0Other Epochs Worth Knowing
A few more that show up in working systems:
- Modified Julian Date (MJD): November 17, 1858, midnight. Used in space-mission control because it drops the leading digits of a full Julian Date, saving bytes in old memory-constrained systems.
- TAI origin: January 1, 1958, midnight UT2. The instant the cesium-coordinated TAI scale started running.
- Year zero of the Gregorian calendar: there isn’t one. The calendar jumps from 1 BC to 1 AD with no year zero in between, breaking date arithmetic across the boundary and serving as a low-grade gotcha in historical software.
The Deep-Time Temptation
Some people, looking at this collection of arbitrary-feeling start points, ask why we don’t just pick something physically meaningful. The formation of the Earth, the formation of the solar system, the Big Bang.
The answer is precision.
We don’t know any of those instants to better than millions of years. Earth formed roughly 4.54 billion years ago, plus or minus 50 million. The solar system, 4.567 billion years ago, plus or minus 1 million. The Big Bang, 13.8 billion years ago, plus or minus 20 million.
A reference epoch that is uncertain to a million years isn’t a reference…
The astronomical zero needs to be knowable to the nanosecond, recoverable in the future from preserved records, and verifiable against real observations.
Of every candidate, J2000.0 is the best at all three.
Modern atomic clocks were running in 2000. Star positions on that day are catalogued.
The exact instant is recorded across thousands of observatories.
If civilization collapses and is rebuilt, J2000 is recoverable from physical artifacts. The formation of the Earth is not.
What the Epoch Is Doing
Pick your epoch and you pick what your system can and can’t represent.
- Unix time can’t go before 1970 without negative numbers, and there is the whole integer-overflow issues after a few centuries.
- GPS time started in 1980 and counts strictly forward. Nothing before is representable.
- J2000.0 sits at the present, so calculations naturally span backwards and forwards by tens of thousands of years with full precision.
The choice of epoch is often the most invisible design decision in a timekeeping system, but it shapes everything downstream.
Some of the strangest bugs in software history, Y2K, the 2038 problem, GPS week rollovers, trace back to picking a zero without thinking about the consequences.
Tomorrow we’ll see what happens when one of those choices has to deal with relativity, gravity, and the curvature of spacetime.
The Gee-Pee-Ess time, and the clocks that ship from the factory wrong on purpose.
Sources
- Unix time — Wikipedia
- GPS time — Wikipedia
- Epoch (astronomy) — Wikipedia
- Julian day — Wikipedia
- Terrestrial Time — Wikipedia
- Year 2038 problem — Wikipedia
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Programming 30daysoftime Astronomy Timekeeping Computing-history
-
A Dotfiles Manager That Snapshots Every Change
Managing dotfiles in 2026 is a solved problem in the same way that managing your own backups is a solved problem: there are five tools for it, all of them work, all of them require you to set up some plumbing first, and once you’re set up you still don’t have a great answer to “I just broke my shell config, get me back to yesterday.”
The conventional answer is some combination of: a git repo for your
~/.zshrcand friends, a symlink script (orstow, orchezmoi, oryadm), and the discipline to remember to commit after every change. The setup is a one-time hassle. The “wait, what did I change?” recovery story is not great. And if you want to sync across machines, you’ve now got opinions about remote repos, SSH keys on a fresh box, and which order things have to happen in.I wanted something different, so not a configuration framework, but a record of every change to the files I care about, in a place I can roll back from, with the lowest possible setup cost.
That’s what dfm is.
What It Does
dfmis a single static Go binary. You point it at the files you want to track (~/.zshrc, anything under~/.config/, whatever), and every time one of them changes it takes a content-addressed snapshot. The snapshots live on disk in~/.local/share/dotfiles/backups/. A small state database (SQLite locally, or libSQL via Turso if you want cross-machine sync) records which file maps to which snapshot at which point in time.You can roll back. You can diff against an old snapshot. You can see when you last touched a file. And because every snapshot is content-addressed, you never re-store the same bytes twice — switching themes in
~/.zshrcten times costs the size of two configs, not ten.The other half is the backup story.
dfm initwalks you through cloning (or creating, viagh) a private GitHub repo that mirrors your tracked files plus their history. The point isn’t to make you adopt a new git workflow. It’s that pulling your config onto a fresh machine should be one command, and recovering fromrm -rfshould never have a “well, hopefully my last commit was recent” caveat.Why Setup Is the Hard Part
The reason people don’t audit their dotfiles is the same reason people don’t back up their laptops: the setup is annoying, and the payoff is theoretical until it isn’t.
dfm initis a six-step interactive wizard. It detects aTURSO_DATABASE_URLenv var if you’ve got one, offers sensible defaults for everything else, lets you opt in to tracking~/.zshrcimmediately, and writes a single config file with the right permissions. Re-run it on an existing config and it pre-fills every prompt with your current value, so the cost of changing your mind later is also low.--yesaccepts every default for scripted setup.If that sounds boring, that’s the point. Boring is what makes a tool actually get used.
The AI Bit
There’s an optional AI integration.
dfm suggest <file>asks a local AI CLI (Claude Code by default, configurable) to propose an improvement to one of your tracked files, returns the proposal as a unified diff, and stores it as a pending suggestion.dfm apply <id>reviews the diff and applies it, with a fresh snapshot first, so you can roll back if the suggestion turns out to be wrong.I’m exited to try this feature out, because I’m sure there is something i"m doing wrong. The “Look at my
~/.zshrcand tell me what I could clean up” is useful feature that doesn’t require me copy and pasting or granting read or write access to my entire home directory.Where to Get It
github.com/llbbl/dotfiles-manager. Pre-built binaries for darwin and linux on arm64/amd64. Current version, as of writing, is v1.4.0.
If you’ve been meaning to actually back up your dotfiles and the friction has stopped you, this is the post where I tell you the friction is solvable.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Your AI Coding Agent Can Read Every Secret on Your Machine
Every developer running an AI coding agent has handed that agent the keys to their machine. Not metaphorically. Literally. The agent runs as your user. It can read every file you can read, execute every command you can execute, and hit every API your stored credentials authorize.
For most workflows, that’s the point. You want the agent to read your code, modify your project, ship your work. But there’s a quieter implication: the agent can also read your
.envfiles. It can invoke your secret-management tooling. It can grep forAPI_KEY=across your home directory. And nothing in the agent stack says “wait, you didn’t ask for this.”Same-UID isolation isn’t isolation. It’s the absence of isolation labeled politely.
The usual answer to “keep secrets safe from your coding agent” is: don’t store them where the agent can find them. Use a cloud secret manager. Rotate aggressively. These are good practices, and for local development, they’re often impractical. The agent is going to encounter secrets whether or not your security-best-practices doc approves.
So over the last week, I built an audit subsystem into lsm, my Local Secrets Manager. The whole thing is designed to answer one forensic question: did anything weird touch my secrets last night?
The Threat Model
A defense without a threat model is theater, so let me be specific.
The threat isn’t a sophisticated remote attacker. lsm is public, open-source code. The threat isn’t a buggy lsm either; bugs happen, and the user can read the source.
The threat is the agent layer running adjacent to lsm. Coding agents have legitimate access to a wide swath of your filesystem. They’re imperfect at intent inference. They sometimes get prompt-injected. They sometimes run in the background while you’re asleep. When an agent calls
lsm get prod DATABASE_URL, the action is indistinguishable from you doing the same thing. The audit log’s job is to make those calls retrospectively distinguishable.A secondary threat is an agent covering its tracks. If something reads a secret and then edits the audit log to erase the evidence, the log is worse than useless.
What Got Built
The audit subsystem records every access as a structured event: a sequence number, a timestamp, the action, the app and environment, an
Actorblock describing the calling process, and two cryptographic fields linking each event to the previous one.The
Actorblock was the interesting design problem. It captures parent process ID, parent process name, TTY device path (or empty if there’s no terminal), current working directory, an agent marker derived from environment variables that tools like Claude Code, Cursor, Aider, and Continue set, and the calling user ID. Every field is captured every time. Noomitempty. UID zero is a real, meaningful value, and silently dropping it would be a footgun.Events land in a hash-chained JSONL file at
~/.lsm/audit.jsonl. Each row carries the SHA-256 of the previous row plus its own body. If anyone edits, inserts, or deletes a row in the middle, the next row’sprevno longer matches andlsm audit verifysurfaces the break.The chain doesn’t catch tail truncation. If you chop off the end of the file, what’s left is internally consistent. A sidecar file storing the last expected hash is the obvious fix, and I deliberately rejected it. lsm is public code. Any local attacker who knows about the sidecar can rewrite both files in lockstep. Tail-truncation detection is deferred to the off-machine path: when events ship to a remote stack, the last hash naturally lives somewhere the local attacker doesn’t control.
Reading the Log
Three commands cover the read side.
lsm audit taildoes what you’d expect.lsm audit show <seq>prints a single event.lsm audit queryis the workhorse, with every field as a filterable dimension:--app,--env,--event,--parent-comm,--agent-marker,--tty present|absent,--since,--until. Output is JSONL when piped and columnar text when interactive.Then there’s
lsm audit suspicious, which runs four hard-coded detectors in one pass:- Outside hours. Events whose timestamps fall outside 07:00–23:00. The 3 a.m. canary.
- Burst. More than N events from a single parent process within a sliding window. The runaway-agent canary.
- New parent_comm. Process names not seen in the prior 30 days. The “what is this new thing” canary.
- Non-interactive, no agent. No TTY, no recognized agent marker. The “what is even running this” canary.
A single event can stack reasons. A 3 a.m. burst from an unknown parent is unambiguously interesting.
The detector doesn’t learn baselines, doesn’t call out to an ML model, doesn’t require a service. High-signal patterns are obvious patterns, and obvious patterns are well-served by hard-coded predicates.
Shipping Events Off the Box
If you already run an observability stack, lsm can ship audit events over OTLP (the OpenTelemetry wire protocol). Three design choices matter here.
The local file sink is always authoritative. The remote sink is a mirror, not a replacement. An lsm operation never fails because the remote endpoint is down.
Redaction is allowlist-based. App and environment names are HMAC-hashed with a per-host salt before becoming labels. The TTY device path is dropped and replaced with a
tty_present: true/falseboolean. Secret values,cwd,hash,prev, and the schema version never leave the host. Secret names are replaced withkey_present: truemarkers; the remote observer can see that a key was accessed, never which key.Events whose name starts with
audit.(chain failures, suspicious matches, sink drops) are always local. Telling a remote attacker that local integrity has been compromised is counterproductive.What’s Still Open
The most important non-feature: no command in lsm emits events yet.
setdoesn’t log.getdoesn’t log.deletedoesn’t log. The plumbing is complete, the calls are not wired in. Each emit site needs careful thought about which fields are appropriate, whether the event should be local-only, and how it interacts with sensitive operations. That’s the next chunk of work.The agent-coding era is normalizing a model where AI tools have wide-ranging access to developer machines. The premise that the agent operates as a fully-trusted local user is unlikely to change soon. Managing the risk means visibility. It means being able to answer “what touched my secrets last night” with a record the agent couldn’t silently rewrite.
The code is at github.com/llbbl/lsm. The full design lives in
docs/observability.md.I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].