DevOps

• Adding Types to JSON with Dhall

  A few months ago I wrote a post asking whether there’s something better than JSON. Two configuration languages that sit above JSON kept coming up: CUE and Dhall. Both give you the things JSON lacks when you author config by hand, and both compile down to plain JSON, YAML, or whatever your services actually read. I spent more time with CUE back then and never gave Dhall a real look. This post is me going back for that second look, because the one feature I kept wanting was a type system over my config.

  JSON is the universal language of API payloads and config files, and I don’t want that to change. But as a format for authoring configuration by hand, it’s rough:

  • No comments.
  • No variables or functions, so you copy-paste the same block ten times.
  • No type system, so "8080" and 8080 look equally valid.
  • No imports, which is how you end up with a 2,000-line monolith nobody wants to touch.

  The usual escape hatch is a templating engine like Jinja or Helm, or a real programming language like Python or TypeScript that spits out JSON. That works, but you’ve traded one problem for a scarier one: your config generator is now Turing-complete. It can crash, hang in an infinite loop, or reach out and read some local environment variable, and it’ll do it at 2 AM when the pipeline runs.

  This is where Dhall comes in.

  What is Dhall?

  The short version: Dhall is JSON plus types, plus functions, plus imports. It’s a programmable, strongly-typed configuration language.

  The part I actually care about is what it doesn’t have. Dhall is not Turing-complete. No arbitrary recursion, no side effects. Every Dhall program is guaranteed to terminate. You get the abstraction power of a functional language like Haskell or Elm, with the guarantee that it will never hang your build. That’s a different trade than “just write a Python script.”

  The problem, in JSON

  Here’s a normal config.json for a microservice:

  {
    "serviceName": "payment-api",
    "port": 8080,
    "environment": "production",
    "database": {
      "host": "db.internal.net",
      "maxConnections": 50
    }
  }
  

  Three ways can be a problem in production: someone writes "port": "8080" and the service won’t boot, someone typos "prodution" and it silently runs in debug mode, or someone forgets maxConnections entirely and you get a null blowup at runtime. Nothing catches any of it until it’s live.

  The same thing, typed

  In Dhall you define the shape up front. Enums, record types, default values:

  -- schema.dhall
  let Environment = < Local | Staging | Production >
  
  let Database =
        { Type = { host : Text, maxConnections : Natural }
        , default = { maxConnections = 20 }
        }
  
  let Config =
        { Type =
            { serviceName : Text
            , port : Natural
            , environment : Environment
            , database : Database.Type
            }
        , default =
            { port = 8080, environment = Environment.Local }
        }
  
  in  { Environment, Database, Config }
  

  Now you author against that schema, and you get defaults and composition for free:

  -- config.dhall
  let Schema = ./schema.dhall
  
  let myConfig : Schema.Config.Type =
        Schema.Config.default
        // { serviceName = "payment-api"
           , environment = Schema.Environment.Production
           , database = Schema.Database.default // { host = "db.internal.net", maxConnections = 50 }
           }
  
  in  myConfig
  

  Misspell Production, or pass "8080" as a string, and Dhall throws a type error before a single line of JSON is generated. Hopefully the benfit is now clear; adding a type safety layer to your config files.

  Compiling down to JSON

  You don’t ship Dhall to your services. You ship the JSON they already understand:

  brew install dhall-json
  dhall-to-json --file config.dhall
  

  Out comes clean, boring, standard JSON. Your services never know Dhall was involved. The part that I like is the safety lives at authoring time, and the runtime artifact stays dumb.

  Two features worth knowing about

  Hermetic imports with hash pinning. Dhall can import from a URL, so shared utilities live in one place instead of being copy-pasted across five repos. To keep someone from swapping the file out from under you, you pin the import to a SHA-256 hash of its normalized form:

  let Prelude =
        https://prelude.dhall-lang.org/v22.0.0/package.dhall
          sha256:10db4c919c25e4d262db3ed0d1d6120da3e3906673f00e3012c1d14e1963976a
  

  If the remote content changes, the hash won’t match and the build fails. The hash above is just an example, and each Prelude version has its own, so don’t copy it by hand. dhall freeze --inplace config.dhall computes the correct hashes for whatever you’ve imported and pins them automatically.

  Exhaustive matching with merge. When you map a union type to output, Dhall makes you handle every variant:

  let getLogPrefix = \(env : Environment) ->
        merge
          { Local = "[DEV] ", Staging = "[STAGE] ", Production = "[PROD] " }
          env
  

  Add a QA variant later, and every merge block that touched Environment fails to compile until you deal with it. No forgotten switch case slipping into production. The compiler keeps a running list of everything you now owe it.

  Is it worth it?

  	Raw JSON	Dhall
  Type safety	None, fails at runtime	Static, at compile time
  Comments & logic	No	Yes
  Termination	N/A	Guaranteed
  Dependency pinning	No	SHA-256
  Output	Consumed directly	Compiles to JSON/YAML/TOML

  For a two-key config file, it doesn’t make sense, but once you’re staring down Kubernetes manifests, a pile of near-identical microservice configs, or anything where a typo takes down a service, the calculus changes. You keep clean static JSON as the thing your services actually read, and you move all the ways-to-get-it-wrong to a place where a compiler catches them first.

  ---

  Sources

  • Dhall Language Tutorial & Cheatsheet: records, union types, default overrides, dhall-to-json, and dhall freeze.
  • Dhall language standard on GitHub: the non-Turing-complete design and the semantic integrity hash spec.

  ---

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  Jul 11, 2026 DevOps Programming Json Configuration Dhall

  Enjoyed this?

• Running Mem0's Memory Backend as Real Infrastructure with Ansible

  Mem0 works best when you treat its memory backend as infrastructure, not as some throwaway process you start by hand and forget about. There are two pieces worth automating early: the vector store and the graph store. Qdrant holds the embeddings for semantic lookup. Neo4j holds the relationship-oriented graph memory.

  Quick note before we go further, because this can be confusing. When folks talk about Mem0’s “graph” support, that is not GraphQL. GraphQL is an API query layer you’d put in front of your app. Mem0’s graph memory is an actual graph database, usually Neo4j, that the client talks to over Bolt. Different thing entirely.

  This post walks through a practical Ansible shape for running those backing services with Docker Compose. I’m not going to hand you a complete role. The point is to show the decisions that make the setup repeatable, because those are the parts people usually get wrong.

  What the deployment actually does

  Three responsibilities, that’s it:

  1. Get the Mem0 compose project onto the server.
  2. Write a server-local .env file with the runtime config.
  3. Start the Compose stack with Qdrant and Neo4j.

  In a small setup, the Python client runs on your laptop while the services run on a LAN host or a small VPS. The server exposes these ports, but only to trusted networks:

  • Qdrant HTTP: 6333
  • Qdrant gRPC: 6334
  • Neo4j browser: 7474
  • Neo4j Bolt: 7687

  If you’re on a public internet host, bind these to localhost or hide them behind a VPN, firewall, or private network. Do not casually publish database ports to the internet. I shouldn’t have to say remind you…

  Role inputs

  A generalized role only needs a handful of variables:

  mem0_home: /opt/mem0
  mem0_repo: [email protected]:your-org/your-mem0-project.git
  mem0_branch: main
  
  mem0_qdrant_collection: default
  mem0_neo4j_enabled: true
  
  mem0_embedder_base_url: https://api.example.com/v1
  mem0_embedder_model: your-embedding-model
  mem0_embedder_dimensions: 1024
  

  The collection name matters more than it looks. Qdrant collections have a fixed vector size. If you switch embedding models and the dimensions change, create a new collection. Don’t try to reuse the old one. This is a common way people break a working Mem0 setup, so treat the collection name as part of the embedding config, not an afterthought.

  Secrets

  Keep API keys and database passwords out of your regular vars files. Ansible Vault, a secrets manager, whatever your deployment system gives you. Then template a .env file with tight permissions:

  - name: Write Mem0 environment file
    ansible.builtin.template:
      src: mem0.env.j2
      dest: "{{ mem0_home }}/.env"
      mode: "0600"
    no_log: true
  

  That no_log: true is not optional. Template diffs will happily leak plaintext API keys and graph passwords into CI logs, terminal scrollback, and ticket attachments. Once a secret lands in a CI log, you’re rotating it, not deleting it.

  The template itself stays small:

  EMBEDDER_API_KEY={{ vault_embedder_api_key }}
  EMBEDDER_MODEL={{ mem0_embedder_model }}
  EMBEDDER_DIMENSIONS={{ mem0_embedder_dimensions }}
  
  QDRANT_HOST=qdrant
  QDRANT_PORT=6333
  QDRANT_COLLECTION={{ mem0_qdrant_collection }}
  
  NEO4J_URL=bolt://neo4j:7687
  NEO4J_USERNAME=neo4j
  NEO4J_PASSWORD={{ vault_neo4j_password }}
  

  Notice QDRANT_HOST=qdrant, not an IP. When the Mem0 container talks to sibling Compose services, use the service names. Save hostnames and LAN DNS for clients that live outside the Compose network. More on that in a second, because it’s a real gotcha.

  Compose shape

  Just what you need to run Mem0’s memory backend in Docker.

  services:
    qdrant:
      image: qdrant/qdrant:v1.12.6
      ports:
        - "6333:6333"
        - "6334:6334"
      volumes:
        - qdrant_storage:/qdrant/storage
  
    neo4j:
      image: neo4j:5.26
      environment:
        NEO4J_AUTH: "neo4j/${NEO4J_PASSWORD}"
      ports:
        - "7474:7474"
        - "7687:7687"
      volumes:
        - neo4j_data:/data
        - neo4j_logs:/logs
  
  volumes:
    qdrant_storage:
    neo4j_data:
    neo4j_logs:
  

  Pin your image versions. Floating tags make it impossible to tell whether a later failure came from your playbook, your app, or an upstream image that changed under you at 2am. Pinning turns a mystery into a diff.

  The two Ansible flags that matter

  Most of the role can stay boring. Ensure the directory exists, clone the repo, template the .env, start the stack with community.docker.docker_compose_v2. The two details I want you to actually read are force: false on the git task and remove_orphans: true on the compose task.

  force: false protects local edits in the checkout. If the role needs to patch a generated file, make that explicit and safe instead of letting Git clobber the tree.

  remove_orphans: true keeps Compose honest. Say you rip out an old local embedding service and switch to a managed embedding API. Without this, the old container just keeps running forever, quietly, and you’ll swear the new config isn’t taking effect.

  And for restarts, use handlers. Notify a restart handler when the checkout changes or when .env changes. Don’t bounce the stack on every single playbook run. The steady-state run should be quiet.

  Service names vs. hostnames

  From a client running outside the Compose network, point Mem0 at the server’s reachable hostname:

  QDRANT_HOST=mem0.example.test
  NEO4J_URL=bolt://mem0.example.test:7687
  

  From a container inside the same Compose project, use service names:

  QDRANT_HOST=qdrant
  NEO4J_URL=bolt://neo4j:7687
  

  Get this backwards and you burn an afternoon on it…

  Check your work

  After the playbook runs, verify the pieces independently:

  docker compose ps
  curl http://localhost:6333/healthz
  docker compose logs --tail=100 neo4j
  

  For Neo4j, actually test Bolt from the network where your client lives. The browser port on 7474 being reachable does not prove the Bolt endpoint on 7687 is usable. Different port, different assumption, don’t confuse a green browser page for a working client.

  The whole pattern

  Strip away the YAML and it’s simple. Docker Compose owns Qdrant and Neo4j. Ansible owns the checkout, the .env, and the Compose lifecycle. Vault owns the credentials. The client picks service names or external hostnames depending on where it runs.

  That gets you a repeatable Mem0 backend without turning your Ansible role into a second copy of the entire app. Pin your images, guard your secrets with no_log, and never reuse a collection after the vector size changes. Do those three and the rest is boring, which is exactly what you want from infrastructure.

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  Jul 9, 2026 DevOps docker Ansible Mem0 Qdrant Neo4j

  Enjoyed this?

• Geni vs. Goose for lightweight database migrations

  I have a pretty simple rule for database migrations: the tool lets me write SQL, check status, apply the next change, and roll back when I need to. I don’t want much more standing in the way.

  That’s why I like both Geni and Goose.

  Neither one is trying to be a full application framework. They don’t require the rest of the app to be written in the same language as the migration tool. They’re small enough to understand, script, and run from CI, but they have enough features that they don’t feel like a pile of shell scripts once a project grows real production databases.

  The catch is that they’re good at slightly different jobs. I’ve ended up using both in the same migration repo: Goose for PostgreSQL, MySQL, and MariaDB, Geni for LibSQL/Turso. That split has worked well enough that I’d reach for the same shape again, even on projects that aren’t written in Go or Rust.

  The short version

  Use Goose when you want a mature, widely used SQL migration tool for the traditional server databases: PostgreSQL, MySQL, MariaDB, and local SQLite. Broad database matrix, familiar single-file format, status and rollback commands, environment-variable substitution, Go migrations when you need them, and a very plain CLI.

  Use Geni when LibSQL or Turso is part of the system. It’s a standalone tool with first-class LibSQL/Turso support, a simple paired-file model, status/up/down/new commands, and optional schema snapshots. It was built for the space where “SQLite-like, but remote and edge-hosted” starts to expose the rough edges of tools designed around local SQLite files.

  Reach for something heavier when migrations stop being ordered SQL files and turn into a schema-management problem. More on that at the end.

  Why I like both

  The best thing about Geni and Goose is that the migration stays close to the database.

  There’s no ceremony around a model layer. You write SQL. You put it in version control. You run a CLI. The database gets a tracking table. A teammate can review the migration in a pull request without learning your ORM. A deploy script can ask for status before it applies anything. For a lot of small and medium projects, that’s exactly the right amount of machinery.

  It also makes both tools language-independent in practice. Goose is written in Go, Geni in Rust, but a TypeScript, Python, Ruby, or PHP app can use either one just fine. The migration tool runs at the edge of the application. It talks to the database and the filesystem. It doesn’t need to be imported by the app.

  That separation is underrated. A migration directory can outlive a framework rewrite. It can be shared by several services, live in an infrastructure repo, and get run by CI, a deploy script, or a human with a terminal.

  Where Goose shines

  Goose is the boring choice, in the best sense. Its migration format is easy to read:

  -- +goose Up
  -- +goose StatementBegin
  CREATE TABLE users (
    id BIGSERIAL PRIMARY KEY,
    email TEXT NOT NULL UNIQUE
  );
  -- +goose StatementEnd
  
  -- +goose Down
  -- +goose StatementBegin
  DROP TABLE users;
  -- +goose StatementEnd
  

  The forward and rollback changes live in one file, and a review comment can talk about both in one place.

  Goose is also strong when a repo has several conventional engines. It documents support for PostgreSQL, MySQL, MariaDB, SQLite, and more, and the command shape stays mostly the same across them:

  goose -dir migrations/postgres/app/schema postgres "$POSTGRES_APP_DSN" up
  goose -dir migrations/mysql/app/schema mysql "$MYSQL_APP_DSN" up
  

  For Go services there’s another useful escape hatch: Go migrations. Most schema changes should stay in SQL, but occasionally one needs batching, custom validation, or a data rewrite that’s clearer in code. Goose supports that without forcing every migration to become code.

  I reach for Goose first on Postgres/MySQL/MariaDB, on teams that already know SQL migrations, and in deploy scripts that just want status, up, down, and version.

  Where Geni shines

  Geni has a similar direct feel, but its file model is different:

  migrations/
    1782054288_create_users_table.up.sql
    1782054288_create_users_table.down.sql
  

  Forward migration in one file, rollback in another, both plain SQL. Geni can create the pair for you:

  geni new create_users_table
  geni status
  geni up
  

  The more important point is LibSQL/Turso. Turso is not just “a local SQLite file with a different name.” LibSQL is SQLite-compatible, but a remote Turso database brings its own connection shape, auth token, protocol, and operational behavior. That’s exactly where a migration tool either feels native or feels like you’re pushing it through a small opening.

  Geni expects the Turso shape. The documented flow uses DATABASE_URL and DATABASE_TOKEN, it speaks LibSQL, and it can dump a schema.sql snapshot after migrations, which is handy as a review artifact and source-control reference.

  On my own Turso projects I like a thin wrapper around Geni. It resolves secrets from whatever store the project uses, splits a composed DSN into DATABASE_URL and DATABASE_TOKEN, points Geni at the right per-database folder, and masks tokens in debug output. That keeps the human command simple:

  mise run geni dev app status
  mise run geni prod app up
  

  One folder per logical database, paired up/down files, secrets outside the repo, a status check before deploy, and a confirmation prompt before production up or down.

  About Goose and Turso

  This is the one place the recommendation needs nuance. Goose has been growing, and its current docs list a Turso driver. So the answer is no longer “Goose can’t talk to Turso.”

  But “there’s a driver” and “this is the smoothest operational path for my Turso migrations” are not the same claim. The friction I care about isn’t just opening a connection. It’s the whole lifecycle: creating the tracking table, applying safely, transaction behavior, the right remote protocol, auth, status, and a workflow I trust in a short-lived feature branch and in production. For that job I still prefer Geni for LibSQL/Turso, and Goose stays my default for Postgres/MySQL/MariaDB.

  When to use something heavier

  Lightweight SQL migration tools aren’t always enough. I’d start looking at Atlas, Liquibase, Flyway, Sqitch, or an ORM-native system when the problem changes shape:

  • You need schema drift detection across many environments.
  • You want to declare desired schema and have the tool compute the migration.
  • You need online migration planning for large tables.
  • You operate hundreds or thousands of tenant databases.
  • You need policy checks, approval workflows, or compliance artifacts before a migration can merge.

  Those are real needs. They’re also real costs. The heavier tool might be worth it, but I wouldn’t start there by default.

  For most projects, the better starting point is still: write a small SQL migration, review it in git, run status, apply it, and keep the history boring.

  My rule of thumb

  PostgreSQL, MySQL, or MariaDB, I reach for Goose. LibSQL/Turso, I reach for Geni. Not written in Go or Rust? I still consider both, because the CLI boundary matters more than the implementation language. Migrations are an operational concern before they’re an application concern.

  And if the story starts needing drift management, online planning, policy gates, or fleet-wide orchestration, I stop trying to stretch a lightweight tool and pick something heavier on purpose. For everything else, Geni and Goose land right in the sweet spot: small, SQL-first, scriptable, and capable enough to keep schema changes moving without turning migrations into a platform.

  Sources and further reading

  • Geni GitHub repository: https://github.com/emilpriver/geni
  • Turso: “Database migrations with Geni and libSQL”: https://turso.tech/blog/database-migrations-with-geni
  • Goose documentation: https://pressly.github.io/goose/
  • Goose GitHub repository: https://github.com/pressly/goose
  • Atlas guide on Turso connection URLs: https://atlasgo.io/guides/sqlite/turso

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  Jul 8, 2026 DevOps Database migrations Geni Goose Turso Libsql Sql

  Enjoyed this?

• Day 15: The Man Who Synchronized the World

  David Mills, the father of internet time, wrote the protocol that synchronizes every computer on Earth. He did it as a professor at the University of Delaware, on a project he started in the early 1980s and never stopped working on.

  The code lives in your laptop, your phone, your router, every cloud server you’ve ever touched, and every satellite in low Earth orbit. The protocol it implements is called NTP. The reason your computer’s clock is correct, right now, within a few milliseconds of UTC, is that Mills spent forty years of his life making sure it would be.

  He once described the early ARPANET days as a “sandbox” where researchers were simply told to “do good deeds.” Part of the allure of the time-synchronization work, he told The New Yorker in 2022, was that he was just about the only one doing it. He had his own “little fief.”

  For forty years, that is exactly what it was.

  The problem

  The early internet had a clock problem. As soon as there were enough machines on the network that “what time is it?” didn’t have a single answer, somebody was going to have to write a protocol. Each computer had its own oscillator. Each oscillator drifted at its own rate. Two machines that agreed at noon could be tens of seconds apart by midnight.

  Why did this matter? For most things, it didn’t. For some things, it mattered a lot. A file saved on one machine and copied to another could look older than the version it overwrote, confusing every backup tool that assumed time moves forward. Cryptographic handshakes that expire after a few seconds could fail because the two ends disagreed on what “a few seconds ago” meant. Database replicas could apply writes in the wrong order and corrupt their own state. Email between two servers could arrive timestamped before it was sent. Debugging a multi-machine bug meant correlating log entries across clocks that didn’t agree about which event came first.

  Mills decided the actual problem was that there was no protocol for negotiating the truth (in this case, time) across multiple systems. The clock on his desk was wrong. Every other clock was also wrong. The question wasn’t “who has the right time?”, it was “given that nobody has the right time and the network adds an unknown delay to every measurement, how does the system converge on a consensus that is closer to UTC than any individual node could achieve alone?”

  His first NTP RFC, RFC 958, was published in September 1985. We now call that protocol NTPv0, or the prototype. In it, Mills nailed down the four-timestamp packet format and the offset/delay math that has been in every revision since. The packet format and the core algorithm haven’t meaningfully changed in forty years. That kind of staying power is rare in any field. In internet infrastructure, where the half-life of a protocol can be measured in single-digit years, it is quite commendable.

  The four timestamps

  NTP’s core insight is that the network delay between client and server can be measured, not just guessed, as long as both sides record their own timestamps for both legs of the conversation. Four timestamps are exchanged in a single round trip:

  Client                            Server
  ──────                            ──────
  T₁ ──── request ───────────────►
                                    T₂
                                    T₃
  T₄ ◄────────────── response ─────
  

  • T₁ — the client sends the request (client clock)
  • T₂ — the server receives it (server clock)
  • T₃ — the server sends the response (server clock)
  • T₄ — the client receives it (client clock)

  Now the client has four numbers. T₁ and T₄ are in the client’s reference frame, T₂ and T₃ are in the server’s. From those four numbers, two things fall out: the round-trip delay (how long the conversation took, minus the time the server spent thinking) and the clock offset (how far the client’s clock is from the server’s). The client now knows how wrong it is, and by how much.

  The math depends on one critical assumption: the network is symmetric. The packet takes the same time to travel in both directions.

  If you’ve been following along in the series, you know there are a lot of ways to measure time. Atomic clocks. GPS receivers. The quartz crystal in your laptop. Radio signals broadcast from government antennas. They don’t all tick at the same rate, and they don’t all agree on what the current time is. How does NTP reconcile across that much varity in time sources?

  The stratum hierarchy

  NTP organizes the world’s clocks into a tree, with depth measured in strata.

  Stratum 0 is the reference. Cesium atomic clocks. Hydrogen masers. GPS receivers. Radio receivers tuned to WWV, DCF77, or MSF. These are not on the network, they’re physical devices wired directly to a small number of computers via PPS pulses on serial ports.

  Stratum 1 is the small group of servers wired directly to Stratum 0. There are perhaps a few thousand of these globally. NIST runs some. Major universities run some. The big internet exchanges run some.

  Stratum 2 servers sync with Stratum 1, Stratum 3 with Stratum 2, and so on down to Stratum 15. Stratum 16 means “unsynchronized, do not trust.”

  A typical Linux laptop syncs against Stratum 2 or 3 servers. A typical cloud VM syncs against its provider’s internal Stratum 1 fleet. Your phone syncs against whatever its carrier provides. The whole tree is held together by NTP itself, recursively.

  The genius of the design is that there is no central authority. Mills did not own the protocol. There is no “official NTP server.” Anyone can run a Stratum 1 with the right hardware, and anyone can run a Stratum 2+ by syncing with a few Stratum 1s of their choice. The largest public pool, pool.ntp.org, is a volunteer effort started in 2003 by Adrian von Bidder. It currently aggregates a few thousand donated stratum-2 servers worldwide and serves several billion requests per day. Nobody is in charge of it. It just works.

  The slew, not the step

  There are three different times to keep track of on every synced computer. The reference time is what UTC says, the truth NTP is chasing. The tick rate is how fast the computer’s oscillator pulses. It’s supposed to produce one second of clock time per real second, but always drifts a little. The system clock is what gets reported when an application asks for the current time. Synchronizing means closing the gap between the system clock and the reference time without breaking anything that depends on the system clock being well-behaved.

  NTP’s primary tool for that is the slew: it adjusts the tick rate, making each tick slightly longer or shorter than nominal, so the system clock drifts into alignment on its own. The alternative would be to jump the clock forward or backward by the full offset (a step), which is fast but can produce duplicate keys in a database, expire valid TLS sessions, or cause a logging system to mis-order events.

  Mills designed ntpd to slew conservatively. A 200ms gap might take several minutes to close, and corrections larger than about 128ms would get stepped because slewing them gradually was prohibitively slow. That trade-off worked for the always-on Unix workstations of the 1980s and 90s. It works less well for the modern reality of laptops that suspend for hours and resume with a clock that hasn’t been touched since last Tuesday, or cloud VMs that get migrated between hosts. Modern variants like chrony slew more aggressively for exactly that reason. When you open your laptop lid, you want the clock right now, not after fifteen minutes of imperceptible easing.

  The legacy

  In a sense, NTP is the thing that made the modern internet possible.

  Without well-synchronized clocks, you cannot have SSL certs. The browser needs to know when the cert expires, and if its clock is off by more than a few minutes, the encryption breaks. The same goes for databases. No matter the type, NoSQL or otherwise, they all depend on a clock to record when an operation took place.

  Without NTP, cell towers wouldn’t agree on when to hand off a call. Financial transactions wouldn’t be enforceable. And all those log files you’ll totally read one day wouldn’t make any sense. NTP is foundational to all of it. It runs as a daemon on every machine, the ones you stare at all day, the ones you don’t see, and the ones you don’t care about.

  We remember Mills as the internet’s “Father Time” and the man who synchronized the world. Neither is a metaphor.

  ---

  Sources

  • In Memoriam: David Mills (UDaily, March 2024) — University of Delaware’s obituary; biographical detail, career timeline.
  • David L. Mills — Wikipedia — congenital glaucoma from birth, vision worsening from ~2012, fully blind by 2022; UDel professor 1986–2008.
  • David Mills, the internet’s Father Time, dies at 85 — The Register — death date (Jan 17, 2024), age 85.
  • RFC 958 — Network Time Protocol (September 1985) — the original NTPv0 specification.
  • Network Time Protocol — Wikipedia — version lineage: RFC 958 (v0, 1985), RFC 1059 (v1, 1988), RFC 1119 (v2, 1989), RFC 1305 (v3, 1992), RFC 5905 (v4, 2010), RFC 8915 (NTS, 2020).
  • NTP pool — Wikipedia — Adrian von Bidder started the pool in January 2003; Ask Bjørn Hansen has run it since 2005.
  • MiFID II RTS 25 clock synchronization (Meinberg) — 100µs requirement for high-frequency trading at sub-1ms gateway latency.
  • A Brief History of NTP Time: Confessions of an Internet Timekeeper (Mills, PDF) — Mills’ own history of NTP.
  • The Thorny Problem of Keeping the Internet’s Time (The New Yorker, September 2022) — Nate Hopper’s profile of David Mills and the fragile state of NTP maintenance.

  ---

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  ---

  Tomorrow: ISO 8601, the format wars, the carnage of MM/DD vs DD/MM, and why 2026-06-07T14:30:00Z won.

  Jun 7, 2026 DevOps Programming 30daysoftime Internet history Protocols

  Enjoyed this?

• Running Terraform in Your Existing CI Pipeline

  The previous post made the case that HCP Terraform’s per-resource pricing model has gotten structurally hostile to modern infrastructure patterns. (The earlier posts in this series argued that OpenTofu is the no-regrets default for new infrastructure, and walked through when to skip Terraform entirely in favor of cloud-native tooling.) The natural follow-up: if you don’t want to pay the commercial orchestration tax, can you run Terraform or OpenTofu properly inside your existing CI/CD? The answer is yes, but the gap between “it works” and “it works well” requires some deliberate architecture. This post is about how to close that gap.

  There are three pieces: where the state lives, how the pipeline authenticates to your cloud, and what handles the orchestration concerns (locking, PR commentary, drift detection) that TACOs sell as their core value. Each one has a sensible 2026 answer that doesn’t involve paying anyone.

  State Management in GitLab

  If you’re on GitLab, the entire state-management problem is solved natively. GitLab ships an HTTP backend for Terraform and OpenTofu state on every tier including Free. You don’t need to provision an S3 bucket. You don’t need a DynamoDB lock table. You don’t need to figure out KMS. The state file is encrypted in transit and at rest, locking is handled by GitLab’s project-scoped role-based access control, and there’s a native UI under Operate > Terraform states that shows you version history and lets you roll back if something corrupts.

  The pipeline pattern uses a backend block like this:

  terraform {
    backend "http" {}
  }
  

  Combined with the gitlab-tofu (or gitlab-terraform) CLI wrapper in your .gitlab-ci.yml, which dynamically configures the HTTP backend at runtime using the per-job ${CI_JOB_TOKEN}. The wrapper avoids passing backend credentials via -backend-config arguments (which cache in pipeline logs) and handles authentication automatically.

  The RBAC story is also worth pointing out, because it’s exactly what TACOs charge thousands of dollars to replicate: the GitLab project’s role model becomes the IaC permissions model. Developers can read state and run tofu plan -lock=false. Maintainers and Owners can lock state and run tofu apply. The audit log is the GitLab activity feed. No additional configuration, no additional vendor.

  For GitLab shops, this is the single highest-leverage decision in the entire IaC stack: stop paying for state management when your VCS gives it to you for free.

  Secretless Authentication on GitHub Actions

  On GitHub Actions, the equivalent problem is authentication. Historically, every Terraform-on-Actions tutorial told you to put a long-lived AWS access key in GitHub Secrets. That’s the worst possible pattern. A compromised repository, a malicious third-party action, or a leaked log line gives the attacker permanent, unscoped access to your cloud.

  The 2026 answer is OpenID Connect with cloud-side trust policies. The pipeline gets ephemeral, short-lived credentials per job, scoped to the specific repository and branch that initiated the run. Nothing persists.

  For AWS: configure GitHub’s OIDC provider (token.actions.githubusercontent.com) as an identity provider in IAM. Create an IAM role with a trust policy that conditionally allows assumption based on JWT claims like sub (subject) and aud (audience). The workflow uses aws-actions/configure-aws-credentials to exchange a GitHub-issued JWT for temporary AWS credentials via AssumeRoleWithWebIdentity. The trust policy can be scoped to a specific repository, a specific branch (main only), or even a specific environment (production).

  For GCP: the equivalent is Workload Identity Federation. You create a Workload Identity Pool that trusts GitHub’s OIDC provider, configure attribute mapping that validates the token claims (e.g., requiring assertion.repository == "company/infra-prod"), and grant the pool’s principal the ability to impersonate a specific GCP service account. The official google-github-actions/auth action handles the token exchange.

  Both patterns produce credentials that expire when the job ends, can’t be exfiltrated to long-term storage, and leave a clean audit trail in your cloud’s IAM logs. There is no good reason to use long-lived cloud credentials in CI in 2026.

  What CI Doesn’t Give You for Free

  Native CI/CD solves the cost problem. It does not, by itself, solve every operational problem that commercial TACOs address. There are three real gaps worth knowing about:

  State locking and race conditions. Standard CI/CD systems are designed for concurrent runs because that’s what application code wants. Infrastructure code wants the opposite. If two PRs merge at the same time and both trigger tofu apply, you have two concurrent processes racing to mutate the same state file. With GitLab’s HTTP backend or an external lock backend like DynamoDB, the lock will prevent corruption but one job will fail with a confusing error. Without it, you get state corruption. You need some queuing logic, either custom or via an orchestrator.

  PR plan commentary. TACOs post the output of terraform plan directly into the PR so reviewers can see what’s about to change before merging. In raw CI/CD this requires a third-party action (terraform-plan-pr-commenter and similar), parsing of the CLI output, handling of PR comment character limits, and securely passing the binary plan file as a workflow artifact from the plan stage to the apply stage. None of this is hard, but it’s a real chunk of YAML you have to maintain.

  Cost estimation. TACOs include built-in cost estimation on every plan. Adding this to your own pipeline means picking up a third-party FinOps or IaC cost-analysis tool (there are several worth comparing), running it against your plan output, parsing the JSON, comparing against budget thresholds, and posting deltas into PRs. None of that is hard, but it’s another bit of integration to own.

  You can build all of this yourself. Plenty of teams do. The question is whether maintaining the bash and YAML is cheaper than using an open-source orchestrator designed for exactly this problem.

  Open-Source Tools to Layer In

  None of these are drop-in replacements for HCP Terraform or Spacelift. They solve specific problems CI/CD doesn’t handle on its own, and you compose them based on what’s actually missing from your setup.

  Tool	What It Solves	Best For
  Atlantis	PR-based workflow automation, plan/apply via PR comments, PR-level locking	Teams that want TACO-style PR workflow but on their own server
  Digger	Same PR workflow + locking, but the IaC actually runs inside your existing CI runners	Teams with secretless OIDC pipelines who don’t want to maintain a separate server
  Terramate	Multi-stack monorepo orchestration, git-based change detection, parallel execution	Teams whose Terraform has grown into hundreds of stacks

  Atlantis is the original PR-automation tool, accepted into the CNCF Sandbox in June 2024. It deploys as a Golang binary or container, listens for VCS webhooks, and runs Terraform on its own server. The architecture is showing its age. It’s stateful, single-threaded, granting it persistent privileged cloud access creates a high-value target, and the maintenance velocity has slowed. If you’re already running it and it works, fine. For new setups, the case for Digger is usually stronger.

  Digger is a thinner orchestration layer. It coordinates Terraform jobs but runs them inside your existing GitHub Actions or GitLab runners, using OIDC for cloud authentication. The orchestrator backend itself never sees state, plan output, or cloud credentials. This is the right pattern if you’ve already built secretless OIDC pipelines and want PR-workflow automation without introducing another long-lived privileged component.

  Terramate solves a different problem: scaling Terraform across many stacks in a monorepo. It parses your Git history to determine which stacks changed, then runs plan and apply only on those, in parallel. For a repo with 200 stacks and a PR that touches one, you skip the 199 unnecessary plans. It also has a code-generation system that reduces HCL boilerplate. Terramate Cloud adds dashboards and drift detection without requiring access to cloud credentials. If your IaC repo has gotten unwieldy, Terramate is the tool for it. It’s a complement to Atlantis or Digger, not a substitute.

  The Recommendation

  The full picture for escaping commercial TACOs in 2026:

  • State: GitLab’s native HTTP backend if you’re on GitLab. S3 + DynamoDB (or OpenTofu state encryption + S3) if you’re on GitHub.
  • Auth: OIDC for AWS, Workload Identity Federation for GCP. Never long-lived secrets.
  • PR workflow: Digger if you want PR automation that runs inside your existing CI. Atlantis if you’re already running it. Skip this layer entirely if your team is small enough that PRs serialize naturally.
  • Stack management: Terramate if you have a large monorepo. Otherwise, not needed.
  • Cost estimation: Pick a third-party FinOps or IaC cost-analysis tool and wire it into your plan stage.

  The total monetary cost of this stack is the price of your existing CI/CD minutes, which you’re already paying. The total time cost is on the order of one to two weeks of platform-engineering time to set up properly, plus ongoing maintenance proportional to how much you customize.

  For most organizations under 300 engineers, that’s cheaper than HCP Terraform Standard or Premium. For larger organizations, the calculus depends on how much custom platform work you’re willing to absorb versus how much you want a vendor to handle.

  This wraps the series. Four posts in: OpenTofu as the no-regrets default engine, the scenarios where cloud-native tools beat Terraform entirely, the HCP pricing model that’s pushing teams to find alternatives, and now the CI-native path that lets you skip commercial orchestration. The throughline is the same as every post in this blog about platform engineering: there isn’t a single open-source tool that drops in for HCP Terraform or Spacelift. You’re assembling a stack from focused pieces (state backend + auth + maybe PR automation + maybe stack management), accepting some operational tax in exchange for not paying the SaaS premium. For most teams under 300 engineers, that tradeoff is worth it.

  Sources

  • GitLab-managed Terraform/OpenTofu state — GitLab Docs
  • How to Manage Terraform State with GitLab — Spacelift
  • Using Terraform to connect GitHub Actions and AWS with OIDC — Thiago Salvatore
  • Deploy Terraform resources to AWS using GitHub Actions via OIDC
  • Configure Workload Identity Federation with deployment pipelines — GCP Docs
  • Terraform Deployment to GCP Using GitHub Actions and Workload Identity Federation
  • Atlantis vs. Terraform Cloud / Terraform Enterprise — Spacelift
  • Digger and Atlantis: key differences
  • Terramate: Turn Your IaC into a Lightning-Fast Platform
  • How to Implement Cost Checks in Terraform CI/CD Pipelines — OneUptime
  • Terraform Plan PR Commenter (GitHub Action)

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 23, 2026 DevOps Github Infrastructure Opentofu Terraform Cicd Gitlab

  Enjoyed this?

• HCP Terraform's Per-Resource Pricing Is a Trap

  The first post in this series argued OpenTofu is the no-regrets default for new infrastructure. The previous post mapped out when to skip cloud-agnostic IaC entirely. This one is about what happens to organizations that picked Terraform years ago, built their orchestration around HCP Terraform (formerly Terraform Cloud), and are now opening renewal quotes that have doubled or tripled year-over-year.

  The short version: HashiCorp’s 2024 pivot to Resource Under Management (RUM) billing penalizes the architectural patterns the DevOps community spent a decade adopting. Modular code, ephemeral environments, and granular resources are all things you were supposed to do with Terraform. They now cost real money under the new pricing model. And the legacy free tier that grandfathered teams into a more sustainable cost structure hit end-of-life on March 31, 2026.

  If you’re still on HCP Terraform in 2026, you need to understand the math.

  How the New Pricing Works

  The 2024 RUM model bills based on the peak number of resources tracked in your terraform.tfstate files, measured hourly. The Free tier covers up to 500 resources with a single concurrent run. Above that, you’re on Pay-As-You-Go tiers:

  Tier	Per-resource cost	Concurrency	What you get
  Free	$0 (first 500)	1	Basic VCS, remote state
  Essentials	~$0.10/month	1	Basic provisioning, no SSO
  Standard	~$0.47/month	3	Up to 5 policy checks, cost estimation, limited RBAC
  Premium	~$0.99/month	10	Full governance, unlimited policies, SSO, audit logs

  On paper, $0.47 per resource per month looks negligible. The math goes sideways quickly because of three things.

  Why “Resources” Is a Footgun

  1. Granularity inflation. A single logical Terraform module produces dozens of underlying resources. An AWS VPC module isn’t one billable resource. It’s the VPC plus every subnet, every route table, every route table association, every IAM policy attachment, every security group rule, every DNS record. A widely-shared Reddit post by user notoriousbpg describes a team whose HCP Terraform bill was about to jump from $0 to over $15,000 a year, because 80% of the resources under management were GraphQL operation mappings to data sources, while the actual AWS infrastructure they cared about cost only $8,000. They were paying more for orchestration than for the infrastructure being orchestrated.

  2. Idle workspaces. RUM billing doesn’t distinguish between active and inactive infrastructure. The proof-of-concept workspace someone spun up last quarter and never destroyed is still on your bill. The staging environment that was deprecated in favor of ephemeral environments is still on your bill. Industry telemetry suggests 30–40% of an average organization’s RUM cost is for completely idle infrastructure nobody has bothered to terraform destroy.

  3. Hourly peak billing on ephemeral resources. HCP Terraform bills based on peak hourly resource count. If your integration test pipeline spins up infrastructure that exists for five minutes and is then torn down, you’re billed as if it existed for the full hour. This is a direct tax on the modern GitOps workflow patterns Terraform itself spent years promoting. The more ephemeral environments you use, the more punitive the billing becomes.

  The compounding effect is severe. Another account describes cloning a 600-resource production workspace to create a pre-production environment. The resource count doubles to 1,200. The annual cost goes from ~$122 to ~$858, a 7x increase for what’s architecturally a trivial change. Multiply that across every environment, every test fixture, every modular abstraction, and the renewal quote stops being theoretical.

  The Alternative TACOs

  Once organizations work through the RUM math and realize the bill is structurally unsustainable, the obvious move is to look at alternative orchestration platforms. The three serious contenders, with very different pricing models:

  Platform	Pricing Model	Entry / Mid-Tier Cost	What It Does Differently
  Spacelift	Resources + runs + seats	$1,500–$3,500/mo	Multi-tool (Pulumi, K8s manifests, Terragrunt). OPA policies. Custom runners. Cross-stack dependencies.
  env0	Per-user	~$50/user/mo	Predictable user-based pricing. Strong TTL/ephemeral environment story.
  Scalr	Per-user	~$50/user/mo	Familiar Terraform Cloud UI replacement. Lower entry price than Spacelift.
  ControlMonkey	Fixed plan (users + assets)	$800/mo (Startup: 10 users, 5k assets, 500 deploys)	One-click Terraform import, automatic drift remediation, daily cloud-config backups, built-in compliance.

  Spacelift is the choice for complex platform engineering teams. It supports Terraform, OpenTofu, Pulumi, Terragrunt, and Kubernetes manifests in one platform, handles cross-stack dependencies, and bakes OPA policy enforcement into the runtime. The catch is that its pricing still factors in managed resources, so the bill scales with infrastructure size, just less aggressively than HCP.

  env0 and Scalr both flipped to user-based pricing specifically as a response to RUM. A 15-engineer team managing 3,000 resources pays roughly the same on env0 as a 15-engineer team managing 500. The price is bounded by headcount, not infrastructure complexity. This is the right model for teams whose resource counts have ballooned because they followed the “do everything as code” advice and now have hundreds of granular Terraform-managed entities they don’t want to pay per-unit fees on.

  When to Pay for Any Commercial TACO

  The harder question is whether the commercial orchestration layer is worth its multi-thousand-dollar monthly bill at all. The features TACOs sell (state locking, PR-level plan output, policy enforcement, drift detection, audit logging) are all things you can build into your own CI/CD pipeline. The question is whether building and maintaining that pipeline is cheaper than paying the SaaS fee.

  For most teams under ~50 engineers, the answer is no. The SaaS fee buys polish and convenience, but the underlying capabilities are available in GitLab’s native state management or in GitHub Actions with the right open-source orchestrator. For larger teams, the calculus shifts: the cost of a dedicated platform engineer maintaining a custom CI/CD pipeline starts to approach the cost of a commercial license, and the operational predictability of a managed platform becomes valuable.

  But the days of HCP Terraform being the obvious default for everyone above the free tier are over. The RUM model made the math too punishing for too many real-world architectures.

  The next and final post in this series gets into the actual mechanics of running Terraform/OpenTofu inside your existing CI/CD: GitLab’s native state backend, GitHub Actions with OIDC/Workload Identity Federation for secretless deploys, and the open-source orchestrators (Atlantis, Digger, Terramate) that close the gap between raw YAML and a real platform.

  Sources

  • Terraform Cloud / Enterprise Pricing — Tiers Overview 2026 — Spacelift
  • Terraform Cloud Pricing Guide: Tiers, Costs, and Optimization Tips — ControlMonkey
  • 10 Best Terraform Cloud Alternatives & Competitors In 2026 — ControlMonkey
  • Continuing HCP Terraform’s enhanced free tier experience — HashiCorp
  • Terraform Cloud Pricing Explained: Resource-Based Guide (2026) — Firefly
  • Spacelift Software Pricing & Plans 2026 — Vendr
  • Terraform Cloud Pricing: A Complete Guide (2026) — env0

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 22, 2026 DevOps Infrastructure Terraform Pricing Hashicorp

  Enjoyed this?

• When You Should Skip Terraform Entirely

  The last post in this series made the case that OpenTofu is the no-regrets default for new infrastructure projects. That’s true for the broad case of cloud-agnostic or multi-cloud setups where HCL parity, provider breadth, and a Linux Foundation governance model matter.

  It’s also not the whole story. There are at least three common scenarios where the right answer in 2026 isn’t Terraform or OpenTofu. It’s the cloud-native tool the hyperscaler ships with its platform. AWS has CloudFormation and the CDK. Azure has Bicep. GCP has Config Connector. Each one is technically superior to Terraform inside its own ecosystem, and each one removes a category of operational pain that Terraform inflicts.

  If you reflexively reach for Terraform every time, you’re probably overpaying in complexity for a multi-cloud option you’ll never exercise.

  The Small AWS-Native Startup: Use CDK

  If your engineering team is small, you’re shipping a SaaS product, and you’re 100% on AWS, you should probably ignore Terraform entirely. The right tool is the AWS Cloud Development Kit, layered on top of CloudFormation.

  The fundamental win is that CloudFormation eliminates state management. There is no terraform.tfstate file. No S3 bucket to provision. No DynamoDB lock table. No state-encryption configuration to figure out. The state lives in the AWS control plane, AWS manages locking and consistency, and your CI pipeline doesn’t need to know about any of that. For a small team, that’s a meaningful operational tax you don’t pay.

  The CDK is the part that makes this pleasant. It lets you define infrastructure in TypeScript, Python, Java, C#, or Go; so the languages your application engineers already know. There’s no HCL learning curve, no Sentinel policy DSL, no jq-in-bash to manipulate plan output. You write code, the CDK synthesizes CloudFormation templates, CloudFormation provisions the infrastructure.

  The objection people raise is “what if you go multi-cloud later?” In practice, most SaaS startups don’t. They get acquired, they pivot, or they grow large enough to have a dedicated platform team that does the migration deliberately. Optimizing for a hypothetical multi-cloud future that 90% of teams will never need is the textbook definition of premature abstraction. If you’re an AWS-native startup with fewer than 50 engineers and no concrete plans to leave AWS, the cost of running Terraform-as-multi-cloud-insurance is higher than the cost of a future migration that probably won’t happen.

  The Azure Enterprise: Bicep, Unless You Need More

  For organizations heavily invested in Microsoft’s stack, so Azure for compute, Azure DevOps for CI/CD and EntraID for identity; Bicep is the technically correct choice for most workloads.

  Bicep is Azure’s domain-specific language for infrastructure, designed as a replacement for the verbose ARM JSON templates everyone hated. Like CloudFormation, it’s stateless. You submit a desired-state Bicep file to the ARM control plane and ARM reconciles. No state file, no remote backend, no risk of corruption. Authentication is whatever RBAC permissions the deploying identity already has, with no provider credential configuration required.

  Bicep also gets day-zero feature support for new Azure capabilities. When Microsoft ships a new service, you can use it in Bicep the same day. The Terraform AzureRM provider has historically lagged by weeks or months, occasionally longer.

  The catch is scope. Bicep manages Azure. That’s the entire surface area. Larger organizations tend to need management of things outside Azure too: GitHub repositories and branch protection, EntraID groups, Datadog monitors, PagerDuty escalation policies, whatever SaaS services your platform touches. Bicep has no answer for any of that.

  That leaves two paths. The first is a hybrid: Bicep for Azure, separate tools for everything else, accept the cost of context-switching and the inability to express cross-domain dependencies in a single deployment. The second is Terraform or OpenTofu for everything, accepting the heavier operational tax of stateful IaC, in exchange for one tool that can do all of it. Neither is wrong; they’re different tradeoffs against the same constraint.

  The decision rule: if you’re managing only Azure resources, use Bicep. If you have cross-domain provisioning needs and you’d rather not maintain two parallel IaC stacks, Terraform (or OpenTofu) earns its keep.

  The GCP/Kubernetes Shop: Hybrid by Design

  For organizations heavily committed to Google Cloud and running most workloads on GKE, the right architecture isn’t either/or. It’s a hybrid that uses Terraform for the foundation and Config Connector for the application layer.

  Config Connector is a GCP-shipped Kubernetes add-on. It lets you manage GCP resources — Cloud SQL instances, Pub/Sub topics, storage buckets, service accounts — as standard Kubernetes Custom Resources. You write a YAML manifest, you kubectl apply, and a controller in the cluster reconciles the real-world GCP resource to match.

  The differentiator is continuous reconciliation. Terraform is episodic: it checks state at plan and apply time, and the rest of the time your infrastructure is unmonitored. If someone clicks around in the GCP console and manually changes a setting, Terraform won’t notice until the next pipeline run. Config Connector runs a controller loop that polls continuously. Manual drift gets reverted in real time.

  The right architectural boundary:

  • Platform layer (Terraform/OpenTofu): VPCs, subnets, foundational IAM, the GKE clusters themselves. These are slow-moving, security-critical, and you want a deliberate pipeline approval flow for them.
  • Application layer (Config Connector): Application-specific buckets, databases, service accounts, Pub/Sub topics. Application teams own these via the same YAML manifests they use for their pods, with the same GitOps workflow they already understand.

  This pattern gives platform teams strict guardrails on the foundation while letting application developers self-serve the resources their services need, without filing a Terraform PR every time they want a new bucket.

  The Decision Rule

  The honest version of all of this: Terraform/OpenTofu is the right answer when you need cross-domain or cross-cloud governance. For everything else, the cloud-native tool is usually less work, more current with the platform, and avoids the operational tax of state management.

  A reasonable decision tree:

  • Single-cloud, small team, AWS: AWS CDK + CloudFormation.
  • Single-cloud, single-domain, Azure: Bicep.
  • GCP with heavy Kubernetes use: Hybrid — Terraform/OpenTofu for foundation, Config Connector for application resources.
  • Multi-cloud, or cross-domain platform engineering (GitHub + cloud + identity + monitoring): OpenTofu.

  The mistake I think most teams are making is to default to Terraform because it’s the tool the senior engineer learned in their last job. The platform-engineering pitch … “we’ll standardize on Terraform so we can move to any cloud later” is correct in theory but almost never exercised in practice. If your team isn’t using the cross-cloud capability today, you’re paying for an insurance policy you’ll never collect on.

  Next post in this series digs into the other side of that calculation: what HCP Terraform actually costs in 2026, and why even teams that need cloud-agnostic IaC are looking for the exit from the commercial orchestration platforms.

  Sources

  • Bicep Vs Terraform: Choosing The Best IaC Tool For Azure — Synextra
  • Terraform vs Bicep vs ARM Templates 2026 Compared — Exodata
  • Comparing Terraform and Bicep — Microsoft Learn
  • Terraform vs Bicep vs ARM: Lessons from the Trenches — Vaibhav Gujral
  • How to Use the GCP Config Connector with Terraform — OneUptime
  • How Config Connector compares for infrastructure management — Google Cloud Blog
  • Are Terraform’s days numbered? — Alistair Grew

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 21, 2026 DevOps Infrastructure Terraform Aws Azure Gcp

  Enjoyed this?

• OpenTofu Is the No-Regrets Default for 2026 Infrastructure

  Hashicorp’s adoption of the Business Source License in late 2023 was a defensive business decision. Companies like Spacelift, env0, and Scalr were building paid commercial platforms on top of MPL-licensed Terraform, capturing significant revenue from an ecosystem Hashicorp was largely funding. The same pattern played out with Redis Labs facing AWS ElastiCache, Elastic facing Amazon OpenSearch, and MongoDB facing the cloud hyperscalers before its move to the SSPL. The BSL is a rational corporate play: keep the core open enough to preserve mindshare, restrict the terms enough that pure resellers can’t extract value without engaging commercially. From the standpoint of a publicly traded company with a board to answer to, it made sense.

  But it also broke a tacit contract. Hashicorp had spent a decade positioning Terraform as infrastructure’s git. Neutral, ubiquitous, irreplaceable. A license that lets a single vendor change the terms when the shareholder math demands it is not neutral, and a large portion of the community decided they weren’t comfortable with that risk. The Linux Foundation forked the last MPL-licensed Terraform release and shipped it as OpenTofu. Two years later, OpenTofu has crossed 10 million downloads, holds HCL parity with Terraform, supports the same provider ecosystem (AWS, Azure, GCP, Kubernetes, everything), and ships features Terraform itself doesn’t have.

  For greenfield infrastructure in 2026, OpenTofu is the no-regrets default. For existing Terraform codebases, the migration is mostly a binary swap. The reasons to still pay for Terraform are mostly inertia. Let me explain.

  The Migration Is Mostly Free

  The technical case for “stay on Terraform” essentially doesn’t exist. OpenTofu reads the same HCL. It produces the same execution plans. It maintains the same state file format. It interfaces with the same providers, including the ones Hashicorp wrote, because the provider API was never the part Hashicorp tried to lock down.

  To migrate a non-trivial Terraform codebase to OpenTofu, you do roughly this:

  1. Swap terraform for tofu in your CI binary install step.
  2. Update any pipeline scripts that hardcoded the binary name.
  3. Run tofu init -migrate-state once.
  4. Run tofu plan and confirm it produces an empty diff against the existing state.

  There are edge cases, like modules pinned to specific Terraform-version constraints or providers that gated features on the Hashicorp-only registry. But for the vast majority of codebases, the migration is a one-afternoon job, including the PR review and the team announcement.

  What you get in exchange is governance under the Linux Foundation, an active multi-vendor contributor base, no future license surprises, and a really nice to have feature not in Terraform currently: native state encryption.

  State Encryption Is the Real Reason

  Terraform state files have a property nobody enjoys discussing. They contain everything sensitive about your infrastructure, and they store it in plaintext.

  That’s not a misconfiguration. That’s the design. The terraform.tfstate JSON file holds resource IDs, ARNs, network topology, credentials surfaced as outputs, RDS connection strings, and any sensitive value a module decided to track. When you use S3 or Azure Blob as a remote backend, you get encryption at rest, meaning the cloud provider’s storage layer is encrypted. The state itself, the thing your CI pipeline downloads and uploads on every run, is plaintext JSON. Anyone with read access to the bucket (your CI runner, your laptop, anything assuming the role) gets the cleartext.

  OpenTofu solves this with native, client-side state encryption introduced as a first-class feature. The state is encrypted by the engine before it leaves the machine. The remote backend never sees plaintext at all. The configuration looks like this:

  terraform {
    encryption {
      key_provider "aws_kms" "primary" {
        kms_key_id = "arn:aws:kms:us-east-1:..."
        region     = "us-east-1"
        key_spec   = "AES_256"
      }
  
      method "aes_gcm" "primary" {
        keys = key_provider.aws_kms.primary
      }
  
      state {
        method = method.aes_gcm.primary
      }
  
      plan {
        method = method.aes_gcm.primary
      }
    }
  }
  

  Three pieces. A key provider (AWS KMS, GCP KMS, OpenBao, or a local passphrase via pbkdf2), an encryption method (AES-GCM is the standard pick), and explicit targets for state, plan, or both.

  The migration path from existing plaintext state requires a fallback block. OpenTofu refuses to read plaintext once encryption is enabled, which is the right default, but it means you need to tell it “this one time, read the legacy state and re-encrypt it.” After one successful apply, you remove the fallback and you’re done.

  Terraform doesn’t have this. Hashicorp’s official answer is still “use a backend that encrypts at rest and audit your IAM policies carefully.” Which is fine, until your CI logs the state diff into a third-party observability tool, or someone runs terraform show over a Slack screenshare, or an attacker gets a transient role to your backend bucket. The threat model OpenTofu’s encryption closes is the threat model that matters.

  The AI Wrinkle

  There’s a meta-argument unfolding alongside all of this: AI is making the choice of execution engine less important.

  Industry telemetry says 71% of cloud teams have seen an exponential increase in IaC volume from generative AI. The thing AI is generating, in most cases, is HCL, which is the lingua franca for both Terraform and OpenTofu. As the volume of AI-authored infrastructure grows, the role of HCL shifts from “the language engineers write” toward “the intermediate representation an agent emits.” Manual HCL authoring is on track to become a niche skill in the same way hand-tuning compiler output is a niche skill.

  In that world, the execution engine is plumbing. The valuable layer is everything around it: state management, drift detection, policy enforcement, cost guardrails, audit trails. Which is exactly the layer where vendor lock-in does the most damage and where open governance matters most. The AI argument doesn’t undercut the OpenTofu case. It reinforces it.

  What To Do

  If you’re starting a new infrastructure project, use OpenTofu. There is no good reason to start a 2026 greenfield project on a single-vendor BSL-licensed engine when the Linux Foundation-governed open-source alternative is right there, with full HCL parity, the same provider ecosystem, and features Terraform doesn’t have.

  If you have an existing Terraform codebase, schedule the migration. It’s a one-afternoon job per repo. Get state encryption while you’re at it.

  If you’re heavily integrated with HCP Terraform, this is the harder case. The migration off the proprietary HCP features (Sentinel policies, the registry, the integrated dashboards) is real work. But it’s also the case where you have the most to lose. HCP Terraform’s pricing model has gotten aggressively worse, and OpenTofu’s existence means you have actual leverage in the next renewal conversation. The next post in this series digs into exactly what HCP pricing looks like in 2026 and why so many organizations are getting six-figure renewal quotes for infrastructure they were paying $20K for two years ago.

  This is the first of a four-part series on the 2026 IaC landscape. Up next: cloud-native vs cloud-agnostic tooling, and when to use AWS CDK, Bicep, or Config Connector instead of Terraform/OpenTofu at all.

  Sources

  • 2026 IaC Predictions: What Cloud Leaders Must Prepare For ControlMonkey
  • Terraform vs OpenTofu in 2026: Should You Stay or Switch?
  • Terraform or OpenTofu in 2026? Here’s What I Actually Think Jae Wook Kim
  • OpenTofu vs Terraform in 2026: Is the Fork Finally Worth It? Mechcloud Academy
  • OpenTofu vs. Terraform: A Practical Guide for Enterprise Infrastructure Teams env0
  • State and Plan Encryption OpenTofu docs
  • How to Use OpenTofu State Encryption OneUptime
  • State Encryption with OpenTofu Ned in the Cloud

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 20, 2026 DevOps Infrastructure Opentofu Terraform

  Enjoyed this?

• A Dotfiles Manager That Snapshots Every Change

  Managing dotfiles in 2026 is a solved problem in the same way that managing your own backups is a solved problem: there are five tools for it, all of them work, all of them require you to set up some plumbing first, and once you’re set up you still don’t have a great answer to “I just broke my shell config, get me back to yesterday.”

  The conventional answer is some combination of: a git repo for your ~/.zshrc and friends, a symlink script (or stow, or chezmoi, or yadm), and the discipline to remember to commit after every change. The setup is a one-time hassle. The “wait, what did I change?” recovery story is not great. And if you want to sync across machines, you’ve now got opinions about remote repos, SSH keys on a fresh box, and which order things have to happen in.

  I wanted something different, so not a configuration framework, but a record of every change to the files I care about, in a place I can roll back from, with the lowest possible setup cost.

  That’s what dfm is.

  What It Does

  dfm is a single static Go binary. You point it at the files you want to track (~/.zshrc, anything under ~/.config/, whatever), and every time one of them changes it takes a content-addressed snapshot. The snapshots live on disk in ~/.local/share/dotfiles/backups/. A small state database (SQLite locally, or libSQL via Turso if you want cross-machine sync) records which file maps to which snapshot at which point in time.

  You can roll back. You can diff against an old snapshot. You can see when you last touched a file. And because every snapshot is content-addressed, you never re-store the same bytes twice — switching themes in ~/.zshrc ten times costs the size of two configs, not ten.

  The other half is the backup story. dfm init walks you through cloning (or creating, via gh) a private GitHub repo that mirrors your tracked files plus their history. The point isn’t to make you adopt a new git workflow. It’s that pulling your config onto a fresh machine should be one command, and recovering from rm -rf should never have a “well, hopefully my last commit was recent” caveat.

  Why Setup Is the Hard Part

  The reason people don’t audit their dotfiles is the same reason people don’t back up their laptops: the setup is annoying, and the payoff is theoretical until it isn’t.

  dfm init is a six-step interactive wizard. It detects a TURSO_DATABASE_URL env var if you’ve got one, offers sensible defaults for everything else, lets you opt in to tracking ~/.zshrc immediately, and writes a single config file with the right permissions. Re-run it on an existing config and it pre-fills every prompt with your current value, so the cost of changing your mind later is also low. --yes accepts every default for scripted setup.

  If that sounds boring, that’s the point. Boring is what makes a tool actually get used.

  The AI Bit

  There’s an optional AI integration. dfm suggest <file> asks a local AI CLI (Claude Code by default, configurable) to propose an improvement to one of your tracked files, returns the proposal as a unified diff, and stores it as a pending suggestion. dfm apply <id> reviews the diff and applies it, with a fresh snapshot first, so you can roll back if the suggestion turns out to be wrong.

  I’m exited to try this feature out, because I’m sure there is something i"m doing wrong. The “Look at my ~/.zshrc and tell me what I could clean up” is useful feature that doesn’t require me copy and pasting or granting read or write access to my entire home directory.

  Where to Get It

  github.com/llbbl/dotfiles-manager. Pre-built binaries for darwin and linux on arm64/amd64. Current version, as of writing, is v1.4.0.

  If you’ve been meaning to actually back up your dotfiles and the friction has stopped you, this is the post where I tell you the friction is solvable.

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 19, 2026 DevOps AI Programming Go Dotfiles

  Enjoyed this?

• Your AI Coding Agent Can Read Every Secret on Your Machine

  Every developer running an AI coding agent has handed that agent the keys to their machine. Not metaphorically. Literally. The agent runs as your user. It can read every file you can read, execute every command you can execute, and hit every API your stored credentials authorize.

  For most workflows, that’s the point. You want the agent to read your code, modify your project, ship your work. But there’s a quieter implication: the agent can also read your .env files. It can invoke your secret-management tooling. It can grep for API_KEY= across your home directory. And nothing in the agent stack says “wait, you didn’t ask for this.”

  Same-UID isolation isn’t isolation. It’s the absence of isolation labeled politely.

  The usual answer to “keep secrets safe from your coding agent” is: don’t store them where the agent can find them. Use a cloud secret manager. Rotate aggressively. These are good practices, and for local development, they’re often impractical. The agent is going to encounter secrets whether or not your security-best-practices doc approves.

  So over the last week, I built an audit subsystem into lsm, my Local Secrets Manager. The whole thing is designed to answer one forensic question: did anything weird touch my secrets last night?

  The Threat Model

  A defense without a threat model is theater, so let me be specific.

  The threat isn’t a sophisticated remote attacker. lsm is public, open-source code. The threat isn’t a buggy lsm either; bugs happen, and the user can read the source.

  The threat is the agent layer running adjacent to lsm. Coding agents have legitimate access to a wide swath of your filesystem. They’re imperfect at intent inference. They sometimes get prompt-injected. They sometimes run in the background while you’re asleep. When an agent calls lsm get prod DATABASE_URL, the action is indistinguishable from you doing the same thing. The audit log’s job is to make those calls retrospectively distinguishable.

  A secondary threat is an agent covering its tracks. If something reads a secret and then edits the audit log to erase the evidence, the log is worse than useless.

  What Got Built

  The audit subsystem records every access as a structured event: a sequence number, a timestamp, the action, the app and environment, an Actor block describing the calling process, and two cryptographic fields linking each event to the previous one.

  The Actor block was the interesting design problem. It captures parent process ID, parent process name, TTY device path (or empty if there’s no terminal), current working directory, an agent marker derived from environment variables that tools like Claude Code, Cursor, Aider, and Continue set, and the calling user ID. Every field is captured every time. No omitempty. UID zero is a real, meaningful value, and silently dropping it would be a footgun.

  Events land in a hash-chained JSONL file at ~/.lsm/audit.jsonl. Each row carries the SHA-256 of the previous row plus its own body. If anyone edits, inserts, or deletes a row in the middle, the next row’s prev no longer matches and lsm audit verify surfaces the break.

  The chain doesn’t catch tail truncation. If you chop off the end of the file, what’s left is internally consistent. A sidecar file storing the last expected hash is the obvious fix, and I deliberately rejected it. lsm is public code. Any local attacker who knows about the sidecar can rewrite both files in lockstep. Tail-truncation detection is deferred to the off-machine path: when events ship to a remote stack, the last hash naturally lives somewhere the local attacker doesn’t control.

  Reading the Log

  Three commands cover the read side. lsm audit tail does what you’d expect. lsm audit show <seq> prints a single event. lsm audit query is the workhorse, with every field as a filterable dimension: --app, --env, --event, --parent-comm, --agent-marker, --tty present|absent, --since, --until. Output is JSONL when piped and columnar text when interactive.

  Then there’s lsm audit suspicious, which runs four hard-coded detectors in one pass:

  • Outside hours. Events whose timestamps fall outside 07:00–23:00. The 3 a.m. canary.
  • Burst. More than N events from a single parent process within a sliding window. The runaway-agent canary.
  • New parent_comm. Process names not seen in the prior 30 days. The “what is this new thing” canary.
  • Non-interactive, no agent. No TTY, no recognized agent marker. The “what is even running this” canary.

  A single event can stack reasons. A 3 a.m. burst from an unknown parent is unambiguously interesting.

  The detector doesn’t learn baselines, doesn’t call out to an ML model, doesn’t require a service. High-signal patterns are obvious patterns, and obvious patterns are well-served by hard-coded predicates.

  Shipping Events Off the Box

  If you already run an observability stack, lsm can ship audit events over OTLP (the OpenTelemetry wire protocol). Three design choices matter here.

  The local file sink is always authoritative. The remote sink is a mirror, not a replacement. An lsm operation never fails because the remote endpoint is down.

  Redaction is allowlist-based. App and environment names are HMAC-hashed with a per-host salt before becoming labels. The TTY device path is dropped and replaced with a tty_present: true/false boolean. Secret values, cwd, hash, prev, and the schema version never leave the host. Secret names are replaced with key_present: true markers; the remote observer can see that a key was accessed, never which key.

  Events whose name starts with audit. (chain failures, suspicious matches, sink drops) are always local. Telling a remote attacker that local integrity has been compromised is counterproductive.

  What’s Still Open

  The most important non-feature: no command in lsm emits events yet. set doesn’t log. get doesn’t log. delete doesn’t log. The plumbing is complete, the calls are not wired in. Each emit site needs careful thought about which fields are appropriate, whether the event should be local-only, and how it interacts with sensitive operations. That’s the next chunk of work.

  The agent-coding era is normalizing a model where AI tools have wide-ranging access to developer machines. The premise that the agent operates as a fully-trusted local user is unlikely to change soon. Managing the risk means visibility. It means being able to answer “what touched my secrets last night” with a record the agent couldn’t silently rewrite.

  The code is at github.com/llbbl/lsm. The full design lives in docs/observability.md.

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 18, 2026 DevOps AI Programming security

  Enjoyed this?