DevOps

• Graphify Turns Your Repos Into a Map You Can Query

  Navigating code dependencies inside a single repository is already hard enough. But if you’re on a microservice setup, or a split frontend and backend, tracking what depends on what across multiple repos is a special kind of misery. A backend API route changes. Which frontend components just broke? Good luck. You’re grepping three workspaces and hoping you didn’t miss one.

  So when I ran across Graphify, an open-source project from Graphify Labs (YC S26), it caught my attention. It maps your code directories into queryable knowledge graphs. Not fuzzy text search. Not an expensive vector RAG lookup that burns tokens every time you ask it a question. A deterministic index of your codebase.

  Let me walk through how it works, why it’s useful for AI coding agents, and the part I wanted to figure out: how to stitch multiple repos into one unified map.

  What Graphify Does

  Instead of guessing at relationships, Graphify parses your source and builds a real graph out of it. Three pieces make it tick:

  1. Deterministic AST parsing. It uses tree-sitter grammars locally to parse roughly 40 languages, pulling out classes, functions, calls, and imports. No LLM tokens, no API rate limits. Just parsing.
  2. Explicit vs. inferred edges. Every relationship gets a confidence tag. EXTRACTED means it’s right there in the syntax, like an import or a direct function call. INFERRED means it deduced the connection from context. You always know how much to trust an edge.
  3. Leiden community clustering. It automatically segments your code into logical domain boundaries, which makes it easy to spot the “god nodes”, the files with way too many dependencies hanging off them. Those are usually the first thing you want to refactor.

  Merging Multiple Repos Into One Graph

  This is the part I cared about. Graphify supports it natively through the CLI, and here’s the flow straight from the docs (I haven’t run it on my own repos yet). Say you’ve got a frontend repo and a backend repo. Three steps.

  Step 1: Scan each repo on its own. Run the scan inside each folder. Results land in a graphify-out/ directory.

  # In your frontend repo
  cd ~/Work/frontend
  graphify .
  
  # In your backend repo
  cd ~/Work/backend
  graphify .
  

  Step 2: Merge the graphs. The merge-graphs subcommand joins the JSON outputs into one combined map of nodes and relationships.

  graphify merge-graphs \
    ~/Work/frontend/graphify-out/graph.json \
    ~/Work/backend/graphify-out/graph.json \
    --out ~/Work/combined_graph.json
  

  Step 3: Traverse it, or hand it to your agent. Now you can trace a call path straight across the service boundary, or serve the combined graph to a coding agent over MCP.

  # Trace a path across the frontend/backend boundary
  graphify path "login_component.ts" "auth_controller.py" --graph ~/Work/combined_graph.json
  
  # Or expose the combined graph to your coding agent over MCP
  python -m graphify.serve --graph ~/Work/combined_graph.json
  

  That path command is the whole pitch, honestly. You point it at a frontend file and a backend file and it tells you how they’re connected. No manual grep archaeology.

  Why This Matters for AI Coding Agents

  If you use Claude Code, Cursor, or Antigravity, you already know the problem. Feed the agent raw files and you torch the context window in about four prompts. Point it at Graphify’s output instead, the GRAPH_REPORT.md or the graph.json over MCP, and the agent can do a few things it otherwise can’t:

  • Figure out exactly which files a refactor will touch before it edits anything.
  • Trace dependency lineage across code boundaries deterministically, not by vibes.
  • Describe your architecture based on the actual shape of the code, not a hallucinated version of it.

  That last one is underrated. Half of “the AI got confused” moments happen because the AI never saw the whole picture.

  Two Gotchas Before You Install

  A couple of things will trip you up, so here they are up front.

  The package name has a typo built in. graphify was already taken on PyPI, so the official package is registered as graphifyy. Two y’s. You install it like this:

  pip install graphifyy
  

  Watch your Python version. The Leiden community detection library has C-extension limits, so Graphify currently runs best on Python under 3.13. Worth checking or switching to a compatible version (like 3.12) using mise.

  ---

  The honest appeal here isn’t the visualization, pretty as the HTML map is. It’s that cross-repo dependency tracing has been a manual, error-prone chore for as long as I’ve worked on split codebases, and this makes it a single command.

  Sources

  • Graphify Labs on GitHub: setup requirements, supported parsers, and CLI options.
  • Auriga IT’s Graphify introduction: explains the three-pass architecture and Leiden clustering optimization.
  • Graphify on PyPI: package installation details and version compatibility.
  • Aider’s Repository Map: on using tree-sitter to parse AST-based codebase maps for token-efficient coding context.

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  Jul 17, 2026 DevOps AI Programming Tools

  Enjoyed this?

• Why Your AI Stack Needs a Gateway

  Picture an autonomous agent loop dying at step 45 of a plan it’s been grinding through for the better part of an hour. Not because the plan was wrong. Because OpenAI handed back an HTTP 429 at exactly the wrong moment, and the whole thing fell over. An hour of work, gone to a transient rate limit.

  That’s the moment most people start thinking about an AI gateway, whether they know the term or not.

  Hardcoding API keys and endpoints straight into your application code feels fine right up until it doesn’t. You start with a simple chat wrapper. Then you’re running agents like OpenClaw or Hermes that chew through hours of command-line work. Then you’re wiring up a real backend that talks to OpenAI, Anthropic, Google, and a couple of self-hosted models. Now every one of those providers is a single point of failure. One of them rate-limits you or goes dark, and your workload crashes.

  So people are dropping a new piece into the stack to deal with it: the AI gateway.

  What It Is

  An AI gateway is a specialized reverse proxy that sits between your application and the model providers upstream. Instead of importing a different SDK and juggling a different set of environment variables for every vendor, your app talks to one OpenAI-compatible endpoint. The gateway handles routing, retries, load balancing, security, and caching behind the scenes.

  The request flow is straightforward. Your app makes a normal OpenAI-style call. The gateway checks its cache first, and if it’s seen a semantically similar prompt it serves the answer in milliseconds. On a miss, it runs the request through whatever security layer you’ve configured, then routes to a provider, with a failover path ready if the primary one is down. Your code never has to know any of that happened.

  Who Needs One

  The value splits cleanly across two kinds of people.

  If you’re building agents, the pitch is survival. An agent that runs for hours is going to hit a 429 or a 500 eventually. A gateway catches those, does exponential-backoff retries, and can swap providers mid-task, falling back to Claude if OpenAI is having a bad day. Your long-running loop stays alive instead of dying at step 45. You also get to keep your real vendor credentials locked in one vault and hand your agent scripts restricted local keys instead.

  If you’re the tech lead shipping customer-facing AI, the gateway becomes your governance layer:

  • Observability. One console showing latency, time to first token, cost, and raw prompts across every team, instead of five fragmented dashboards.
  • Spend management. Hard dollar budgets per team or per key, so a runaway recursive loop can’t quietly drain the corporate card.
  • AI firewalls. Automatic PII masking for emails, phone numbers, and stray API keys, plus prompt-injection blocking at the edge before anything leaves your network.
  • Semantic caching. Vector similarity checks catch prompts that mean the same thing and serve a cached answer, cutting both the token bill and the latency to near zero.

  The Landscape

  This space is filling up fast, and the options sort themselves by how you want to deploy. A quick tour of the ones worth knowing:

  • OpenRouter is the managed broker. Hundreds of models under one credit balance, with dynamic pricing, fallbacks, and bring-your-own-key support. The easiest place to start.
  • LiteLLM is the self-hosting standard. Python, wildly popular for building a private gateway inside your own VPC, with database-backed key budgets.
  • Bifrost is the same idea written in Go for teams that care about throughput. It adds almost no latency overhead and benchmarks its P99 routing well ahead of the Python options.
  • Portkey leans into prompt management. Versioned prompt templates live in a central playground and get called by API, which is handy if your prompts change more than your code.
  • Cloudflare AI Gateway is the zero-devops edge play, built on Cloudflare’s CDN with fast caching, Logpush exports, and native edge firewalls.
  • Vercel AI Gateway plugs straight into the Vercel AI SDK, so you route serverless traffic through it with a simple string change.

  There’s no single right answer here… and i’m pretty sure I’m leaving a few out. If you just want to stop thinking about it, OpenRouter or Cloudflare. If you want control and a VPC, LiteLLM or Bifrost. If prompts are your headache, Portkey.

  The real takeaway is smaller than the tooling makes it look. The moment your app depends on more than one model, or on any single model staying up, you’ve got an infrastructure problem, not an application problem. A gateway is just where you put the solution so your code doesn’t have to carry it.

  An agent like that crashes at step 45 today. Put a gateway in front of it, and it doesn’t.

  ---

  Sources

  • OpenRouter
  • Vercel AI Gateway documentation
  • LiteLLM Proxy
  • Bifrost (Maxim AI)
  • Cloudflare AI Gateway docs
  • Portkey Gateway docs

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  Jul 15, 2026 DevOps AI Tools LLM Infrastructure

  Enjoyed this?

• Autopilot for Your Docs: A Look at LangChain's OpenWiki

  Writing documentation for a repo is the chore nobody volunteers for. Keeping it current is somehow worse. You refactor one service or change a database schema, and the architecture overview you wrote three months ago is instantly a lie.

  So when the LangChain team dropped OpenWiki, I had to take a look. It’s an open-source CLI tool and agent framework that writes and maintains documentation for your codebase automatically. I read through the source rather than running it, so here’s an honest read on the idea, how it works under the hood, and what you’ll run into if you point it at a real repo today.

  What Is OpenWiki?

  The premise is simple: AI coding assistants are only as good as the context they can reach. Instead of stuffing your prompts full of raw source files, OpenWiki generates a structured, interlinked Markdown wiki inside an openwiki/ directory right in your repo. It’s built to be read by coding assistants like Cursor, Claude Code, or Codex CLI.

  graph TD
      Diff[Git Workspace Diff] -->|CLI/CI| CLI[OpenWiki CLI]
      CLI -->|LangGraph| Agent[Documentation Agent]
      Agent --> Analyzer{Code Analyzer}
      Analyzer -->|Incremental Update| Docs[openwiki/ Markdown]
      Docs -->|Appends Pointer| Config[CLAUDE.md / AGENTS.md]
  

  Three things it does that I like:

  • Autopilot docs. You don’t hand-write or hand-format anything. The agent inspects your layout and produces overview, architecture, workflow, and API reference pages.
  • Incremental git-diff sync. It doesn’t re-read your whole workspace on every commit. It looks at the diff since the last scan and only rewrites the pages those changes touched.
  • Prompt hooking. It appends a reference pointer to your CLAUDE.md and AGENTS.md so downstream assistants check the wiki first.

  That last one is interesting and is starting to show up in other harness adjacent tools. For this project, it turns the docs into a context layer your agents are told to read.

  The LangChain Footprint

  This is built by the LangChain team, and it shows in the dependency list.

  • Orchestration runs on @langchain/core and deepagents, with a stateful LangGraph engine backed by @langchain/langgraph-checkpoint-sqlite that stores checkpoints and agent state locally.
  • The CLI uses ink, so it renders a clean React-based interface right in the terminal.
  • Tracing ships with LangSmith support out of the box, which is worth more than it sounds. When you want to know why a particular page got rewritten, or what a run cost you, that audit trail is right there.

  If you’re already living in the LangChain ecosystem, none of this will surprise you. If you’re not, it’s a lot of framework to pull in for a docs tool. Fair tradeoff or not depends on how much you value the tracing.

  The Catch: Tokens and Rate Limits

  It’s still early but here is what I would wathc out for. The first run is expensive.

  When you bootstrap OpenWiki on a medium-to-large repo, the agent has to read, analyze, and index everything. Two things happen:

  • You’ll hit rate limits. That initial pass will saturate your provider’s API limits fast. Expect a parade of 429 Too Many Requests on any large source tree.
  • You’ll spend real money. Bootstrapping a large repo can burn through a huge number of tokens in a single run.

  If you’re going to try it, configure your .openwiki settings to exclude the folders that don’t need documenting. node_modules, dist, generated assets, all of it. There’s no reason to spend tokens teaching an agent about your vendored dependencies. If you have access to a high-throughput endpoint or a fast local model through Ollama or LiteLLM, the bootstrap is a lot less painful.

  The steady state is fine, since the git-diff sync keeps ongoing runs cheap. It’s that first index that hurts.

  Should You Use It?

  OpenWiki is new and moving fast, which means you should expect config keys and command arguments to shift under you for a while. This is not a set-and-forget tool yet.

  But the core idea is interesting. An agent running quietly in a pre-commit hook or CI, keeping your repo’s context layer in sync so your other agents have something accurate to read, is a real quality-of-life upgrade. Docs that maintain themselves have been a fantasy for as long as I’ve been writing code. On a read of the source, this is the most serious attempt I’ve seen at pulling it off.

  I haven’t run it yet, just read through the code, and I’m not putting it anywhere just yet. But the idea is right, and I’m watching where this one goes.

  ---

  Sources

  • LangChain OpenWiki repository for setup, commands, and configuration.
  • LangChain blog for the launch announcement and design philosophy.
  • LangGraph JS docs for the local SQLite checkpointing and state details.

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  Jul 14, 2026 DevOps AI Langchain Developer tools Documentation

  Enjoyed this?

• Adding Types to JSON with Dhall

  A few months ago I wrote a post asking whether there’s something better than JSON. Two configuration languages that sit above JSON kept coming up: CUE and Dhall. Both give you the things JSON lacks when you author config by hand, and both compile down to plain JSON, YAML, or whatever your services actually read. I spent more time with CUE back then and never gave Dhall a real look. This post is me going back for that second look, because the one feature I kept wanting was a type system over my config.

  JSON is the universal language of API payloads and config files, and I don’t want that to change. But as a format for authoring configuration by hand, it’s rough:

  • No comments.
  • No variables or functions, so you copy-paste the same block ten times.
  • No type system, so "8080" and 8080 look equally valid.
  • No imports, which is how you end up with a 2,000-line monolith nobody wants to touch.

  The usual escape hatch is a templating engine like Jinja or Helm, or a real programming language like Python or TypeScript that spits out JSON. That works, but you’ve traded one problem for a scarier one: your config generator is now Turing-complete. It can crash, hang in an infinite loop, or reach out and read some local environment variable, and it’ll do it at 2 AM when the pipeline runs.

  This is where Dhall comes in.

  What is Dhall?

  The short version: Dhall is JSON plus types, plus functions, plus imports. It’s a programmable, strongly-typed configuration language.

  The part I actually care about is what it doesn’t have. Dhall is not Turing-complete. No arbitrary recursion, no side effects. Every Dhall program is guaranteed to terminate. You get the abstraction power of a functional language like Haskell or Elm, with the guarantee that it will never hang your build. That’s a different trade than “just write a Python script.”

  The problem, in JSON

  Here’s a normal config.json for a microservice:

  {
    "serviceName": "payment-api",
    "port": 8080,
    "environment": "production",
    "database": {
      "host": "db.internal.net",
      "maxConnections": 50
    }
  }
  

  Three ways can be a problem in production: someone writes "port": "8080" and the service won’t boot, someone typos "prodution" and it silently runs in debug mode, or someone forgets maxConnections entirely and you get a null blowup at runtime. Nothing catches any of it until it’s live.

  The same thing, typed

  In Dhall you define the shape up front. Enums, record types, default values:

  -- schema.dhall
  let Environment = < Local | Staging | Production >
  
  let Database =
        { Type = { host : Text, maxConnections : Natural }
        , default = { maxConnections = 20 }
        }
  
  let Config =
        { Type =
            { serviceName : Text
            , port : Natural
            , environment : Environment
            , database : Database.Type
            }
        , default =
            { port = 8080, environment = Environment.Local }
        }
  
  in  { Environment, Database, Config }
  

  Now you author against that schema, and you get defaults and composition for free:

  -- config.dhall
  let Schema = ./schema.dhall
  
  let myConfig : Schema.Config.Type =
        Schema.Config.default
        // { serviceName = "payment-api"
           , environment = Schema.Environment.Production
           , database = Schema.Database.default // { host = "db.internal.net", maxConnections = 50 }
           }
  
  in  myConfig
  

  Misspell Production, or pass "8080" as a string, and Dhall throws a type error before a single line of JSON is generated. Hopefully the benfit is now clear; adding a type safety layer to your config files.

  Compiling down to JSON

  You don’t ship Dhall to your services. You ship the JSON they already understand:

  brew install dhall-json
  dhall-to-json --file config.dhall
  

  Out comes clean, boring, standard JSON. Your services never know Dhall was involved. The part that I like is the safety lives at authoring time, and the runtime artifact stays dumb.

  Two features worth knowing about

  Hermetic imports with hash pinning. Dhall can import from a URL, so shared utilities live in one place instead of being copy-pasted across five repos. To keep someone from swapping the file out from under you, you pin the import to a SHA-256 hash of its normalized form:

  let Prelude =
        https://prelude.dhall-lang.org/v22.0.0/package.dhall
          sha256:10db4c919c25e4d262db3ed0d1d6120da3e3906673f00e3012c1d14e1963976a
  

  If the remote content changes, the hash won’t match and the build fails. The hash above is just an example, and each Prelude version has its own, so don’t copy it by hand. dhall freeze --inplace config.dhall computes the correct hashes for whatever you’ve imported and pins them automatically.

  Exhaustive matching with merge. When you map a union type to output, Dhall makes you handle every variant:

  let getLogPrefix = \(env : Environment) ->
        merge
          { Local = "[DEV] ", Staging = "[STAGE] ", Production = "[PROD] " }
          env
  

  Add a QA variant later, and every merge block that touched Environment fails to compile until you deal with it. No forgotten switch case slipping into production. The compiler keeps a running list of everything you now owe it.

  Is it worth it?

  	Raw JSON	Dhall
  Type safety	None, fails at runtime	Static, at compile time
  Comments & logic	No	Yes
  Termination	N/A	Guaranteed
  Dependency pinning	No	SHA-256
  Output	Consumed directly	Compiles to JSON/YAML/TOML

  For a two-key config file, it doesn’t make sense, but once you’re staring down Kubernetes manifests, a pile of near-identical microservice configs, or anything where a typo takes down a service, the calculus changes. You keep clean static JSON as the thing your services actually read, and you move all the ways-to-get-it-wrong to a place where a compiler catches them first.

  ---

  Sources

  • Dhall Language Tutorial & Cheatsheet: records, union types, default overrides, dhall-to-json, and dhall freeze.
  • Dhall language standard on GitHub: the non-Turing-complete design and the semantic integrity hash spec.

  ---

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  Jul 11, 2026 DevOps Programming Json Configuration Dhall

  Enjoyed this?

• Running Mem0's Memory Backend as Real Infrastructure with Ansible

  Mem0 works best when you treat its memory backend as infrastructure, not as some throwaway process you start by hand and forget about. There are two pieces worth automating early: the vector store and the graph store. Qdrant holds the embeddings for semantic lookup. Neo4j holds the relationship-oriented graph memory.

  Quick note before we go further, because this can be confusing. When folks talk about Mem0’s “graph” support, that is not GraphQL. GraphQL is an API query layer you’d put in front of your app. Mem0’s graph memory is an actual graph database, usually Neo4j, that the client talks to over Bolt. Different thing entirely.

  This post walks through a practical Ansible shape for running those backing services with Docker Compose. I’m not going to hand you a complete role. The point is to show the decisions that make the setup repeatable, because those are the parts people usually get wrong.

  What the deployment actually does

  Three responsibilities, that’s it:

  1. Get the Mem0 compose project onto the server.
  2. Write a server-local .env file with the runtime config.
  3. Start the Compose stack with Qdrant and Neo4j.

  In a small setup, the Python client runs on your laptop while the services run on a LAN host or a small VPS. The server exposes these ports, but only to trusted networks:

  • Qdrant HTTP: 6333
  • Qdrant gRPC: 6334
  • Neo4j browser: 7474
  • Neo4j Bolt: 7687

  If you’re on a public internet host, bind these to localhost or hide them behind a VPN, firewall, or private network. Do not casually publish database ports to the internet. I shouldn’t have to say remind you…

  Role inputs

  A generalized role only needs a handful of variables:

  mem0_home: /opt/mem0
  mem0_repo: [email protected]:your-org/your-mem0-project.git
  mem0_branch: main
  
  mem0_qdrant_collection: default
  mem0_neo4j_enabled: true
  
  mem0_embedder_base_url: https://api.example.com/v1
  mem0_embedder_model: your-embedding-model
  mem0_embedder_dimensions: 1024
  

  The collection name matters more than it looks. Qdrant collections have a fixed vector size. If you switch embedding models and the dimensions change, create a new collection. Don’t try to reuse the old one. This is a common way people break a working Mem0 setup, so treat the collection name as part of the embedding config, not an afterthought.

  Secrets

  Keep API keys and database passwords out of your regular vars files. Ansible Vault, a secrets manager, whatever your deployment system gives you. Then template a .env file with tight permissions:

  - name: Write Mem0 environment file
    ansible.builtin.template:
      src: mem0.env.j2
      dest: "{{ mem0_home }}/.env"
      mode: "0600"
    no_log: true
  

  That no_log: true is not optional. Template diffs will happily leak plaintext API keys and graph passwords into CI logs, terminal scrollback, and ticket attachments. Once a secret lands in a CI log, you’re rotating it, not deleting it.

  The template itself stays small:

  EMBEDDER_API_KEY={{ vault_embedder_api_key }}
  EMBEDDER_MODEL={{ mem0_embedder_model }}
  EMBEDDER_DIMENSIONS={{ mem0_embedder_dimensions }}
  
  QDRANT_HOST=qdrant
  QDRANT_PORT=6333
  QDRANT_COLLECTION={{ mem0_qdrant_collection }}
  
  NEO4J_URL=bolt://neo4j:7687
  NEO4J_USERNAME=neo4j
  NEO4J_PASSWORD={{ vault_neo4j_password }}
  

  Notice QDRANT_HOST=qdrant, not an IP. When the Mem0 container talks to sibling Compose services, use the service names. Save hostnames and LAN DNS for clients that live outside the Compose network. More on that in a second, because it’s a real gotcha.

  Compose shape

  Just what you need to run Mem0’s memory backend in Docker.

  services:
    qdrant:
      image: qdrant/qdrant:v1.12.6
      ports:
        - "6333:6333"
        - "6334:6334"
      volumes:
        - qdrant_storage:/qdrant/storage
  
    neo4j:
      image: neo4j:5.26
      environment:
        NEO4J_AUTH: "neo4j/${NEO4J_PASSWORD}"
      ports:
        - "7474:7474"
        - "7687:7687"
      volumes:
        - neo4j_data:/data
        - neo4j_logs:/logs
  
  volumes:
    qdrant_storage:
    neo4j_data:
    neo4j_logs:
  

  Pin your image versions. Floating tags make it impossible to tell whether a later failure came from your playbook, your app, or an upstream image that changed under you at 2am. Pinning turns a mystery into a diff.

  The two Ansible flags that matter

  Most of the role can stay boring. Ensure the directory exists, clone the repo, template the .env, start the stack with community.docker.docker_compose_v2. The two details I want you to actually read are force: false on the git task and remove_orphans: true on the compose task.

  force: false protects local edits in the checkout. If the role needs to patch a generated file, make that explicit and safe instead of letting Git clobber the tree.

  remove_orphans: true keeps Compose honest. Say you rip out an old local embedding service and switch to a managed embedding API. Without this, the old container just keeps running forever, quietly, and you’ll swear the new config isn’t taking effect.

  And for restarts, use handlers. Notify a restart handler when the checkout changes or when .env changes. Don’t bounce the stack on every single playbook run. The steady-state run should be quiet.

  Service names vs. hostnames

  From a client running outside the Compose network, point Mem0 at the server’s reachable hostname:

  QDRANT_HOST=mem0.example.test
  NEO4J_URL=bolt://mem0.example.test:7687
  

  From a container inside the same Compose project, use service names:

  QDRANT_HOST=qdrant
  NEO4J_URL=bolt://neo4j:7687
  

  Get this backwards and you burn an afternoon on it…

  Check your work

  After the playbook runs, verify the pieces independently:

  docker compose ps
  curl http://localhost:6333/healthz
  docker compose logs --tail=100 neo4j
  

  For Neo4j, actually test Bolt from the network where your client lives. The browser port on 7474 being reachable does not prove the Bolt endpoint on 7687 is usable. Different port, different assumption, don’t confuse a green browser page for a working client.

  The whole pattern

  Strip away the YAML and it’s simple. Docker Compose owns Qdrant and Neo4j. Ansible owns the checkout, the .env, and the Compose lifecycle. Vault owns the credentials. The client picks service names or external hostnames depending on where it runs.

  That gets you a repeatable Mem0 backend without turning your Ansible role into a second copy of the entire app. Pin your images, guard your secrets with no_log, and never reuse a collection after the vector size changes. Do those three and the rest is boring, which is exactly what you want from infrastructure.

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  Jul 9, 2026 DevOps docker Ansible Mem0 Qdrant Neo4j

  Enjoyed this?

• Geni vs. Goose for lightweight database migrations

  I have a pretty simple rule for database migrations: the tool lets me write SQL, check status, apply the next change, and roll back when I need to. I don’t want much more standing in the way.

  That’s why I like both Geni and Goose.

  Neither one is trying to be a full application framework. They don’t require the rest of the app to be written in the same language as the migration tool. They’re small enough to understand, script, and run from CI, but they have enough features that they don’t feel like a pile of shell scripts once a project grows real production databases.

  The catch is that they’re good at slightly different jobs. I’ve ended up using both in the same migration repo: Goose for PostgreSQL, MySQL, and MariaDB, Geni for LibSQL/Turso. That split has worked well enough that I’d reach for the same shape again, even on projects that aren’t written in Go or Rust.

  The short version

  Use Goose when you want a mature, widely used SQL migration tool for the traditional server databases: PostgreSQL, MySQL, MariaDB, and local SQLite. Broad database matrix, familiar single-file format, status and rollback commands, environment-variable substitution, Go migrations when you need them, and a very plain CLI.

  Use Geni when LibSQL or Turso is part of the system. It’s a standalone tool with first-class LibSQL/Turso support, a simple paired-file model, status/up/down/new commands, and optional schema snapshots. It was built for the space where “SQLite-like, but remote and edge-hosted” starts to expose the rough edges of tools designed around local SQLite files.

  Reach for something heavier when migrations stop being ordered SQL files and turn into a schema-management problem. More on that at the end.

  Why I like both

  The best thing about Geni and Goose is that the migration stays close to the database.

  There’s no ceremony around a model layer. You write SQL. You put it in version control. You run a CLI. The database gets a tracking table. A teammate can review the migration in a pull request without learning your ORM. A deploy script can ask for status before it applies anything. For a lot of small and medium projects, that’s exactly the right amount of machinery.

  It also makes both tools language-independent in practice. Goose is written in Go, Geni in Rust, but a TypeScript, Python, Ruby, or PHP app can use either one just fine. The migration tool runs at the edge of the application. It talks to the database and the filesystem. It doesn’t need to be imported by the app.

  That separation is underrated. A migration directory can outlive a framework rewrite. It can be shared by several services, live in an infrastructure repo, and get run by CI, a deploy script, or a human with a terminal.

  Where Goose shines

  Goose is the boring choice, in the best sense. Its migration format is easy to read:

  -- +goose Up
  -- +goose StatementBegin
  CREATE TABLE users (
    id BIGSERIAL PRIMARY KEY,
    email TEXT NOT NULL UNIQUE
  );
  -- +goose StatementEnd
  
  -- +goose Down
  -- +goose StatementBegin
  DROP TABLE users;
  -- +goose StatementEnd
  

  The forward and rollback changes live in one file, and a review comment can talk about both in one place.

  Goose is also strong when a repo has several conventional engines. It documents support for PostgreSQL, MySQL, MariaDB, SQLite, and more, and the command shape stays mostly the same across them:

  goose -dir migrations/postgres/app/schema postgres "$POSTGRES_APP_DSN" up
  goose -dir migrations/mysql/app/schema mysql "$MYSQL_APP_DSN" up
  

  For Go services there’s another useful escape hatch: Go migrations. Most schema changes should stay in SQL, but occasionally one needs batching, custom validation, or a data rewrite that’s clearer in code. Goose supports that without forcing every migration to become code.

  I reach for Goose first on Postgres/MySQL/MariaDB, on teams that already know SQL migrations, and in deploy scripts that just want status, up, down, and version.

  Where Geni shines

  Geni has a similar direct feel, but its file model is different:

  migrations/
    1782054288_create_users_table.up.sql
    1782054288_create_users_table.down.sql
  

  Forward migration in one file, rollback in another, both plain SQL. Geni can create the pair for you:

  geni new create_users_table
  geni status
  geni up
  

  The more important point is LibSQL/Turso. Turso is not just “a local SQLite file with a different name.” LibSQL is SQLite-compatible, but a remote Turso database brings its own connection shape, auth token, protocol, and operational behavior. That’s exactly where a migration tool either feels native or feels like you’re pushing it through a small opening.

  Geni expects the Turso shape. The documented flow uses DATABASE_URL and DATABASE_TOKEN, it speaks LibSQL, and it can dump a schema.sql snapshot after migrations, which is handy as a review artifact and source-control reference.

  On my own Turso projects I like a thin wrapper around Geni. It resolves secrets from whatever store the project uses, splits a composed DSN into DATABASE_URL and DATABASE_TOKEN, points Geni at the right per-database folder, and masks tokens in debug output. That keeps the human command simple:

  mise run geni dev app status
  mise run geni prod app up
  

  One folder per logical database, paired up/down files, secrets outside the repo, a status check before deploy, and a confirmation prompt before production up or down.

  About Goose and Turso

  This is the one place the recommendation needs nuance. Goose has been growing, and its current docs list a Turso driver. So the answer is no longer “Goose can’t talk to Turso.”

  But “there’s a driver” and “this is the smoothest operational path for my Turso migrations” are not the same claim. The friction I care about isn’t just opening a connection. It’s the whole lifecycle: creating the tracking table, applying safely, transaction behavior, the right remote protocol, auth, status, and a workflow I trust in a short-lived feature branch and in production. For that job I still prefer Geni for LibSQL/Turso, and Goose stays my default for Postgres/MySQL/MariaDB.

  When to use something heavier

  Lightweight SQL migration tools aren’t always enough. I’d start looking at Atlas, Liquibase, Flyway, Sqitch, or an ORM-native system when the problem changes shape:

  • You need schema drift detection across many environments.
  • You want to declare desired schema and have the tool compute the migration.
  • You need online migration planning for large tables.
  • You operate hundreds or thousands of tenant databases.
  • You need policy checks, approval workflows, or compliance artifacts before a migration can merge.

  Those are real needs. They’re also real costs. The heavier tool might be worth it, but I wouldn’t start there by default.

  For most projects, the better starting point is still: write a small SQL migration, review it in git, run status, apply it, and keep the history boring.

  My rule of thumb

  PostgreSQL, MySQL, or MariaDB, I reach for Goose. LibSQL/Turso, I reach for Geni. Not written in Go or Rust? I still consider both, because the CLI boundary matters more than the implementation language. Migrations are an operational concern before they’re an application concern.

  And if the story starts needing drift management, online planning, policy gates, or fleet-wide orchestration, I stop trying to stretch a lightweight tool and pick something heavier on purpose. For everything else, Geni and Goose land right in the sweet spot: small, SQL-first, scriptable, and capable enough to keep schema changes moving without turning migrations into a platform.

  Sources and further reading

  • Geni GitHub repository: https://github.com/emilpriver/geni
  • Turso: “Database migrations with Geni and libSQL”: https://turso.tech/blog/database-migrations-with-geni
  • Goose documentation: https://pressly.github.io/goose/
  • Goose GitHub repository: https://github.com/pressly/goose
  • Atlas guide on Turso connection URLs: https://atlasgo.io/guides/sqlite/turso

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  Jul 8, 2026 DevOps Database migrations Geni Goose Turso Libsql Sql

  Enjoyed this?

• Day 15: The Man Who Synchronized the World

  David Mills, the father of internet time, wrote the protocol that synchronizes every computer on Earth. He did it as a professor at the University of Delaware, on a project he started in the early 1980s and never stopped working on.

  The code lives in your laptop, your phone, your router, every cloud server you’ve ever touched, and every satellite in low Earth orbit. The protocol it implements is called NTP. The reason your computer’s clock is correct, right now, within a few milliseconds of UTC, is that Mills spent forty years of his life making sure it would be.

  He once described the early ARPANET days as a “sandbox” where researchers were simply told to “do good deeds.” Part of the allure of the time-synchronization work, he told The New Yorker in 2022, was that he was just about the only one doing it. He had his own “little fief.”

  For forty years, that is exactly what it was.

  The problem

  The early internet had a clock problem. As soon as there were enough machines on the network that “what time is it?” didn’t have a single answer, somebody was going to have to write a protocol. Each computer had its own oscillator. Each oscillator drifted at its own rate. Two machines that agreed at noon could be tens of seconds apart by midnight.

  Why did this matter? For most things, it didn’t. For some things, it mattered a lot. A file saved on one machine and copied to another could look older than the version it overwrote, confusing every backup tool that assumed time moves forward. Cryptographic handshakes that expire after a few seconds could fail because the two ends disagreed on what “a few seconds ago” meant. Database replicas could apply writes in the wrong order and corrupt their own state. Email between two servers could arrive timestamped before it was sent. Debugging a multi-machine bug meant correlating log entries across clocks that didn’t agree about which event came first.

  Mills decided the actual problem was that there was no protocol for negotiating the truth (in this case, time) across multiple systems. The clock on his desk was wrong. Every other clock was also wrong. The question wasn’t “who has the right time?", it was “given that nobody has the right time and the network adds an unknown delay to every measurement, how does the system converge on a consensus that is closer to UTC than any individual node could achieve alone?"

  His first NTP RFC, RFC 958, was published in September 1985. We now call that protocol NTPv0, or the prototype. In it, Mills nailed down the four-timestamp packet format and the offset/delay math that has been in every revision since. The packet format and the core algorithm haven’t meaningfully changed in forty years. That kind of staying power is rare in any field. In internet infrastructure, where the half-life of a protocol can be measured in single-digit years, it is quite commendable.

  The four timestamps

  NTP’s core insight is that the network delay between client and server can be measured, not just guessed, as long as both sides record their own timestamps for both legs of the conversation. Four timestamps are exchanged in a single round trip:

  Client                            Server
  ──────                            ──────
  T₁ ──── request ───────────────►
                                    T₂
                                    T₃
  T₄ ◄────────────── response ─────
  

  • T₁ — the client sends the request (client clock)
  • T₂ — the server receives it (server clock)
  • T₃ — the server sends the response (server clock)
  • T₄ — the client receives it (client clock)

  Now the client has four numbers. T₁ and T₄ are in the client’s reference frame, T₂ and T₃ are in the server’s. From those four numbers, two things fall out: the round-trip delay (how long the conversation took, minus the time the server spent thinking) and the clock offset (how far the client’s clock is from the server’s). The client now knows how wrong it is, and by how much.

  The math depends on one critical assumption: the network is symmetric. The packet takes the same time to travel in both directions.

  If you’ve been following along in the series, you know there are a lot of ways to measure time. Atomic clocks. GPS receivers. The quartz crystal in your laptop. Radio signals broadcast from government antennas. They don’t all tick at the same rate, and they don’t all agree on what the current time is. How does NTP reconcile across that much varity in time sources?

  The stratum hierarchy

  NTP organizes the world’s clocks into a tree, with depth measured in strata.

  Stratum 0 is the reference. Cesium atomic clocks. Hydrogen masers. GPS receivers. Radio receivers tuned to WWV, DCF77, or MSF. These are not on the network, they’re physical devices wired directly to a small number of computers via PPS pulses on serial ports.

  Stratum 1 is the small group of servers wired directly to Stratum 0. There are perhaps a few thousand of these globally. NIST runs some. Major universities run some. The big internet exchanges run some.

  Stratum 2 servers sync with Stratum 1, Stratum 3 with Stratum 2, and so on down to Stratum 15. Stratum 16 means “unsynchronized, do not trust.”

  A typical Linux laptop syncs against Stratum 2 or 3 servers. A typical cloud VM syncs against its provider’s internal Stratum 1 fleet. Your phone syncs against whatever its carrier provides. The whole tree is held together by NTP itself, recursively.

  The genius of the design is that there is no central authority. Mills did not own the protocol. There is no “official NTP server.” Anyone can run a Stratum 1 with the right hardware, and anyone can run a Stratum 2+ by syncing with a few Stratum 1s of their choice. The largest public pool, pool.ntp.org, is a volunteer effort started in 2003 by Adrian von Bidder. It currently aggregates a few thousand donated stratum-2 servers worldwide and serves several billion requests per day. Nobody is in charge of it. It just works.

  The slew, not the step

  There are three different times to keep track of on every synced computer. The reference time is what UTC says, the truth NTP is chasing. The tick rate is how fast the computer’s oscillator pulses. It’s supposed to produce one second of clock time per real second, but always drifts a little. The system clock is what gets reported when an application asks for the current time. Synchronizing means closing the gap between the system clock and the reference time without breaking anything that depends on the system clock being well-behaved.

  NTP’s primary tool for that is the slew: it adjusts the tick rate, making each tick slightly longer or shorter than nominal, so the system clock drifts into alignment on its own. The alternative would be to jump the clock forward or backward by the full offset (a step), which is fast but can produce duplicate keys in a database, expire valid TLS sessions, or cause a logging system to mis-order events.

  Mills designed ntpd to slew conservatively. A 200ms gap might take several minutes to close, and corrections larger than about 128ms would get stepped because slewing them gradually was prohibitively slow. That trade-off worked for the always-on Unix workstations of the 1980s and 90s. It works less well for the modern reality of laptops that suspend for hours and resume with a clock that hasn’t been touched since last Tuesday, or cloud VMs that get migrated between hosts. Modern variants like chrony slew more aggressively for exactly that reason. When you open your laptop lid, you want the clock right now, not after fifteen minutes of imperceptible easing.

  The legacy

  In a sense, NTP is the thing that made the modern internet possible.

  Without well-synchronized clocks, you cannot have SSL certs. The browser needs to know when the cert expires, and if its clock is off by more than a few minutes, the encryption breaks. The same goes for databases. No matter the type, NoSQL or otherwise, they all depend on a clock to record when an operation took place.

  Without NTP, cell towers wouldn’t agree on when to hand off a call. Financial transactions wouldn’t be enforceable. And all those log files you’ll totally read one day wouldn’t make any sense. NTP is foundational to all of it. It runs as a daemon on every machine, the ones you stare at all day, the ones you don’t see, and the ones you don’t care about.

  We remember Mills as the internet’s “Father Time” and the man who synchronized the world. Neither is a metaphor.

  ---

  Sources

  • In Memoriam: David Mills (UDaily, March 2024) — University of Delaware’s obituary; biographical detail, career timeline.
  • David L. Mills — Wikipedia — congenital glaucoma from birth, vision worsening from ~2012, fully blind by 2022; UDel professor 1986–2008.
  • David Mills, the internet’s Father Time, dies at 85 — The Register — death date (Jan 17, 2024), age 85.
  • RFC 958 — Network Time Protocol (September 1985) — the original NTPv0 specification.
  • Network Time Protocol — Wikipedia — version lineage: RFC 958 (v0, 1985), RFC 1059 (v1, 1988), RFC 1119 (v2, 1989), RFC 1305 (v3, 1992), RFC 5905 (v4, 2010), RFC 8915 (NTS, 2020).
  • NTP pool — Wikipedia — Adrian von Bidder started the pool in January 2003; Ask Bjørn Hansen has run it since 2005.
  • MiFID II RTS 25 clock synchronization (Meinberg) — 100µs requirement for high-frequency trading at sub-1ms gateway latency.
  • A Brief History of NTP Time: Confessions of an Internet Timekeeper (Mills, PDF) — Mills' own history of NTP.
  • The Thorny Problem of Keeping the Internet’s Time (The New Yorker, September 2022) — Nate Hopper’s profile of David Mills and the fragile state of NTP maintenance.

  ---

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  ---

  Tomorrow: ISO 8601, the format wars, the carnage of MM/DD vs DD/MM, and why 2026-06-07T14:30:00Z won.

  Jun 7, 2026 DevOps Programming 30daysoftime Internet history Protocols

  Enjoyed this?

• Running Terraform in Your Existing CI Pipeline

  The previous post made the case that HCP Terraform’s per-resource pricing model has gotten structurally hostile to modern infrastructure patterns. (The earlier posts in this series argued that OpenTofu is the no-regrets default for new infrastructure, and walked through when to skip Terraform entirely in favor of cloud-native tooling.) The natural follow-up: if you don’t want to pay the commercial orchestration tax, can you run Terraform or OpenTofu properly inside your existing CI/CD? The answer is yes, but the gap between “it works” and “it works well” requires some deliberate architecture. This post is about how to close that gap.

  There are three pieces: where the state lives, how the pipeline authenticates to your cloud, and what handles the orchestration concerns (locking, PR commentary, drift detection) that TACOs sell as their core value. Each one has a sensible 2026 answer that doesn’t involve paying anyone.

  State Management in GitLab

  If you’re on GitLab, the entire state-management problem is solved natively. GitLab ships an HTTP backend for Terraform and OpenTofu state on every tier including Free. You don’t need to provision an S3 bucket. You don’t need a DynamoDB lock table. You don’t need to figure out KMS. The state file is encrypted in transit and at rest, locking is handled by GitLab’s project-scoped role-based access control, and there’s a native UI under Operate > Terraform states that shows you version history and lets you roll back if something corrupts.

  The pipeline pattern uses a backend block like this:

  terraform {
    backend "http" {}
  }
  

  Combined with the gitlab-tofu (or gitlab-terraform) CLI wrapper in your .gitlab-ci.yml, which dynamically configures the HTTP backend at runtime using the per-job ${CI_JOB_TOKEN}. The wrapper avoids passing backend credentials via -backend-config arguments (which cache in pipeline logs) and handles authentication automatically.

  The RBAC story is also worth pointing out, because it’s exactly what TACOs charge thousands of dollars to replicate: the GitLab project’s role model becomes the IaC permissions model. Developers can read state and run tofu plan -lock=false. Maintainers and Owners can lock state and run tofu apply. The audit log is the GitLab activity feed. No additional configuration, no additional vendor.

  For GitLab shops, this is the single highest-leverage decision in the entire IaC stack: stop paying for state management when your VCS gives it to you for free.

  Secretless Authentication on GitHub Actions

  On GitHub Actions, the equivalent problem is authentication. Historically, every Terraform-on-Actions tutorial told you to put a long-lived AWS access key in GitHub Secrets. That’s the worst possible pattern. A compromised repository, a malicious third-party action, or a leaked log line gives the attacker permanent, unscoped access to your cloud.

  The 2026 answer is OpenID Connect with cloud-side trust policies. The pipeline gets ephemeral, short-lived credentials per job, scoped to the specific repository and branch that initiated the run. Nothing persists.

  For AWS: configure GitHub’s OIDC provider (token.actions.githubusercontent.com) as an identity provider in IAM. Create an IAM role with a trust policy that conditionally allows assumption based on JWT claims like sub (subject) and aud (audience). The workflow uses aws-actions/configure-aws-credentials to exchange a GitHub-issued JWT for temporary AWS credentials via AssumeRoleWithWebIdentity. The trust policy can be scoped to a specific repository, a specific branch (main only), or even a specific environment (production).

  For GCP: the equivalent is Workload Identity Federation. You create a Workload Identity Pool that trusts GitHub’s OIDC provider, configure attribute mapping that validates the token claims (e.g., requiring assertion.repository == "company/infra-prod"), and grant the pool’s principal the ability to impersonate a specific GCP service account. The official google-github-actions/auth action handles the token exchange.

  Both patterns produce credentials that expire when the job ends, can’t be exfiltrated to long-term storage, and leave a clean audit trail in your cloud’s IAM logs. There is no good reason to use long-lived cloud credentials in CI in 2026.

  What CI Doesn’t Give You for Free

  Native CI/CD solves the cost problem. It does not, by itself, solve every operational problem that commercial TACOs address. There are three real gaps worth knowing about:

  State locking and race conditions. Standard CI/CD systems are designed for concurrent runs because that’s what application code wants. Infrastructure code wants the opposite. If two PRs merge at the same time and both trigger tofu apply, you have two concurrent processes racing to mutate the same state file. With GitLab’s HTTP backend or an external lock backend like DynamoDB, the lock will prevent corruption but one job will fail with a confusing error. Without it, you get state corruption. You need some queuing logic, either custom or via an orchestrator.

  PR plan commentary. TACOs post the output of terraform plan directly into the PR so reviewers can see what’s about to change before merging. In raw CI/CD this requires a third-party action (terraform-plan-pr-commenter and similar), parsing of the CLI output, handling of PR comment character limits, and securely passing the binary plan file as a workflow artifact from the plan stage to the apply stage. None of this is hard, but it’s a real chunk of YAML you have to maintain.

  Cost estimation. TACOs include built-in cost estimation on every plan. Adding this to your own pipeline means picking up a third-party FinOps or IaC cost-analysis tool (there are several worth comparing), running it against your plan output, parsing the JSON, comparing against budget thresholds, and posting deltas into PRs. None of that is hard, but it’s another bit of integration to own.

  You can build all of this yourself. Plenty of teams do. The question is whether maintaining the bash and YAML is cheaper than using an open-source orchestrator designed for exactly this problem.

  Open-Source Tools to Layer In

  None of these are drop-in replacements for HCP Terraform or Spacelift. They solve specific problems CI/CD doesn’t handle on its own, and you compose them based on what’s actually missing from your setup.

  Tool	What It Solves	Best For
  Atlantis	PR-based workflow automation, plan/apply via PR comments, PR-level locking	Teams that want TACO-style PR workflow but on their own server
  Digger	Same PR workflow + locking, but the IaC actually runs inside your existing CI runners	Teams with secretless OIDC pipelines who don’t want to maintain a separate server
  Terramate	Multi-stack monorepo orchestration, git-based change detection, parallel execution	Teams whose Terraform has grown into hundreds of stacks

  Atlantis is the original PR-automation tool, accepted into the CNCF Sandbox in June 2024. It deploys as a Golang binary or container, listens for VCS webhooks, and runs Terraform on its own server. The architecture is showing its age. It’s stateful, single-threaded, granting it persistent privileged cloud access creates a high-value target, and the maintenance velocity has slowed. If you’re already running it and it works, fine. For new setups, the case for Digger is usually stronger.

  Digger is a thinner orchestration layer. It coordinates Terraform jobs but runs them inside your existing GitHub Actions or GitLab runners, using OIDC for cloud authentication. The orchestrator backend itself never sees state, plan output, or cloud credentials. This is the right pattern if you’ve already built secretless OIDC pipelines and want PR-workflow automation without introducing another long-lived privileged component.

  Terramate solves a different problem: scaling Terraform across many stacks in a monorepo. It parses your Git history to determine which stacks changed, then runs plan and apply only on those, in parallel. For a repo with 200 stacks and a PR that touches one, you skip the 199 unnecessary plans. It also has a code-generation system that reduces HCL boilerplate. Terramate Cloud adds dashboards and drift detection without requiring access to cloud credentials. If your IaC repo has gotten unwieldy, Terramate is the tool for it. It’s a complement to Atlantis or Digger, not a substitute.

  The Recommendation

  The full picture for escaping commercial TACOs in 2026:

  • State: GitLab’s native HTTP backend if you’re on GitLab. S3 + DynamoDB (or OpenTofu state encryption + S3) if you’re on GitHub.
  • Auth: OIDC for AWS, Workload Identity Federation for GCP. Never long-lived secrets.
  • PR workflow: Digger if you want PR automation that runs inside your existing CI. Atlantis if you’re already running it. Skip this layer entirely if your team is small enough that PRs serialize naturally.
  • Stack management: Terramate if you have a large monorepo. Otherwise, not needed.
  • Cost estimation: Pick a third-party FinOps or IaC cost-analysis tool and wire it into your plan stage.

  The total monetary cost of this stack is the price of your existing CI/CD minutes, which you’re already paying. The total time cost is on the order of one to two weeks of platform-engineering time to set up properly, plus ongoing maintenance proportional to how much you customize.

  For most organizations under 300 engineers, that’s cheaper than HCP Terraform Standard or Premium. For larger organizations, the calculus depends on how much custom platform work you’re willing to absorb versus how much you want a vendor to handle.

  This wraps the series. Four posts in: OpenTofu as the no-regrets default engine, the scenarios where cloud-native tools beat Terraform entirely, the HCP pricing model that’s pushing teams to find alternatives, and now the CI-native path that lets you skip commercial orchestration. The throughline is the same as every post in this blog about platform engineering: there isn’t a single open-source tool that drops in for HCP Terraform or Spacelift. You’re assembling a stack from focused pieces (state backend + auth + maybe PR automation + maybe stack management), accepting some operational tax in exchange for not paying the SaaS premium. For most teams under 300 engineers, that tradeoff is worth it.

  Sources

  • GitLab-managed Terraform/OpenTofu state — GitLab Docs
  • How to Manage Terraform State with GitLab — Spacelift
  • Using Terraform to connect GitHub Actions and AWS with OIDC — Thiago Salvatore
  • Deploy Terraform resources to AWS using GitHub Actions via OIDC
  • Configure Workload Identity Federation with deployment pipelines — GCP Docs
  • Terraform Deployment to GCP Using GitHub Actions and Workload Identity Federation
  • Atlantis vs. Terraform Cloud / Terraform Enterprise — Spacelift
  • Digger and Atlantis: key differences
  • Terramate: Turn Your IaC into a Lightning-Fast Platform
  • How to Implement Cost Checks in Terraform CI/CD Pipelines — OneUptime
  • Terraform Plan PR Commenter (GitHub Action)

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 23, 2026 DevOps Github Infrastructure Opentofu Terraform Cicd Gitlab

  Enjoyed this?

• HCP Terraform's Per-Resource Pricing Is a Trap

  The first post in this series argued OpenTofu is the no-regrets default for new infrastructure. The previous post mapped out when to skip cloud-agnostic IaC entirely. This one is about what happens to organizations that picked Terraform years ago, built their orchestration around HCP Terraform (formerly Terraform Cloud), and are now opening renewal quotes that have doubled or tripled year-over-year.

  The short version: HashiCorp’s 2024 pivot to Resource Under Management (RUM) billing penalizes the architectural patterns the DevOps community spent a decade adopting. Modular code, ephemeral environments, and granular resources are all things you were supposed to do with Terraform. They now cost real money under the new pricing model. And the legacy free tier that grandfathered teams into a more sustainable cost structure hit end-of-life on March 31, 2026.

  If you’re still on HCP Terraform in 2026, you need to understand the math.

  How the New Pricing Works

  The 2024 RUM model bills based on the peak number of resources tracked in your terraform.tfstate files, measured hourly. The Free tier covers up to 500 resources with a single concurrent run. Above that, you’re on Pay-As-You-Go tiers:

  Tier	Per-resource cost	Concurrency	What you get
  Free	$0 (first 500)	1	Basic VCS, remote state
  Essentials	~$0.10/month	1	Basic provisioning, no SSO
  Standard	~$0.47/month	3	Up to 5 policy checks, cost estimation, limited RBAC
  Premium	~$0.99/month	10	Full governance, unlimited policies, SSO, audit logs

  On paper, $0.47 per resource per month looks negligible. The math goes sideways quickly because of three things.

  Why “Resources” Is a Footgun

  1. Granularity inflation. A single logical Terraform module produces dozens of underlying resources. An AWS VPC module isn’t one billable resource. It’s the VPC plus every subnet, every route table, every route table association, every IAM policy attachment, every security group rule, every DNS record. A widely-shared Reddit post by user notoriousbpg describes a team whose HCP Terraform bill was about to jump from $0 to over $15,000 a year, because 80% of the resources under management were GraphQL operation mappings to data sources, while the actual AWS infrastructure they cared about cost only $8,000. They were paying more for orchestration than for the infrastructure being orchestrated.

  2. Idle workspaces. RUM billing doesn’t distinguish between active and inactive infrastructure. The proof-of-concept workspace someone spun up last quarter and never destroyed is still on your bill. The staging environment that was deprecated in favor of ephemeral environments is still on your bill. Industry telemetry suggests 30–40% of an average organization’s RUM cost is for completely idle infrastructure nobody has bothered to terraform destroy.

  3. Hourly peak billing on ephemeral resources. HCP Terraform bills based on peak hourly resource count. If your integration test pipeline spins up infrastructure that exists for five minutes and is then torn down, you’re billed as if it existed for the full hour. This is a direct tax on the modern GitOps workflow patterns Terraform itself spent years promoting. The more ephemeral environments you use, the more punitive the billing becomes.

  The compounding effect is severe. Another account describes cloning a 600-resource production workspace to create a pre-production environment. The resource count doubles to 1,200. The annual cost goes from ~$122 to ~$858, a 7x increase for what’s architecturally a trivial change. Multiply that across every environment, every test fixture, every modular abstraction, and the renewal quote stops being theoretical.

  The Alternative TACOs

  Once organizations work through the RUM math and realize the bill is structurally unsustainable, the obvious move is to look at alternative orchestration platforms. The three serious contenders, with very different pricing models:

  Platform	Pricing Model	Entry / Mid-Tier Cost	What It Does Differently
  Spacelift	Resources + runs + seats	$1,500–$3,500/mo	Multi-tool (Pulumi, K8s manifests, Terragrunt). OPA policies. Custom runners. Cross-stack dependencies.
  env0	Per-user	~$50/user/mo	Predictable user-based pricing. Strong TTL/ephemeral environment story.
  Scalr	Per-user	~$50/user/mo	Familiar Terraform Cloud UI replacement. Lower entry price than Spacelift.
  ControlMonkey	Fixed plan (users + assets)	$800/mo (Startup: 10 users, 5k assets, 500 deploys)	One-click Terraform import, automatic drift remediation, daily cloud-config backups, built-in compliance.

  Spacelift is the choice for complex platform engineering teams. It supports Terraform, OpenTofu, Pulumi, Terragrunt, and Kubernetes manifests in one platform, handles cross-stack dependencies, and bakes OPA policy enforcement into the runtime. The catch is that its pricing still factors in managed resources, so the bill scales with infrastructure size, just less aggressively than HCP.

  env0 and Scalr both flipped to user-based pricing specifically as a response to RUM. A 15-engineer team managing 3,000 resources pays roughly the same on env0 as a 15-engineer team managing 500. The price is bounded by headcount, not infrastructure complexity. This is the right model for teams whose resource counts have ballooned because they followed the “do everything as code” advice and now have hundreds of granular Terraform-managed entities they don’t want to pay per-unit fees on.

  When to Pay for Any Commercial TACO

  The harder question is whether the commercial orchestration layer is worth its multi-thousand-dollar monthly bill at all. The features TACOs sell (state locking, PR-level plan output, policy enforcement, drift detection, audit logging) are all things you can build into your own CI/CD pipeline. The question is whether building and maintaining that pipeline is cheaper than paying the SaaS fee.

  For most teams under ~50 engineers, the answer is no. The SaaS fee buys polish and convenience, but the underlying capabilities are available in GitLab’s native state management or in GitHub Actions with the right open-source orchestrator. For larger teams, the calculus shifts: the cost of a dedicated platform engineer maintaining a custom CI/CD pipeline starts to approach the cost of a commercial license, and the operational predictability of a managed platform becomes valuable.

  But the days of HCP Terraform being the obvious default for everyone above the free tier are over. The RUM model made the math too punishing for too many real-world architectures.

  The next and final post in this series gets into the actual mechanics of running Terraform/OpenTofu inside your existing CI/CD: GitLab’s native state backend, GitHub Actions with OIDC/Workload Identity Federation for secretless deploys, and the open-source orchestrators (Atlantis, Digger, Terramate) that close the gap between raw YAML and a real platform.

  Sources

  • Terraform Cloud / Enterprise Pricing — Tiers Overview 2026 — Spacelift
  • Terraform Cloud Pricing Guide: Tiers, Costs, and Optimization Tips — ControlMonkey
  • 10 Best Terraform Cloud Alternatives & Competitors In 2026 — ControlMonkey
  • Continuing HCP Terraform’s enhanced free tier experience — HashiCorp
  • Terraform Cloud Pricing Explained: Resource-Based Guide (2026) — Firefly
  • Spacelift Software Pricing & Plans 2026 — Vendr
  • Terraform Cloud Pricing: A Complete Guide (2026) — env0

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 22, 2026 DevOps Infrastructure Terraform Pricing Hashicorp

  Enjoyed this?

• When You Should Skip Terraform Entirely

  The last post in this series made the case that OpenTofu is the no-regrets default for new infrastructure projects. That’s true for the broad case of cloud-agnostic or multi-cloud setups where HCL parity, provider breadth, and a Linux Foundation governance model matter.

  It’s also not the whole story. There are at least three common scenarios where the right answer in 2026 isn’t Terraform or OpenTofu. It’s the cloud-native tool the hyperscaler ships with its platform. AWS has CloudFormation and the CDK. Azure has Bicep. GCP has Config Connector. Each one is technically superior to Terraform inside its own ecosystem, and each one removes a category of operational pain that Terraform inflicts.

  If you reflexively reach for Terraform every time, you’re probably overpaying in complexity for a multi-cloud option you’ll never exercise.

  The Small AWS-Native Startup: Use CDK

  If your engineering team is small, you’re shipping a SaaS product, and you’re 100% on AWS, you should probably ignore Terraform entirely. The right tool is the AWS Cloud Development Kit, layered on top of CloudFormation.

  The fundamental win is that CloudFormation eliminates state management. There is no terraform.tfstate file. No S3 bucket to provision. No DynamoDB lock table. No state-encryption configuration to figure out. The state lives in the AWS control plane, AWS manages locking and consistency, and your CI pipeline doesn’t need to know about any of that. For a small team, that’s a meaningful operational tax you don’t pay.

  The CDK is the part that makes this pleasant. It lets you define infrastructure in TypeScript, Python, Java, C#, or Go; so the languages your application engineers already know. There’s no HCL learning curve, no Sentinel policy DSL, no jq-in-bash to manipulate plan output. You write code, the CDK synthesizes CloudFormation templates, CloudFormation provisions the infrastructure.

  The objection people raise is “what if you go multi-cloud later?” In practice, most SaaS startups don’t. They get acquired, they pivot, or they grow large enough to have a dedicated platform team that does the migration deliberately. Optimizing for a hypothetical multi-cloud future that 90% of teams will never need is the textbook definition of premature abstraction. If you’re an AWS-native startup with fewer than 50 engineers and no concrete plans to leave AWS, the cost of running Terraform-as-multi-cloud-insurance is higher than the cost of a future migration that probably won’t happen.

  The Azure Enterprise: Bicep, Unless You Need More

  For organizations heavily invested in Microsoft’s stack, so Azure for compute, Azure DevOps for CI/CD and EntraID for identity; Bicep is the technically correct choice for most workloads.

  Bicep is Azure’s domain-specific language for infrastructure, designed as a replacement for the verbose ARM JSON templates everyone hated. Like CloudFormation, it’s stateless. You submit a desired-state Bicep file to the ARM control plane and ARM reconciles. No state file, no remote backend, no risk of corruption. Authentication is whatever RBAC permissions the deploying identity already has, with no provider credential configuration required.

  Bicep also gets day-zero feature support for new Azure capabilities. When Microsoft ships a new service, you can use it in Bicep the same day. The Terraform AzureRM provider has historically lagged by weeks or months, occasionally longer.

  The catch is scope. Bicep manages Azure. That’s the entire surface area. Larger organizations tend to need management of things outside Azure too: GitHub repositories and branch protection, EntraID groups, Datadog monitors, PagerDuty escalation policies, whatever SaaS services your platform touches. Bicep has no answer for any of that.

  That leaves two paths. The first is a hybrid: Bicep for Azure, separate tools for everything else, accept the cost of context-switching and the inability to express cross-domain dependencies in a single deployment. The second is Terraform or OpenTofu for everything, accepting the heavier operational tax of stateful IaC, in exchange for one tool that can do all of it. Neither is wrong; they’re different tradeoffs against the same constraint.

  The decision rule: if you’re managing only Azure resources, use Bicep. If you have cross-domain provisioning needs and you’d rather not maintain two parallel IaC stacks, Terraform (or OpenTofu) earns its keep.

  The GCP/Kubernetes Shop: Hybrid by Design

  For organizations heavily committed to Google Cloud and running most workloads on GKE, the right architecture isn’t either/or. It’s a hybrid that uses Terraform for the foundation and Config Connector for the application layer.

  Config Connector is a GCP-shipped Kubernetes add-on. It lets you manage GCP resources — Cloud SQL instances, Pub/Sub topics, storage buckets, service accounts — as standard Kubernetes Custom Resources. You write a YAML manifest, you kubectl apply, and a controller in the cluster reconciles the real-world GCP resource to match.

  The differentiator is continuous reconciliation. Terraform is episodic: it checks state at plan and apply time, and the rest of the time your infrastructure is unmonitored. If someone clicks around in the GCP console and manually changes a setting, Terraform won’t notice until the next pipeline run. Config Connector runs a controller loop that polls continuously. Manual drift gets reverted in real time.

  The right architectural boundary:

  • Platform layer (Terraform/OpenTofu): VPCs, subnets, foundational IAM, the GKE clusters themselves. These are slow-moving, security-critical, and you want a deliberate pipeline approval flow for them.
  • Application layer (Config Connector): Application-specific buckets, databases, service accounts, Pub/Sub topics. Application teams own these via the same YAML manifests they use for their pods, with the same GitOps workflow they already understand.

  This pattern gives platform teams strict guardrails on the foundation while letting application developers self-serve the resources their services need, without filing a Terraform PR every time they want a new bucket.

  The Decision Rule

  The honest version of all of this: Terraform/OpenTofu is the right answer when you need cross-domain or cross-cloud governance. For everything else, the cloud-native tool is usually less work, more current with the platform, and avoids the operational tax of state management.

  A reasonable decision tree:

  • Single-cloud, small team, AWS: AWS CDK + CloudFormation.
  • Single-cloud, single-domain, Azure: Bicep.
  • GCP with heavy Kubernetes use: Hybrid — Terraform/OpenTofu for foundation, Config Connector for application resources.
  • Multi-cloud, or cross-domain platform engineering (GitHub + cloud + identity + monitoring): OpenTofu.

  The mistake I think most teams are making is to default to Terraform because it’s the tool the senior engineer learned in their last job. The platform-engineering pitch … “we’ll standardize on Terraform so we can move to any cloud later” is correct in theory but almost never exercised in practice. If your team isn’t using the cross-cloud capability today, you’re paying for an insurance policy you’ll never collect on.

  Next post in this series digs into the other side of that calculation: what HCP Terraform actually costs in 2026, and why even teams that need cloud-agnostic IaC are looking for the exit from the commercial orchestration platforms.

  Sources

  • Bicep Vs Terraform: Choosing The Best IaC Tool For Azure — Synextra
  • Terraform vs Bicep vs ARM Templates 2026 Compared — Exodata
  • Comparing Terraform and Bicep — Microsoft Learn
  • Terraform vs Bicep vs ARM: Lessons from the Trenches — Vaibhav Gujral
  • How to Use the GCP Config Connector with Terraform — OneUptime
  • How Config Connector compares for infrastructure management — Google Cloud Blog
  • Are Terraform’s days numbered? — Alistair Grew

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 21, 2026 DevOps Infrastructure Terraform Aws Azure Gcp

  Enjoyed this?

• OpenTofu Is the No-Regrets Default for 2026 Infrastructure

  Hashicorp’s adoption of the Business Source License in late 2023 was a defensive business decision. Companies like Spacelift, env0, and Scalr were building paid commercial platforms on top of MPL-licensed Terraform, capturing significant revenue from an ecosystem Hashicorp was largely funding. The same pattern played out with Redis Labs facing AWS ElastiCache, Elastic facing Amazon OpenSearch, and MongoDB facing the cloud hyperscalers before its move to the SSPL. The BSL is a rational corporate play: keep the core open enough to preserve mindshare, restrict the terms enough that pure resellers can’t extract value without engaging commercially. From the standpoint of a publicly traded company with a board to answer to, it made sense.

  But it also broke a tacit contract. Hashicorp had spent a decade positioning Terraform as infrastructure’s git. Neutral, ubiquitous, irreplaceable. A license that lets a single vendor change the terms when the shareholder math demands it is not neutral, and a large portion of the community decided they weren’t comfortable with that risk. The Linux Foundation forked the last MPL-licensed Terraform release and shipped it as OpenTofu. Two years later, OpenTofu has crossed 10 million downloads, holds HCL parity with Terraform, supports the same provider ecosystem (AWS, Azure, GCP, Kubernetes, everything), and ships features Terraform itself doesn’t have.

  For greenfield infrastructure in 2026, OpenTofu is the no-regrets default. For existing Terraform codebases, the migration is mostly a binary swap. The reasons to still pay for Terraform are mostly inertia. Let me explain.

  The Migration Is Mostly Free

  The technical case for “stay on Terraform” essentially doesn’t exist. OpenTofu reads the same HCL. It produces the same execution plans. It maintains the same state file format. It interfaces with the same providers, including the ones Hashicorp wrote, because the provider API was never the part Hashicorp tried to lock down.

  To migrate a non-trivial Terraform codebase to OpenTofu, you do roughly this:

  1. Swap terraform for tofu in your CI binary install step.
  2. Update any pipeline scripts that hardcoded the binary name.
  3. Run tofu init -migrate-state once.
  4. Run tofu plan and confirm it produces an empty diff against the existing state.

  There are edge cases, like modules pinned to specific Terraform-version constraints or providers that gated features on the Hashicorp-only registry. But for the vast majority of codebases, the migration is a one-afternoon job, including the PR review and the team announcement.

  What you get in exchange is governance under the Linux Foundation, an active multi-vendor contributor base, no future license surprises, and a really nice to have feature not in Terraform currently: native state encryption.

  State Encryption Is the Real Reason

  Terraform state files have a property nobody enjoys discussing. They contain everything sensitive about your infrastructure, and they store it in plaintext.

  That’s not a misconfiguration. That’s the design. The terraform.tfstate JSON file holds resource IDs, ARNs, network topology, credentials surfaced as outputs, RDS connection strings, and any sensitive value a module decided to track. When you use S3 or Azure Blob as a remote backend, you get encryption at rest, meaning the cloud provider’s storage layer is encrypted. The state itself, the thing your CI pipeline downloads and uploads on every run, is plaintext JSON. Anyone with read access to the bucket (your CI runner, your laptop, anything assuming the role) gets the cleartext.

  OpenTofu solves this with native, client-side state encryption introduced as a first-class feature. The state is encrypted by the engine before it leaves the machine. The remote backend never sees plaintext at all. The configuration looks like this:

  terraform {
    encryption {
      key_provider "aws_kms" "primary" {
        kms_key_id = "arn:aws:kms:us-east-1:..."
        region     = "us-east-1"
        key_spec   = "AES_256"
      }
  
      method "aes_gcm" "primary" {
        keys = key_provider.aws_kms.primary
      }
  
      state {
        method = method.aes_gcm.primary
      }
  
      plan {
        method = method.aes_gcm.primary
      }
    }
  }
  

  Three pieces. A key provider (AWS KMS, GCP KMS, OpenBao, or a local passphrase via pbkdf2), an encryption method (AES-GCM is the standard pick), and explicit targets for state, plan, or both.

  The migration path from existing plaintext state requires a fallback block. OpenTofu refuses to read plaintext once encryption is enabled, which is the right default, but it means you need to tell it “this one time, read the legacy state and re-encrypt it.” After one successful apply, you remove the fallback and you’re done.

  Terraform doesn’t have this. Hashicorp’s official answer is still “use a backend that encrypts at rest and audit your IAM policies carefully.” Which is fine, until your CI logs the state diff into a third-party observability tool, or someone runs terraform show over a Slack screenshare, or an attacker gets a transient role to your backend bucket. The threat model OpenTofu’s encryption closes is the threat model that matters.

  The AI Wrinkle

  There’s a meta-argument unfolding alongside all of this: AI is making the choice of execution engine less important.

  Industry telemetry says 71% of cloud teams have seen an exponential increase in IaC volume from generative AI. The thing AI is generating, in most cases, is HCL, which is the lingua franca for both Terraform and OpenTofu. As the volume of AI-authored infrastructure grows, the role of HCL shifts from “the language engineers write” toward “the intermediate representation an agent emits.” Manual HCL authoring is on track to become a niche skill in the same way hand-tuning compiler output is a niche skill.

  In that world, the execution engine is plumbing. The valuable layer is everything around it: state management, drift detection, policy enforcement, cost guardrails, audit trails. Which is exactly the layer where vendor lock-in does the most damage and where open governance matters most. The AI argument doesn’t undercut the OpenTofu case. It reinforces it.

  What To Do

  If you’re starting a new infrastructure project, use OpenTofu. There is no good reason to start a 2026 greenfield project on a single-vendor BSL-licensed engine when the Linux Foundation-governed open-source alternative is right there, with full HCL parity, the same provider ecosystem, and features Terraform doesn’t have.

  If you have an existing Terraform codebase, schedule the migration. It’s a one-afternoon job per repo. Get state encryption while you’re at it.

  If you’re heavily integrated with HCP Terraform, this is the harder case. The migration off the proprietary HCP features (Sentinel policies, the registry, the integrated dashboards) is real work. But it’s also the case where you have the most to lose. HCP Terraform’s pricing model has gotten aggressively worse, and OpenTofu’s existence means you have actual leverage in the next renewal conversation. The next post in this series digs into exactly what HCP pricing looks like in 2026 and why so many organizations are getting six-figure renewal quotes for infrastructure they were paying $20K for two years ago.

  This is the first of a four-part series on the 2026 IaC landscape. Up next: cloud-native vs cloud-agnostic tooling, and when to use AWS CDK, Bicep, or Config Connector instead of Terraform/OpenTofu at all.

  Sources

  • 2026 IaC Predictions: What Cloud Leaders Must Prepare For ControlMonkey
  • Terraform vs OpenTofu in 2026: Should You Stay or Switch?
  • Terraform or OpenTofu in 2026? Here’s What I Actually Think Jae Wook Kim
  • OpenTofu vs Terraform in 2026: Is the Fork Finally Worth It? Mechcloud Academy
  • OpenTofu vs. Terraform: A Practical Guide for Enterprise Infrastructure Teams env0
  • State and Plan Encryption OpenTofu docs
  • How to Use OpenTofu State Encryption OneUptime
  • State Encryption with OpenTofu Ned in the Cloud

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 20, 2026 DevOps Infrastructure Opentofu Terraform

  Enjoyed this?

• A Dotfiles Manager That Snapshots Every Change

  Managing dotfiles in 2026 is a solved problem in the same way that managing your own backups is a solved problem: there are five tools for it, all of them work, all of them require you to set up some plumbing first, and once you’re set up you still don’t have a great answer to “I just broke my shell config, get me back to yesterday.”

  The conventional answer is some combination of: a git repo for your ~/.zshrc and friends, a symlink script (or stow, or chezmoi, or yadm), and the discipline to remember to commit after every change. The setup is a one-time hassle. The “wait, what did I change?” recovery story is not great. And if you want to sync across machines, you’ve now got opinions about remote repos, SSH keys on a fresh box, and which order things have to happen in.

  I wanted something different, so not a configuration framework, but a record of every change to the files I care about, in a place I can roll back from, with the lowest possible setup cost.

  That’s what dfm is.

  What It Does

  dfm is a single static Go binary. You point it at the files you want to track (~/.zshrc, anything under ~/.config/, whatever), and every time one of them changes it takes a content-addressed snapshot. The snapshots live on disk in ~/.local/share/dotfiles/backups/. A small state database (SQLite locally, or libSQL via Turso if you want cross-machine sync) records which file maps to which snapshot at which point in time.

  You can roll back. You can diff against an old snapshot. You can see when you last touched a file. And because every snapshot is content-addressed, you never re-store the same bytes twice — switching themes in ~/.zshrc ten times costs the size of two configs, not ten.

  The other half is the backup story. dfm init walks you through cloning (or creating, via gh) a private GitHub repo that mirrors your tracked files plus their history. The point isn’t to make you adopt a new git workflow. It’s that pulling your config onto a fresh machine should be one command, and recovering from rm -rf should never have a “well, hopefully my last commit was recent” caveat.

  Why Setup Is the Hard Part

  The reason people don’t audit their dotfiles is the same reason people don’t back up their laptops: the setup is annoying, and the payoff is theoretical until it isn’t.

  dfm init is a six-step interactive wizard. It detects a TURSO_DATABASE_URL env var if you’ve got one, offers sensible defaults for everything else, lets you opt in to tracking ~/.zshrc immediately, and writes a single config file with the right permissions. Re-run it on an existing config and it pre-fills every prompt with your current value, so the cost of changing your mind later is also low. --yes accepts every default for scripted setup.

  If that sounds boring, that’s the point. Boring is what makes a tool actually get used.

  The AI Bit

  There’s an optional AI integration. dfm suggest <file> asks a local AI CLI (Claude Code by default, configurable) to propose an improvement to one of your tracked files, returns the proposal as a unified diff, and stores it as a pending suggestion. dfm apply <id> reviews the diff and applies it, with a fresh snapshot first, so you can roll back if the suggestion turns out to be wrong.

  I’m exited to try this feature out, because I’m sure there is something i"m doing wrong. The “Look at my ~/.zshrc and tell me what I could clean up” is useful feature that doesn’t require me copy and pasting or granting read or write access to my entire home directory.

  Where to Get It

  github.com/llbbl/dotfiles-manager. Pre-built binaries for darwin and linux on arm64/amd64. Current version, as of writing, is v1.4.0.

  If you’ve been meaning to actually back up your dotfiles and the friction has stopped you, this is the post where I tell you the friction is solvable.

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 19, 2026 DevOps AI Programming Go Dotfiles

  Enjoyed this?

• Your AI Coding Agent Can Read Every Secret on Your Machine

  Every developer running an AI coding agent has handed that agent the keys to their machine. Not metaphorically. Literally. The agent runs as your user. It can read every file you can read, execute every command you can execute, and hit every API your stored credentials authorize.

  For most workflows, that’s the point. You want the agent to read your code, modify your project, ship your work. But there’s a quieter implication: the agent can also read your .env files. It can invoke your secret-management tooling. It can grep for API_KEY= across your home directory. And nothing in the agent stack says “wait, you didn’t ask for this.”

  Same-UID isolation isn’t isolation. It’s the absence of isolation labeled politely.

  The usual answer to “keep secrets safe from your coding agent” is: don’t store them where the agent can find them. Use a cloud secret manager. Rotate aggressively. These are good practices, and for local development, they’re often impractical. The agent is going to encounter secrets whether or not your security-best-practices doc approves.

  So over the last week, I built an audit subsystem into lsm, my Local Secrets Manager. The whole thing is designed to answer one forensic question: did anything weird touch my secrets last night?

  The Threat Model

  A defense without a threat model is theater, so let me be specific.

  The threat isn’t a sophisticated remote attacker. lsm is public, open-source code. The threat isn’t a buggy lsm either; bugs happen, and the user can read the source.

  The threat is the agent layer running adjacent to lsm. Coding agents have legitimate access to a wide swath of your filesystem. They’re imperfect at intent inference. They sometimes get prompt-injected. They sometimes run in the background while you’re asleep. When an agent calls lsm get prod DATABASE_URL, the action is indistinguishable from you doing the same thing. The audit log’s job is to make those calls retrospectively distinguishable.

  A secondary threat is an agent covering its tracks. If something reads a secret and then edits the audit log to erase the evidence, the log is worse than useless.

  What Got Built

  The audit subsystem records every access as a structured event: a sequence number, a timestamp, the action, the app and environment, an Actor block describing the calling process, and two cryptographic fields linking each event to the previous one.

  The Actor block was the interesting design problem. It captures parent process ID, parent process name, TTY device path (or empty if there’s no terminal), current working directory, an agent marker derived from environment variables that tools like Claude Code, Cursor, Aider, and Continue set, and the calling user ID. Every field is captured every time. No omitempty. UID zero is a real, meaningful value, and silently dropping it would be a footgun.

  Events land in a hash-chained JSONL file at ~/.lsm/audit.jsonl. Each row carries the SHA-256 of the previous row plus its own body. If anyone edits, inserts, or deletes a row in the middle, the next row’s prev no longer matches and lsm audit verify surfaces the break.

  The chain doesn’t catch tail truncation. If you chop off the end of the file, what’s left is internally consistent. A sidecar file storing the last expected hash is the obvious fix, and I deliberately rejected it. lsm is public code. Any local attacker who knows about the sidecar can rewrite both files in lockstep. Tail-truncation detection is deferred to the off-machine path: when events ship to a remote stack, the last hash naturally lives somewhere the local attacker doesn’t control.

  Reading the Log

  Three commands cover the read side. lsm audit tail does what you’d expect. lsm audit show <seq> prints a single event. lsm audit query is the workhorse, with every field as a filterable dimension: --app, --env, --event, --parent-comm, --agent-marker, --tty present|absent, --since, --until. Output is JSONL when piped and columnar text when interactive.

  Then there’s lsm audit suspicious, which runs four hard-coded detectors in one pass:

  • Outside hours. Events whose timestamps fall outside 07:00–23:00. The 3 a.m. canary.
  • Burst. More than N events from a single parent process within a sliding window. The runaway-agent canary.
  • New parent_comm. Process names not seen in the prior 30 days. The “what is this new thing” canary.
  • Non-interactive, no agent. No TTY, no recognized agent marker. The “what is even running this” canary.

  A single event can stack reasons. A 3 a.m. burst from an unknown parent is unambiguously interesting.

  The detector doesn’t learn baselines, doesn’t call out to an ML model, doesn’t require a service. High-signal patterns are obvious patterns, and obvious patterns are well-served by hard-coded predicates.

  Shipping Events Off the Box

  If you already run an observability stack, lsm can ship audit events over OTLP (the OpenTelemetry wire protocol). Three design choices matter here.

  The local file sink is always authoritative. The remote sink is a mirror, not a replacement. An lsm operation never fails because the remote endpoint is down.

  Redaction is allowlist-based. App and environment names are HMAC-hashed with a per-host salt before becoming labels. The TTY device path is dropped and replaced with a tty_present: true/false boolean. Secret values, cwd, hash, prev, and the schema version never leave the host. Secret names are replaced with key_present: true markers; the remote observer can see that a key was accessed, never which key.

  Events whose name starts with audit. (chain failures, suspicious matches, sink drops) are always local. Telling a remote attacker that local integrity has been compromised is counterproductive.

  What’s Still Open

  The most important non-feature: no command in lsm emits events yet. set doesn’t log. get doesn’t log. delete doesn’t log. The plumbing is complete, the calls are not wired in. Each emit site needs careful thought about which fields are appropriate, whether the event should be local-only, and how it interacts with sensitive operations. That’s the next chunk of work.

  The agent-coding era is normalizing a model where AI tools have wide-ranging access to developer machines. The premise that the agent operates as a fully-trusted local user is unlikely to change soon. Managing the risk means visibility. It means being able to answer “what touched my secrets last night” with a record the agent couldn’t silently rewrite.

  The code is at github.com/llbbl/lsm. The full design lives in docs/observability.md.

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 18, 2026 DevOps AI Programming security

  Enjoyed this?

• Buying Supply Chain Security in 2026: A Vendor Map

  The last post was for solo developers and people without a security budget. This one is for everyone else: the platform engineers, the security leads, and the directors who are getting pitched by four different supply chain security vendors a week and need to figure out which, if any, of them are worth signing a contract with.

  The honest answer is that the vendor landscape in 2026 is overheated. Every SCA company is now also a malicious-package firewall company. Every malicious-package firewall company is also pitching AI-native remediation. The pricing pages are mostly “Contact Sales.” And underneath all of it, the actual problem these tools solve splits cleanly into three layers, and you should know which layer you’re buying.

  The Three Layers

  Layer 1: Update automation. Dependabot (free, GitHub-native) and Renovate (free, more configurable) generate pull requests when new versions of your dependencies are released. They don’t find vulnerabilities. They just shrink the window where you’re running outdated code. Dependabot is the right answer for most teams under 50 engineers. Renovate is what you reach for when you’re tired of triaging 80 individual PRs a week and want grouped updates with auto-merge based on community confidence signals. Neither costs anything. Both should be on.

  Layer 2: Software Composition Analysis (SCA). Parses your lockfiles, matches dependencies against CVE databases, tells you what’s vulnerable. The open-source side of this is fully mature: Trivy, Grype, OWASP Dependency-Check, and OWASP Dependency-Track collectively cover most of what you’d pay Snyk for ten years ago. Dependency-Track in particular is a serious tool. It ingests CycloneDX and SPDX SBOMs, tracks portfolio-wide risk, and integrates EPSS scoring. If you self-host it, the bill is zero.

  The thing the commercial vendors actually sell at this layer is reachability analysis. A vulnerability in a transitive dependency that you import but never actually call is technically a CVE in your inventory. Realistically it’s noise. Snyk, Endor Labs, and Mend.io all build call-graph analysis that determines whether a vulnerable code path is actually invoked by your application. Endor Labs claims their reachability reduces actionable alerts by 90 to 95%. That number is marketing, but the underlying capability is real, and it’s the single biggest differentiator between commercial SCA and the open-source stack.

  Layer 3: Malicious package firewalls. This is the layer that didn’t exist five years ago. Tools like Socket, Phylum, Endor Labs, and Sonatype Repository Firewall sit between your developers and the public registries and analyze package behavior before installation. Socket evaluates 70+ behavioral indicators: does the package read OAuth tokens from disk, does it use marshal.loads to self-deobfuscate, does it inject into HTTP headers. This is the only layer that defends against zero-day malicious packages, because SCA fundamentally can’t. There’s no CVE for “this package was uploaded ten minutes ago and steals AWS keys.”

  What This Actually Costs

  The pricing pages tell you most of what you need to know about who each vendor is for.

  Vendor	Pricing	Who it’s for
  Dependabot	Free	Everyone on GitHub
  Socket	Free up to 1000 scans/mo, Team $25/dev/mo, Business $50/dev/mo	Developers who want low-friction zero-day protection
  Snyk	Free tier (100-300 tests/mo per product), Team $25/dev/mo (5-10 dev cap), Ignite ~$105/dev/mo, Enterprise custom	Teams that want SCA + SAST + IDE integration in one bundle
  Endor Labs	Custom (free tier for small OSS teams)	Orgs drowning in CVE noise; multi-language including C/C++ and Rust
  Mend.io	$300-$1000/dev/year	Enterprise environments that want consolidated dashboards
  Sonatype	$6K-$150K+ in bundled tiers	Large regulated enterprises that need a centralized artifact gateway
  Phylum	Custom enterprise	Teams that want programmatic policy via Open Policy Agent

  Two patterns stand out. Socket and Snyk are product-led growth plays with transparent per-developer pricing, predictable as you scale, accessible at the lower end. Sonatype, Mend.io, and Phylum are enterprise sales motions with significant minimums and multi-month implementation cycles. Endor Labs sits awkwardly in the middle (mid-market and enterprise deals) with credible reachability claims that are hard to replicate with open source.

  The Real Cost of “Free”

  The argument for going all-in on open source, Dependabot plus Trivy plus Dependency-Track plus maybe Socket’s free tier, looks compelling on the spreadsheet. The honest math is more complicated.

  Running this stack at a 100-engineer organization requires somebody to maintain the Dependency-Track server, tune the rulesets to keep false positives from drowning your security team, manually triage alerts that have no reachability context, and respond to the inevitable “is this critical CVE actually exploitable in our environment?” questions from leadership. Realistic estimates put that workload around 20 to 30 hours per week — call it half an FTE of senior engineering time, which fully-loaded lands in the low six figures per year. That’s not zero, and it’s the line item that “we’ll just use open source” plans consistently leave out of the spreadsheet.

  The flip side is the Endor Labs ROI pitch: 90% noise reduction means 9 fewer FTEs needed for triage in a 300-dev org, which they price at roughly $1.5M in saved salary against a five-figure license. That’s a vendor calculation, so take it with the appropriate salt. But the underlying logic that alert noise has real labor cost is correct, and it’s the part most “we’ll just use open source” plans underestimate.

  What I’d Actually Recommend

  For a team of 5 to 50 engineers: Dependabot or Renovate on, Socket’s free tier or paid Team plan for firewall coverage, and npm audit / pip-audit / cargo-audit running in CI. Total spend: $0 to roughly $1,500/month at the high end. This is the configuration that covers 80% of the threat for a small fraction of what a Snyk or Mend contract costs.

  For 50 to 300 engineers: the math starts favoring a paid SCA platform with reachability. Snyk if you also want SAST in the same tool. Endor Labs if you have a polyglot codebase (especially anything with C++ or Rust) and severe alert fatigue. Keep Socket or Phylum as a separate firewall layer. The firewall vendors are still meaningfully better at malicious-package detection than the SCA vendors who bolted it on.

  For 300+ engineers in a regulated industry: you probably need Sonatype or JFrog as a centralized proxy whether you want them or not, because compliance demands a single audited path from developer to registry. Bundle it with Endor Labs or Mend for the reachability layer.

  What I would not do is buy the platform pitch, the “one tool for SCA + SAST + secrets + container scanning + firewall + AI remediation.” Those bundles exist because the vendors want a bigger contract, not because the unified product is actually best-of-breed at any single thing. The companies winning each individual layer (Socket for firewalls, Endor Labs for reachability, Trivy for open-source SCA) are doing so by being focused.

  Closing the Series

  Four posts in: the threat model, the per-ecosystem mitigations, local isolation for the budget-constrained, and now the commercial landscape for everyone else. The unifying thesis across all of them is that supply chain security is not solved by a single tool or a single layer. It’s a stack. Lockfiles at the bottom, audit tooling above that, behavioral analysis on top, isolation as the last line of defense. The right composition depends on who you are and how much risk you can afford to absorb. If your stack right now is “we trust the registry,” you are the threat model.

  Sources

  • Supply Chain Security Tool Selection Framework - SoftwareSeni
  • Endor Labs vs Snyk: SCA, SAST, and Containers Compared
  • Malware Package Firewall: Block Threats Before They Hit Your Code
  • Socket Pricing
  • Introducing Socket Firewall
  • Snyk Software Pricing & Plans 2026 - Vendr
  • Endor Labs Pricing
  • Mend.io Pricing
  • Sonatype Nexus Pricing Guide 2026 - CloudRepo
  • Open Source vs Commercial SCA Tools Comparison - Safeguard
  • OWASP Dependency-Track

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 17, 2026 DevOps security Tooling Supply-chain

  Enjoyed this?

• Sandboxing AI Agents Without Buying Anything

  The previous post (and the one before it) covered the threat model and the per-ecosystem mitigations: lockfiles, --ignore-scripts, cargo-audit, Trusted Publishing. All of that helps. None of it answers the question that keeps me up at night, which is: what happens when an AI agent on my laptop installs a malicious package, and the malicious package was the literal point of the operation?

  This is the new shape of the threat. You’re not getting compromised because you typed npm install wrong. You’re getting compromised because Claude or Cursor confidently invented a package name that didn’t exist, an attacker registered it five hours ago, and the agent ran pip install hallucinated-thing on your behalf without asking. The agent has shell access. Your SSH keys are right there. Your ~/.aws/credentials file is right there. The entire premise of giving an AI agent the ability to just figure it out depends on it being able to execute untrusted code at the speed of conversation, which is also the worst possible threat model.

  If you’re a solo developer, an open-source maintainer, or a startup with no budget for Socket or Endor Labs licenses (more on those next post), the answer isn’t a commercial firewall. The answer is local isolation, and the tools have gotten dramatically better in the last 18 months.

  Containers as the Baseline

  The minimum viable isolation in 2026 is don’t run untrusted code as your user on your host OS. The cleanest way to do that on macOS or Linux is a devcontainer, a fully described, reproducible Linux environment that VS Code, Cursor, and the Claude Code CLI all natively support. You give the agent the container as its sandbox. Project files mount in. SSH keys, AWS credentials, and the rest of your home directory don’t.

  The container runtime matters. Docker Desktop on macOS is a memory pig, 3 to 4 GB resident at idle, with sluggish startup times that make iterative work miserable. OrbStack is the obvious replacement: free for personal use, native Apple Silicon, dynamically allocates memory instead of reserving fixed blocks, and benchmarks show container startup times around 0.2 seconds versus Docker Desktop’s multi-second cold starts. If Docker Desktop is eating half your RAM before you even start Claude Code, OrbStack will give you that memory back.

  The thing to internalize, though, is that a container is not a security boundary by default. It’s a deployment mechanism that happens to have isolation properties when configured correctly. Misconfigured developer containers have been implicated in some of the largest crypto-industry breaches of the last few years. The pattern: a container running with privileged flags, or mounting the wrong host directory, turns into a path straight to the host. Containers help. They don’t save you from yourself.

  The configuration mistakes that void the isolation:

  • Mounting ~/.ssh into the container so the agent can git push. Now any process inside the container can read your SSH keys.
  • Mounting your entire home directory as a convenience. Now everything is accessible.
  • Running with --privileged or sharing the host’s Docker socket. Container escape becomes trivial.
  • Letting the agent run sudo inside the container. The container’s root can chain to host kernel exploits.

  Least privilege, applied seriously. The agent gets the project directory and nothing else. If it needs to commit, it pushes through a credential helper that lives on the host, not by mounting your SSH keys.

  Lighter-Weight Sandboxes

  Spinning up a full container for every test this snippet the LLM wrote interaction is too heavy. There’s a middle layer worth knowing about.

  Python. Pyodide compiles CPython to WebAssembly, which means Python code runs in a deny-by-default memory sandbox with no filesystem or network access unless you explicitly grant it. Works great for evaluating LLM-generated snippets, struggles with C extensions and heavy dependencies. safe-py-runner is the pragmatic alternative: it runs Python in a restricted subprocess with timeouts, memory limits, and I/O marshaling. No container needed. For code that absolutely cannot touch your machine, remote V8-isolate services like Deno Sandbox boot pre-snapshotted Python environments in the cloud and air-gap execution entirely.

  Rust. The build.rs problem from the last post has no first-class solution yet, but on Linux you can wrap cargo build in Landlock, a kernel feature available on 5.13+ that lets unprivileged processes restrict their own filesystem access. Combined with seccomp-bpf for syscall filtering and cgroups v2 for resource limits, you can run a build script that genuinely cannot read your SSH keys or open arbitrary network sockets. Projects like sandbox-rs wrap these primitives into something usable without writing your own seccomp filters. None of this works on macOS without a Linux VM in the way, which is another reason OrbStack plus a devcontainer is the path of least resistance for most people.

  The Mindset Shift

  The honest version of all of this: if you’re running AI agents locally, you have to assume they will eventually install something malicious. Not might. Will. The question is whether the blast radius is the contents of one project directory inside a container, or every credential on your machine plus your entire git history. That gap is what isolation buys you.

  Containers, Landlock, WASM sandboxes, none of these are particularly hard to set up. They’re just things most developers haven’t bothered with because the threat model didn’t feel real. After Shai-Hulud, faster_log, and a year of watching AI agents pip install whatever they invent, the threat model is real.

  Next post I’ll wrap up the series with the commercial side: Socket, Snyk, Endor Labs, Mend, Sonatype, the pricing comparison, and the actual ROI math for whether any of it makes sense for teams below 50 developers.

  Sources

  • State of Dependency Management 2025 — Endor Labs
  • Securing AI Coding Assistants: A Total Cost Analysis — Endor Labs
  • A step closer to isolation — devcontainer-wizard — The Red Guild
  • OrbStack vs Docker Desktop: Performance Facts for Mac
  • Apple Containers vs Docker Desktop vs OrbStack benchmark
  • How to Safely Run AI Agents Like Cursor and Claude Code Inside a DevContainer
  • DevContainers for Secure AI: Isolated & Scalable
  • safe-py-runner: Secure Python execution for LLM Agents
  • mcp-run-python — Pydantic
  • How to Run Rust Binaries Without Root Using Sandboxing — OneUptime
  • sandbox-rs
  • Explore sandboxed build scripts — Rust Project Goals

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 16, 2026 DevOps AI security Supply-chain Containers

  Enjoyed this?

• Python and Rust Have the Same Supply Chain Problem as NPM

  Last post I walked through the threat model for supply chain attacks and dug into the NPM ecosystem specifically: postinstall scripts, npm ci, pnpm’s release-age cooldown. The same structural problems exist in Python and Rust, but the failure modes are different and the tooling has evolved in some surprising directions. Worth understanding both, because if you write any backend code in 2026 you’re probably touching at least one of these ecosystems.

  Python: setup.py Is a Remote Code Execution Primitive

  The thing most Python developers don’t appreciate is that pip install runs arbitrary code by default. Not after install. During install. If a package ships a setup.py, that file is executed in a Python interpreter the moment pip resolves the dependency. Whatever the author wrote, including reading ~/.aws/credentials, scraping environment variables, or opening a reverse shell, runs as your user with full filesystem access.

  This is the part that confuses people coming from other ecosystems: venv and virtualenv don’t help. They isolate Python package versions to avoid conflicts. They are not a security boundary. A package installed inside a virtualenv has the exact same privileges as the user who ran pip install. None of this is a bug, exactly. It’s just an artifact of setup.py being a regular Python script that pip has always been willing to execute.

  The defense-in-depth stack for Python looks like this:

  Stop using pip. I mean it. pip is the worst package manager in mainstream use today and it is the single biggest reason Python’s supply chain story is a disaster. It has no native lockfile. requirements.txt is a shopping list, not a lockfile; it tells pip what to fetch, not what you actually got last time. Run pip install -r requirements.txt twice on two different days and you can get two different dependency trees, because pip resolves transitive deps fresh every time against whatever happens to be on PyPI in that moment. Builds aren’t reproducible. Hashes aren’t verified by default. There’s no separation between “what I asked for” and “what was actually resolved.”

  Every other ecosystem solved this a decade ago. npm has package-lock.json. Cargo has Cargo.lock. Bundler has Gemfile.lock. pip has vibes.

  The --require-hashes flag exists, technically, but it’s duct tape on a broken design. You have to generate the hashes with a separate tool (pip-tools), maintain them by hand, and remember to pass the flag on every install. Nobody does this in practice. The Python Packaging Authority spent fifteen years insisting pip was fine while every other community built proper lockfile-based managers.

  Use uv or Poetry. Both produce real lockfiles with SHA-256 hashes for every direct and transitive dependency, both make installs reproducible by default, both are dramatically faster than pip. uv in particular is the obvious default for new projects in 2026, it’s a drop-in replacement that’s roughly 10-100x faster and treats the lockfile as a first-class artifact instead of an afterthought. Hash verification isn’t a flag you have to remember. It’s how the tool works.

  This doesn’t protect you from a malicious package you pinned on day one. But it does slam the door on silent registry tampering, makes “what’s actually deployed?” a question with an answer, and gets you out of the pip swamp.

  pip-audit for known vulnerabilities. Scans your environment or requirements file against the OSV database, PyPA advisories, and GitHub advisories. Run it in CI. Combined with a real lockfile you get a tight loop: pin exact versions, scan those versions for CVEs, fail the build if anything critical shows up.

  Trusted Publishing (OIDC). If you maintain a package on PyPI, get rid of your long-lived API token and switch to OIDC-based publishing. Your CI runner generates ephemeral, short-lived tokens scoped to a specific repository, branch, and workflow. Leaked PyPI tokens have been the source of multiple high-profile compromises. Trusted Publishing makes the credential effectively un-leakable because it doesn’t exist as a persistent secret.

  The thing I’d actually call out, though, is that none of the Python tooling addresses the setup.py execution problem at install time. Hash pinning verifies you got the right bytes. It doesn’t tell you those bytes aren’t malicious. For that you’re back to either sandboxing the install (Docker, devcontainers) or trusting the registry’s malware detection, which lags by hours to days.

  Rust: The Safety Guarantees Stop at the Compiler

  Rust’s reputation for safety is real, but it’s a property of the compiled language, not the supply chain. The borrow checker doesn’t help you when the crate you’re depending on exfiltrates your SSH key during cargo build.

  The mechanism is build.rs. Crates can include a build script that runs before the compiler, with full user privileges. Procedural macros do the same thing at compile time. In both cases, the code can read files, open network sockets, do whatever it wants. A malicious build.rs is effectively an unsandboxed unsafe block that bypasses code review because nobody reads build scripts. The Rust core team has been discussing sandboxing for years, but nothing has shipped.

  This isn’t theoretical. Two examples from the last six months:

  • September 2025: faster_log and async_println were caught scraping Ethereum and Solana private keys at runtime and exfiltrating them to Cloudflare workers.
  • March 2026: chrono_anchor, dnp3times, and time-sync, all masquerading as time utilities, were transmitting .env file contents to threat actors.

  Both clusters used compromised GitHub OAuth credentials to push under legitimate-looking namespaces. crates.io authenticates via GitHub, so a phished GitHub account is a phished crates.io account.

  The defensive tooling is actually better than what most ecosystems have:

  Tool	What it does
  cargo-audit	Scans Cargo.lock against the RustSec Advisory Database. Run in CI.
  cargo-deny	Lints the dependency graph. Block specific crates, enforce license policies, restrict registries.
  cargo-crev	Decentralized “web of trust” where developers cryptographically sign crate reviews. Elegant, but heavy lift in practice.
  cargo-vet	Mozilla’s pragmatic answer to crev. Centralized audit records per org, with the ability to import audits from peer orgs (Google, Mozilla, Embark) instead of re-auditing every transitive dep yourself.

  If you’re picking one to start with, cargo-audit is the easy baseline. It’s npm audit for Rust and you should be running it in CI yesterday. cargo-deny is the next step up. It lets you actually enforce policy, which is what you want once you’ve used cargo-audit long enough to be tired of triaging the same warnings.

  cargo-vet is the interesting one for any team beyond about five engineers. The insight is that you don’t actually need to audit every crate. You just need to know that someone you trust did. By importing audit records from Mozilla and Google, a small team can effectively delegate the audit work for hundreds of common dependencies without running anything themselves. It’s the closest thing the Rust ecosystem has to a working trust network, and it works because the cryptographic overhead lives at the org level instead of being pushed onto individual developers.

  The Pattern Across All Three Ecosystems

  NPM, PyPI, and crates.io all share the same fundamental design flaw: package installation executes attacker-controlled code by default. NPM has postinstall. Python has setup.py. Rust has build.rs and proc macros. Different files, same problem.

  The mitigations also rhyme. Lock your versions to specific hashes. Run an audit tool in CI. Where possible, prevent install-time execution entirely (--ignore-scripts, pre-built wheels, sandboxed build scripts when they finally land in Cargo). Where you can’t, isolate the install with devcontainers, ephemeral CI runners, anything that contains the blast radius when a dependency turns out to be hostile.

  Next post I’ll get into the isolation side specifically: devcontainers, OrbStack, Landlock, and the practical question of how a solo developer with no security budget actually keeps their laptop from getting owned by an AI agent that just pip installed a hallucinated package name.

  Sources

  • Securing Package Managers: Why NPM, PyPI, and Cargo Are High-Value Targets
  • Defense in Depth: A Practical Guide to Python Supply Chain
  • PyPI Security: How to Safely Install Python Packages
  • Rust Supply Chain Security — Managing crates.io Risk
  • crates.io: Malicious crates faster_log and async_println
  • Five Malicious Rust Crates and AI Bot Exploit CI/CD Pipelines
  • About RustSec Advisory Database
  • cargo-vet FAQ
  • Auditing Rust Crates Effectively (arXiv)
  • Explore sandboxed build scripts — Rust Project Goals

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 15, 2026 DevOps Python security Rust Supply-chain

  Enjoyed this?

• Your Software Is Mostly Strangers' Code

  Modern applications aren’t really written anymore. They’re assembled. Seventy to ninety percent of a typical proprietary codebase is open-source code pulled from public registries, NPM, PyPI, crates.io, maintained by thousands of people you’ve never met. Every npm install is an act of implicit trust extended to strangers, and that trust model has quietly become the weakest link in most security architectures.

  Attackers figured this out a long time ago. Compromising one popular package gives you a blast radius that phishing campaigns can only dream about: CI pipelines, developer laptops, production workloads, client devices, all simultaneously. SolarWinds. The XZ Utils backdoor. The Shai-Hulud worm, which self-propagated through 170+ npm and PyPI packages by hijacking GitHub Actions OIDC tokens and quietly minted new publish credentials as it spread. The ByBit developer compromise. These aren’t outliers anymore. They’re the shape of the threat.

  I want to dig into the mechanics of how this actually happens, and then look at the ecosystem most developers touch every day: NPM.

  How Supply Chain Attacks Actually Work

  The first thing to understand is that supply chain attacks aren’t really “vulnerabilities” in the classic sense. A buffer overflow is an accidental weakness. A malicious package is intentional code, written to steal credentials, drop a reverse shell, or exfiltrate environment variables the moment it lands on your machine. Traditional appsec tools were built to find the former. They are largely blind to the latter.

  The attack patterns cluster into a few categories.

  Typosquatting. Publish axois and wait for someone to fat-finger axios. Sounds trivial, but it works constantly because developers install packages at high velocity and rarely double-check spelling.

  Dependency confusion. If your company has an internal package called corp-auth, an attacker publishes a public package with the same name and a higher version number. Many package managers default to “highest version wins,” and your build pulls the public one instead of your internal one.

  Maintainer hijacking. Compromise a real maintainer through phishing, credential stuffing, or a missing 2FA setup, and push a poisoned update to a package that already has millions of weekly downloads. The Axios compromise in March 2026 followed exactly this pattern. The XZ Utils backdoor was a slower variant. The attacker spent months building trust as a “helpful” co-maintainer before slipping a backdoor into the build.

  The thing that makes all of this so effective is the automation downstream. Unpinned versions, auto-merging update bots, transitive dependencies five layers deep. Once a malicious version hits the registry, it propagates fast.

  Why NPM Is the Highest-Stakes Ecosystem

  NPM serves tens of billions of downloads a week. A typical JavaScript project today pulls in well over a thousand transitive dependencies. Ten years ago that number was in the dozens. The dependency graph is just structurally enormous, and it’s getting worse.

  The specific architectural problem in NPM is the lifecycle script, specifically postinstall. NPM lets package authors define scripts in package.json that run automatically when the package is installed. This was designed for legitimate reasons: compiling native bindings, configuring environments. But it also means arbitrary shell commands execute on your machine the moment you type npm install. No code review. No second thought. Just immediate execution as your user.

  There are a few practical mitigations, and they’re worth knowing whether you’re a solo developer or running platform security at a large org.

  Disable lifecycle scripts. Either pass --ignore-scripts ad hoc, or set it globally:

  npm config set ignore-scripts true
  

  This breaks some legitimate packages (esbuild, bcrypt, anything compiling native code). To manage that, tools like can-i-ignore-scripts scan your node_modules and generate an allowlist of packages that genuinely need scripts to run. Frameworks like @lavamoat/allow-scripts formalize this with a deterministic config you can check into the repo.

  Use npm ci in CI, not npm install. This is non-negotiable for production builds. npm install will happily resolve newer minor versions inside your semver ranges and rewrite package-lock.json. npm ci refuses to do that. If the lockfile doesn’t match exactly, the install fails. That’s the behavior you want when the question is “did anything change that I didn’t approve.”

  Consider switching to pnpm. pnpm 10+ has been quietly building some of the best structural defenses in the ecosystem. Postinstall scripts are off by default and require an explicit allowBuilds list. blockExoticSubdeps prevents transitive deps from resolving via random Git URLs or tarballs.

  The killer feature, though, is minimumReleaseAge. As of pnpm v11 (May 2026), the default is 1440 minutes, so pnpm simply refuses to resolve any package version less than 24 hours old. Most malicious packages get pulled from the registry within hours of being detected. A 24-hour cooldown turns the community into your early warning system, with no behavioral analysis or commercial tooling needed.

  That last one is the single highest-leverage change you can make as an individual developer. It costs nothing, it doesn’t break your workflow, and it neutralizes most day-zero registry malware before it ever reaches you.

  Next post I’ll dig into the Python and Rust sides of this. pip’s setup.py execution problem, Rust’s build.rs issue, and the surprisingly mature auditing toolchain the Rust community has built around cargo-audit, cargo-deny, and cargo-vet.

  Sources

  • Securing Package Managers: Why NPM, PyPI, and Cargo Are High-Value Targets
  • Defending Against NPM Supply Chain Attacks: A Practical Guide
  • NPM Ignore Scripts Best Practices
  • Mitigating supply chain attacks
  • Get safe and remain productive with can-i-ignore-scripts
  • The Landscape of Malicious Open Source Packages: 2025 Mid-Year Threat Report
  • The Evolving Software Supply Chain Attack Surface
  • Introducing OpenSSF’s Malicious Packages Repository

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 14, 2026 DevOps security javascript Npm Supply-chain

  Enjoyed this?

• Semantic Docs Spring Update: Astro 6, Auto-Releases, npm

  The last two months on Semantic Docs have mostly been maintenance work, but a few things I wanted to talk about. I pushed through a major framework upgrade, swapped out a vendored library for a real published package, and finally automated the release pipeline. Five tagged releases later, here’s where we are.

  The Headlines

  • Upgraded to Astro 6
  • Switched from a vendored logger to the published logan-logger npm package
  • Shipped an auto-release workflow driven by Conventional Commits
  • Three rounds of dependency updates plus a security-focused sweep
  • Five tagged releases, v1.3.3 through v1.5.0

  Astro 6

  The Astro 6 upgrade was easy. Semantic Docs runs a hybrid setup, static article pages plus a server-rendered search endpoint, and that part barely needed any attention. Most of the work was in the dependency layout, not the application code.

  One note if you’re forking or syncing this theme: if you’re upgrading from v1.3.5 or earlier (anything pre-Astro-6, which landed in v1.4.0), delete your node_modules and your lockfile and do a clean install. Skip that step and you’ll get weird errors that look like your code is broken when it’s really just leftover state.

  A Real npm Package Instead of a Vendored Logger

  For a while, the project was using a logger I wrote to experiment with publishing to both npm and JSR. It was a useful exercise. I wanted to see what a clean foundational package looked like across both registries, and I think it turned out well.

  But for this repo, I wanted consistency over experimentation. So I swapped the vendored copy for the published logan-logger npm package. Behavior is the same, the surface area is the same, it’s just back on the npm registry.

  Automated Releases

  I’ve liked using Conventional Commits to drive automated releases. When a PR merges to main, the workflow figures out the next version from the commit messages, tags it, and publishes a GitHub release with a generated changelog.

  The commit type determines the version bump. feat: bumps the minor, fix: bumps the patch, breaking changes bump the major. The changelog falls out of the same metadata. More automation here the better.

  If you’ve been on the fence about Conventional Commits, this is the use case that sold me.

  What’s Next: Embedding Quality

  The reference implementation uses TEI for search embeddings, and that’s been fine. But “fine” is not the same as “good,” and I want to actually compare quality across providers before I commit to anything long term.

  Two I want to test:

  • Jina (now owned by Elastic)
  • Mistral, which has been putting out genuinely strong embedding models

  The goal is to run the same corpus through each, evaluate the search results, and figure out which one earns a highlight. Whatever I learn from that work will get folded back into the open source Semantic Docs repo so anyone running their own instance can make an informed choice instead of just trusting my defaults.

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 9, 2026 DevOps Open-source Astro Semantic-docs

  Enjoyed this?

• SAST vs AI PR Review: Two Tools, Different Jobs

  If you have worked in DevSecOps, you might be wondering if AI pull request review tools are going to replace traditional SAST scanners. Short answer: no. Longer answer: they’re solving different problems, and if you’re picking one over the other, you might be making a mistake.

  Here is how I think about it.

  SAST is the Compliance Gatekeeper

  Static Application Security Testing tools, think Semgrep, SonarQube, Checkmarx, Fortify, parse your source code (usually into an Abstract Syntax Tree) and hunt for known vulnerability patterns. They don’t run the code. They just read it and “pattern-match” against rules.

  The focus here is security, compliance, and strict rule enforcement. SAST is the automated gatekeeper that makes sure your code clears the OWASP Top 10 bar before it merges.

  What SAST does well:

  • It’s deterministic. If a rule matches a pattern, the engine flags it every single time. Run it twice on the same code, get the same result.
  • It satisfies auditors. Frameworks like PCI-DSS, SOC 2, and HIPAA expect documented secure-development practices, and a formal SAST scanner is the easiest way to produce that evidence. AI agents don’t count here, at least not yet.
  • It can do real taint analysis. Enterprise tools can track untrusted input from the moment it enters your app to the moment it hits a dangerous sink.

  Where SAST falls down:

  • The false positive rate is brutal. Rigid rules with no context means a lot of noise. Developer fatigue is real, and once your team starts ignoring scanner output, you’ve lost the game.
  • It can’t see your business logic. A SAST tool has no idea what your application is supposed to do, so it can’t tell you when the logic itself is broken.
  • Comprehensive scans are slow. Hours on large codebases isn’t unusual, though Semgrep has been doing good work on this front.

  AI PR Agents are the Peer Reviewer

  Tools like CodeRabbit, Qodo, Greptile, GitHub Copilot Code Review, Cursor Bugbot, and Claude Code (set up as a review skill) plug into your version control and read the PR diff with the surrounding code context. They behave less like a scanner and more like a colleague who actually read your changes.

  The focus is developer productivity, code quality, logic bugs, and contextual feedback.

  What they do well:

  • They understand intent. LLMs can reason about why the code is changing, not just whether it matches a rule. That’s a different category of feedback.
  • The signal-to-noise ratio is good. When an AI flags something, it usually comes with an explanation that makes sense. Less noise, more useful comments.
  • They suggest fixes. Not just “this is wrong” but “here’s a diff you can apply.” That’s huge for actually closing the loop on review feedback.
  • The scope is broader. Architecture, performance, style, security, all in one pass.

  Where they fall down:

  • They’re non-deterministic. Same vulnerability, two PRs, two different outcomes. That’s not a bug, that’s how LLMs work, and it’s why auditors don’t trust them.
  • They don’t satisfy compliance. No auditor is going to accept “the AI looked at it” as a substitute for a formal scanner.
  • Hallucinations happen. Invented issues, misread intent, suggestions that refactor things that didn’t need refactoring. You still need a human filtering the output.

  The Quick Comparison

  Feature	SAST	AI PR Review
  Primary Goal	Security & Compliance	Code Quality & Productivity
  Analysis Method	Deterministic rules & AST	Non-deterministic LLMs
  Business Logic	Blind	Context-aware
  False Positives	Often high	Usually low
  Compliance Proof	Accepted as evidence	Not accepted
  Feedback Loop	Dashboard / CI output	PR comments / chat

  The Lines Are Starting to Blur

  The interesting thing happening right now is convergence from both directions.

  On the SAST side, tools like DryRun Security are pitching themselves as “AI-native SAST,” trying to keep the deterministic backbone while using LLMs to filter out the false positives that make traditional scanners painful to live with.

  On the AI agent side, CodeRabbit and Greptile keep getting better at catching real security vulnerabilities, not just style issues. They’re slowly creeping into territory that used to belong exclusively to SAST.

  This is going somewhere, but it’s not there yet.

  Where to Start Your Evaluation

  Treat them as complementary, not competitive.

  For SAST, evaluate against your audit footprint, the languages in your codebase, and how much false-positive triage your team can absorb. Semgrep, SonarQube, Checkmarx, and Fortify all sit in different price-and-friction zones, and the right one depends on what your business actually needs to prove.

  For AI PR review, evaluate based on how it fits your existing review workflow, what languages and frameworks it understands well, and the signal-to-noise ratio in practice on your codebase. CodeRabbit, Qodo, Greptile, Copilot Code Review, Bugbot, and a Claude Code review skill all approach the problem differently.

  If you pick one category and skip the other, you’re either passing compliance with mediocre code review, or getting great review feedback while failing your next audit. Neither is a win.

  The AI tools aren’t replacing SAST. They’re filling in the gap SAST was never designed to cover.

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 8, 2026 DevOps AI Programming security

  Enjoyed this?

• Your Data Lake's Vulnerability Problem Is Really an Identity Problem

  I’ve been reading through the post-mortems on the last few years of data lake breaches, and the pattern is depressing. We keep blaming the platforms. We should be blaming ourselves.

  Let me give you an example.

  The Snowflake Breach Wasn’t a Snowflake Breach

  In mid-2024, at least 165 organizations got hit through their Snowflake instances. AT&T lost over 50 billion call records. Ticketmaster, Santander, Advance Auto Parts. The headlines wrote themselves: Snowflake hacked.

  Except Snowflake wasn’t hacked. Mandiant, CrowdStrike, and Snowflake all reached the same conclusion in their forensics. No zero-day. No flaw in the cryptographic platform. No internal compromise of Snowflake’s corporate network. No brute-force attacks against API limits.

  What actually happened? UNC5537, a financially motivated group also tracked as Scattered Spider and ShinyHunters, walked through the front door with valid stolen credentials. Those credentials were harvested over years by commodity infostealer malware (VIDAR, LUMMA, REDLINE) running on the personal laptops of third-party contractors. The same laptops these contractors used for gaming and pirated software also held the keys to their clients' enterprise data lakes.

  One contractor laptop. Multiple enterprise environments compromised. That’s the actual story.

  79.7% of the accounts UNC5537 used had prior credential exposure. Some had been valid and un-rotated since November 2020.

  The Two Doors They Walked Through

  The first attack vector was the SSO side door. Plenty of victim organizations had a perfectly fine enterprise IdP enforcing strong passwords and MFA. They just forgot to make SSO mandatory. A local authentication pathway was left active alongside it. Attackers logged in directly with stolen local credentials, completely bypassing the IdP, and the MFA requirement never fired.

  The second was credential stuffing against inactive, orphaned, and demo accounts belonging to former employees. Nobody audits those. Nobody enforces MFA on those. So they don’t get protected by the controls that exist on the production accounts.

  Once inside, the kill chain was almost boring. SHOW TABLES to enumerate. CREATE TEMPORARY STAGE to make an ephemeral staging area that disappears when the session ends, erasing forensic evidence. COPY INTO with GZIP compression to keep the payload small enough that volumetric alarms didn’t trigger. GET to pull it down to a VPS in some offshore jurisdiction. Done.

  No IP allowlisting was in place anywhere. The connections from Mullvad and PIA exit nodes were treated with the same trust as an employee on the corporate VPN.

  The Bucket Problem Hasn’t Gone Away Either

  Alongside the identity attacks, the boring stuff keeps working. Misconfigured S3 buckets are still the most reliable way to expose a data lake. In late 2024, an open bucket used as a shared network drive was found containing raw customer data, cryptographic keys, and secrets. In 2025, a US healthcare provider left millions of patient records readable for weeks before anyone noticed.

  Then there’s Codefinger. In January 2025, that group used compromised AWS credentials to access S3 buckets and then weaponized AWS’s own Server-Side Encryption with Customer-Provided Keys (SSE-C) to ransomware the data in place. They didn’t even need to exfiltrate it. They just encrypted it with a key the victim didn’t have and demanded Bitcoin.

  That’s a native cloud feature being turned against you because somebody granted too many permissions to a service account.

  The Boring Conclusions Are the Important Ones

  Identity is the perimeter now. The encryption-at-rest story we’ve been telling ourselves for a decade is irrelevant when the attacker authenticates as a real user. Stop treating SSO as optional. Stop leaving local auth paths open next to it. Enforce MFA on every account, including the demo and service accounts you forgot about.

  Your data lake should not be reachable from the public internet. Route everything through PrivateLink or the equivalent in your cloud. Allowlist the IPs that should be touching analytical workloads, and don’t make exceptions for “just this one contractor.”

  And as you start handing access to AI agents, remember that static roles aren’t going to cut it. Just-in-time entitlements and contextual access control are the only way you’re going to keep up with autonomous systems making queries on your behalf.

  The data lake industry spent years arguing about table formats, vendor lock-in, and egress fees. Meanwhile, attackers were just collecting passwords from gaming laptops and walking in.

  Fix the doors first.

  Sources

  • UNC5537 Targets Snowflake Customer Instances (Mandiant / Google Cloud) — Forensic analysis, kill chain, infostealer attribution
  • Snowflake Data Breach: Lessons Learned (AppOmni) — SSO side door, MFA bypass mechanics
  • Major AWS S3 Bucket Breach Exposes Data (NHIMG) — Codefinger SSE-C ransomware tactic
  • Misconfigured Cloud Assets: How Attackers Find Them (CybelAngel) — Recent open-bucket exposure incidents
  • 5 Key Lessons from the Snowflake Data Breach (Tanium) — Defensive posture summary

  ---

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 6, 2026 DevOps security Cloud Data-lake

  Enjoyed this?

• Your Data Lake Has a Permissions Problem

  Consolidating every business unit’s data into one giant lakehouse sounds like a win until you realize the security model from your old data warehouse can’t scale to it. You took ten silos, each with their own access rules, and merged them into one location. Now everyone wants in, and your security team is the bottleneck.

  Let me walk through three places where the cracks usually show up.

  RBAC Falls Over Faster Than You Think

  Role-Based Access Control is the model most teams start with. Permissions are tied to a job function. Sales reps get read access to sales tables, data engineers get write access to staging, and so on. It works fine when you have ten roles.

  It does not work when you have a thousand.

  Say your sales reps should only see accounts in their territory, and only accounts they personally manage. Under pure RBAC, you need a unique role for every territory-by-account-owner combination. That’s role explosion, and it’s how compliance audits become impossible and legitimate access slows to a crawl. The roles list grows faster than anyone can review it, which means stale permissions sit there forever.

  The answer is Attribute-Based Access Control. Instead of asking “what role is this user in,” the system asks “what attributes does this user have, what attributes does this data have, and what’s the policy at this exact moment.” Tag a column as PII. Tag a schema as HR. Write one policy that says anyone outside the HR compliance group sees masked data when they touch a PII column. Done. That single policy replaces hundreds of bespoke roles.

  This is what Unity Catalog and Starburst Galaxy are built around, and it’s the model that will scale with the data.

  Column and Row Security Should Be Boring

  Once you have ABAC and a real metadata catalog, column-level masking and row-level filtering become a non-event. You write a SQL expression that masks the first five digits of an SSN for lower-privileged roles. You write a row filter that silently appends WHERE region = 'user_region' to every executive’s SELECT *.

  The key word is silently. The user doesn’t see a different table. They don’t have a sanitized copy. The policy is enforced at the catalog layer, so it works the same whether they’re querying through Spark, Trino, a BI dashboard, or a pipeline. One source of truth, one policy, every engine.

  If you’re still maintaining separate “sanitized” copies of tables for different audiences, you’re doing it the 2015 way and you’re going to drift.

  The IAM Default Problem

  Most cloud services ship with default IAM roles, and a surprising number of those defaults attach AmazonS3FullAccess or something equally permissive.

  SageMaker does it. The Ray autoscaler role does it. There are more.

  Picture the failure mode. An attacker compromises some peripheral app, maybe a forgotten Jupyter notebook, maybe a misconfigured Lambda. That workload has an IAM role attached because that’s how cloud workloads talk to S3 without hardcoded credentials. The attacker inherits the role. And because the role has full S3 access, they’re not constrained to the bucket the application actually uses. They can enumerate every bucket in the entire account.

  That’s how a single compromised container becomes a full data lake breach. Researchers call it a bucket monopoly attack. I call it the most predictable incident in the industry.

  The fix is not glamorous. Stop using s3:* in any policy. Write resource-scoped policies that name the exact buckets and prefixes a workload needs. Audit the default roles every cloud service hands you and replace them. Use Security Lake or Detective to flag cross-service API calls that don’t match normal patterns. None of this is fun. All of it is necessary.

  And Then There’s the Agent Problem

  The new wrinkle is that humans are no longer the primary consumers of your data. Autonomous agents are. They issue more queries, hit more tables, and move faster than any human team.

  Long-lived credentials and static roles don’t fit that workload. The pattern emerging is Just-In-Time entitlements, where an agent gets a narrow, ephemeral permission for the duration of a single execution thread, then loses it. Pair that with declarative policy metadata baked into the data assets themselves, so the agent knows what it’s allowed to do with a dataset before it ever runs the query.

  We’re early on this. Most organizations are still working through the basics, and that’s fine. But if you’re designing access controls today, design them assuming the next thing hitting your lake isn’t a person.

  What to Actually Do

  If you’re auditing your own data lake security, the order I’d work in:

  1. Find every IAM role with a wildcard permission. Replace them.
  2. Move from RBAC to ABAC at the catalog layer. Stop creating new roles.
  3. Pull your data lake off the public internet. PrivateLink, private endpoints, IP allowlists for the legacy stuff that can’t move.
  4. Then start thinking about agents.

  The lakehouse pitch is unification. The lakehouse reality is that unification multiplies the cost of every bad permission. Get the basics right before you bolt on anything fancy.

  Sources

  • AWS Default IAM Roles Found to Enable Lateral Movement (The Hacker News) — SageMaker / Ray autoscaler default roles, bucket monopoly attacks
  • What Is Fine-Grained Data Access Control? (TrustLogix) — RBAC role explosion, ABAC fundamentals
  • Core concepts for ABAC (Databricks Unity Catalog docs) — Tag-driven policy enforcement
  • Top 12 Data Governance Predictions for 2026 (Hyperight) — Just-in-time entitlements, declarative policy metadata

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 5, 2026 DevOps security Cloud Data

  Enjoyed this?

• The Real Cost of Your Data Lake (It's Not the Storage)

  If you’re sketching out a data platform on a whiteboard right now, I want you to do something. Stop calculating storage costs. They’re not the bill.

  I pulled the public pricing for AWS, Azure, GCP, Databricks, and Snowflake and stacked them next to each other. Storage is the cheap part. The expensive part is everything that moves the data, and the expensive part is the part you’re least likely to model correctly when you’re picking a vendor.

  Let me walk through what actually shows up on the invoice.

  Raw Object Storage Is Basically Free

  For hot, frequently accessed data, the big three are within a rounding error of each other:

  • Azure Blob (LRS, Hot): $0.018 per GB/month
  • Google Cloud Standard: $0.020 per GB/month
  • AWS S3 Standard: $0.023 per GB/month (first 50 TB)

  Drop into the cool tiers and AWS S3 takes the lead at $0.0125 per GB. Drop into deep archive and you’re paying $0.00099 per GB on either AWS Glacier Deep Archive or Azure Archive. That’s a tenth of a cent per gigabyte, per month, for data you almost never touch.

  Good for you, but I think anyone leading with “per-GB storage cost” in a procurement deck is selling you a story. Storage capacity is roughly five percent of a typical Databricks bill. Five. The other 95% is the part nobody wants to talk about.

  The Egress Trap

  Ingress is free. Always. The cloud providers want your data in.

  Getting it back out is where they collect.

  • Azure Blob: $0.087/GB external egress
  • AWS S3: $0.090/GB
  • Google Cloud: $0.120/GB (but free if you stay inside Google’s ecosystem, which is the whole point of that pricing)

  Then layer on API operations. A million GET requests on S3 costs about $0.40. The same million GETs on Google Cloud Storage can run closer to $5.00 because they classify operations differently. If your analytics workload is hammering small files, those API calls add up faster than the storage they’re reading.

  Storing 10 TB? Maybe $200 a month. Storing 500 TB? You’re at $10,000 a month before a single byte leaves the region or a single query fires.

  Databricks: Two Bills, One Headache

  Databricks uses what’s commonly called a Two-Bill Model. You get one invoice from your cloud provider for the actual VMs and storage, and a separate invoice from Databricks for the software, measured in DBUs (Databricks Units).

  In a typical mid-sized deployment around $18,000/month, the breakdown looks like this:

  • VM compute from the cloud provider: ~55%
  • Databricks DBU fees: ~30%
  • Storage: ~5%
  • Network egress: ~5%

  The DBU rate changes based on what you’re doing. Automated jobs start at $0.15/DBU. Interactive notebooks for analysts start at $0.40/DBU. That’s not an accident. Databricks wants you running production workloads on cheap job clusters, not on the expensive all-purpose clusters your data scientists love to leave running over a weekend.

  If you’re not actively pushing teams toward job clusters and ARM-based instances, you’re leaving real money on the table.

  Snowflake: The Hidden Storage Multiplier

  Snowflake’s pricing pitch sounds clean. Pass-through storage at $40/TB/month on-demand, dropping to $23/TB/month with a capacity commitment. Compute as Credits. Done.

  Except it isn’t done. Snowflake stores data in immutable 16MB micro-partitions. Immutable. You can’t change them in place. Update a single row in a 1 TB table and Snowflake writes a new file and keeps the old one around.

  Why keep the old one? Two features:

  • Time Travel: query historical states of your data for up to 90 days
  • Fail-Safe: a 7-day disaster recovery window you cannot turn off

  This is the part that gets people. A 1 TB table that’s getting updated multiple times a day can balloon to 25 TB of billed storage because Snowflake is retaining every prior version of every micro-partition you’ve touched. Your dashboard says “1 TB table.” Your invoice says otherwise.

  And compute? Virtual Warehouses bill per second, but with a 60-second minimum every single time you resume or resize. Aggressive auto-suspend sounds like a cost optimization. It’s not. If you’re spinning a warehouse up and down every 30 seconds, you’re paying the 60-second minimum every time and quietly multiplying your bill.

  What I’d Actually Do

  A few things I’d put on the wall before signing anything:

  • Model egress, not storage. Run your worst-case query pattern through the calculator. Storage is noise.
  • Lifecycle everything. Cool tier and archive pricing are 10x to 100x cheaper. If your data is older than 90 days and nobody’s queried it, it shouldn’t be in hot storage.
  • For Databricks: push every recurring workload to job compute. Audit interactive cluster usage monthly.
  • For Snowflake: if you have high-frequency update patterns, profile your actual storage footprint, not your logical table size. The gap will surprise you.
  • For multi-cloud: don’t. Egress will eat the savings before you finish the architecture diagram.

  The vendors all have a story about why their model is the cheap one. Read past the per-GB number on the slide. The bill is somewhere else.

  Happy modeling.

  Sources

  • Databricks Pricing Explained (Dawiso) — Two-Bill Model, DBU breakdown
  • Snowflake Pricing Explained (SELECT.dev) — Time Travel storage multiplier, micro-partition behavior
  • Cloud & AI Storage Pricing Comparison 2026 (Finout) — AWS / Azure / GCP per-GB and tier pricing
  • S3 vs GCS vs Azure Blob Storage (ai-infra-link) — Egress and API operation pricing
  • Snowflake Pricing in 2026 (CloudZero) — Virtual Warehouse 60-second minimum behavior

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 4, 2026 DevOps Cloud Data Snowflake Databricks

  Enjoyed this?

• AI Code Reviewers Won't Save You

  Dropping an AI reviewer into your pull request pipeline is just a band-aid. Tools like CodeRabbit or Greptile are great for catching syntax errors or basic anti-patterns, but they can’t assess architectural intent or domain-specific business logic. They’re spell-checkers for code. Useful, sure. But nobody ever said “our codebase is solid because we run spell check.”

  AI doesn’t change your engineering baseline. It just accelerates it. If your foundational guardrails are weak, agentic tools will help your team generate technical debt at unprecedented speeds. So the real question isn’t “how do we review AI code?” It’s “how do we build systems that prevent slop from ever reaching production?”

  Shift Left, Hard

  When engineers use agents to scaffold a new Go service or spin up a SvelteKit frontend, they’re inevitably pulling in generated dependencies or utilizing unfamiliar libraries. Models hallucinate packages. They suggest insecure patterns with total confidence.

  Your CI pipeline needs to be ruthless before a human ever looks at the code. Aggressive SAST and SCA should automatically block PRs that introduce vulnerable dependencies or hardcoded secrets. If the agent generates slop, the pipeline rejects it instantly. No discussion.

  Make the Agents Write the Tests

  Agents are incredibly eager to generate feature code, but humans are historically lazy about writing the tests for it. The influx of AI-generated code means human reviewers can’t possibly step through every logic branch manually.

  So flip the script. Use the agentic tools to build the guardrails themselves. Mandate that any generated feature code must be accompanied by generated, human-verified unit tests. If an agent writes a sprawling TypeScript function, the build should fail if the test coverage doesn’t meet a strict threshold. You’re already using AI to write the code. Use it to prove the code works, too.

  Context Boundaries Matter

  Bloated AI output often happens because the model is given too much context or allowed to generate too much at once. Heavyweight IDEs with aggressive multi-file auto-completion can easily create cascading messes across a codebase.

  Define strict architectural boundaries and API contracts upfront. Agents should be tasked with solving small, well-defined, modular problems. “Write a function that parses this specific JSON schema” is a good prompt. “Build the backend” is not. The tighter the scope, the less room for generated nonsense.

  Observability Is Your Safety Net

  You can’t catch all generated slop at the PR level. Some of it only reveals itself under load. An agent might write a technically correct query that causes an N+1 database issue, or introduce a subtle memory leak that passes all unit tests.

  Your ultimate safety net is what happens at runtime. You need an airtight observability stack to trust the velocity AI brings. Logs, distributed tracing, metrics, all feeding into dashboards your team actually watches. When generated code hits staging, you need the immediate telemetry to spot performance regressions before they reach production.

  Redefine the Human Review

  Because AI makes the “typing” part of coding trivial, the human code review needs to fundamentally shift. Reviewers should no longer be looking for missing semicolons. They should be asking: “Does this component fit our architecture?” and “Did the agent over-engineer this solution?”

  Train your senior engineers to review for intent and systemic impact. That’s the stuff AI genuinely can’t do yet. Leave the syntax checking to the robots.

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  May 2, 2026 DevOps AI Software-development Code-review

  Enjoyed this?

• What Is a Runbook and Why Should You Care?

  If you’ve ever been woken up at 3 AM by a pager and stared at your screen trying to remember how the database failover works, you already know why runbooks matter. You just might not have had one yet.

  A runbook is a step-by-step guide for handling a specific operational scenario. Database goes down? There’s a runbook for that. Failed deployment needs a rollback? Runbook. Routine certificate rotation? You get the idea. They range from simple markdown files to fully automated scripts where a human only needs to click “approve.”

  That’s the idea anyways. The impact of having good ones versus not having CAN be massive.

  Why They Matter

  When something breaks in production, your brain is not at its best. Adrenaline kicks in, Slack is blowing up, and suddenly you can’t remember if you’re supposed to restart the service first or check the connection pool. A runbook takes the thinking out of the equation. You follow the steps. You restore the service. You go back to sleep.

  This directly lowers your Mean Time To Recovery (MTTR). Instead of spending twenty minutes in a group call debating what to try next, you open the runbook and start executing.

  Runbooks also solve the consistency problem. If five different engineers respond to the same alert five different ways, you’re rolling the dice every time. One of those approaches might cause a secondary outage. A runbook ensures everyone follows the same diagnostic and remediation path, which means fewer surprises.

  And then there’s the tribal knowledge issue. Every team has that one senior engineer who knows exactly how to fix the weird thing that happens once a quarter. What happens when they’re on vacation? Or they leave the company? A runbook gets that knowledge out of their head and into a document the whole team can use.

  It also makes onboarding way faster. New engineers can start handling on-call rotations with confidence instead of hoping nothing breaks on their watch.

  Treat Them Like Code

  This is the part a lot of teams get wrong. Runbooks shouldn’t live in a random Confluence page that hasn’t been updated since 2023. They should live in version control. Sometimes they’re kept in the repo with the code. Other times they’re kept separate. It’s up to you. It’s up to the team on where to put it.

  If a developer changes how a service authenticates or connects to a database, the associated runbook needs to be updated in the same pull request. An outdated runbook is worse than no runbook at all. It sends engineers down the wrong path during an outage, which burns time and trust.

  Share Early, Share Often

  A runbook sitting in someone’s private folder is doing exactly nothing for your team.

  Start during the draft phase. Have someone who didn’t write the runbook try to follow it. If they get confused or stuck, the runbook needs work. This is the cheapest way to find gaps.

  When a new service is heading to production, the runbook should be part of the readiness review. I’d argue a service shouldn’t go live without one. And after an incident, if the runbook was wrong or didn’t exist, creating or fixing it should be a mandatory action item from the post-mortem.

  One more thing. Practice them. Run game days where the team actually walks through runbooks before a real emergency happens. The worst time to discover your runbook has a missing step is when production is on fire.

  So Here We Are

  Runbooks aren’t glamorous. Nobody’s giving a conference talk about the beautiful runbook they wrote last quarter. But they’re the difference between a calm, methodical incident response and a panicked Slack thread full of guesses. Write them, version them, share them, and practice them. Your future self will thank you.

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  Apr 30, 2026 DevOps Sre Runbooks Incident-response

  Enjoyed this?

• What Temporal Actually Does (And Why You'd Want It)

  Building a multi-step process across microservices usually goes something like this. You wire up a message queue, add retry logic, build a state machine backed by a Postgres status column, throw in some cron jobs, and pray. It sounds complicated because it is.

  Temporal is an open-source “durable execution” system that replaces all of that duct tape with a single, opinionated framework. Lets break it down.

  Workflows and Activities

  Temporal splits your application into two concepts:

  • Workflows are your business logic, written in standard code (Go, Python, TypeScript) using a Temporal SDK. They must be deterministic. They define the order of operations, branching, loops, and error handling.
  • Activities are the actual tasks your services perform. HTTP requests, database writes, external API calls. Activities are where the non-deterministic, real-world work happens.

  When a workflow runs, it executes on your own worker services. Every time it schedules an activity, starts a timer, or completes a step, the Temporal Server records that event internally. If the worker crashes, another worker picks it up, replays the workflow’s event history to the exact point of failure, and resumes. No data loss. No half-finished state.

  All of all the things that you would have to build yourself simplified Into A framework that handles it for you.

  What It Replaces

  Without something like Temporal, teams generally land in one of two camps:

  • Choreography (event-driven): Services emit and listen to events through a message broker like Kafka or RabbitMQ. Highly decoupled, sure. But in practice it turns into a pinball machine. There’s no single place to understand the flow of a business transaction. Debugging becomes detective work across dozens of services and topics.
  • Ad-hoc orchestration: You build a custom state machine with a database, message queues, background workers, and cron jobs. Then you write a ton of boilerplate for retries, dead-letter queues, and idempotency. Every team ends up building a slightly different version of this, and none of them are great.

  Temporal gives you the reliability of a custom state machine without making you build and maintain one.

  Why It’s Worth Looking At

  A few things stand out:

  • Durable sleep. A workflow can execute sleep(30_DAYS). Temporal suspends the execution, frees the worker’s resources, and wakes it back up a month later exactly where it left off. Hard to do with a cron job.
  • Built-in resiliency. Exponential backoffs, timeouts, and retry policies are configured on the activity invocation. You’re not writing custom while loops and try/catch blocks to handle network jitter.
  • Centralized observability. Instead of piecing together distributed traces or searching through logs to figure out why step 4 of 7 failed, the Temporal UI shows the exact execution state of every workflow. Inputs, outputs, errors, all in one place.
  • Code over configuration. Unlike AWS Step Functions or YAML-heavy tools like Airflow, you write workflows in a real programming language. You can unit test them, store them in version control, and run them through your normal CI/CD pipeline.

  That last point is worth reading and thinking through again. If your orchestration logic lives in code, it gets all the benefits code gets. Reviews, tests, refactoring, IDE support. Visual workflow builders look great in demos, but they don’t scale the way code does.

  Should You Use It?

  Temporal isn’t free in terms of operational complexity. You’re running the Temporal Server (or paying for Temporal Cloud), and your team needs to understand the replay model and determinism constraints. It’s not something you bolt on to a simple CRUD app.

  But if you’re managing distributed transactions with queues, cron jobs, and hand-rolled state machines, Temporal is worth a serious look. It takes the hardest parts of that problem and makes them someone else’s. Durability, retries, observability. All handled.

  I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].

  Apr 27, 2026 DevOps Temporal Microservices Distributed-systems

  Enjoyed this?