Configuration
-
My Logger Documented Four Environment Variables. None of Them Worked.
I shipped a logging library with a
LOG_LEVELenvironment variable. The README documented it. A dedicated docs page documented it with examples and runtime-specific caveats. There were unit tests covering it, and they passed on every CI run for just under thirteen months.Nothing called the function.
Here’s the command that found it. One line, run against the last 1.x tag:
$ git grep loadConfigFromEnvironment v1.1.21 -- src/ v1.1.21:src/utils/config.ts:22:export function loadConfigFromEnvironment(): Partial<LoggerConfig> {That’s the whole result. One hit, and it’s the definition of the function itself. Exported, documented, tested, and reachable by exactly nobody.
The function was fine
That’s the part that took me a minute to accept. There was no bug in it. Given a
process.env, it did the right thing:export function loadConfigFromEnvironment(): Partial<LoggerConfig> { const config: Partial<LoggerConfig> = {}; if (typeof process !== 'undefined' && process.env) { const env = process.env; if (env.LOG_LEVEL) { config.level = stringToLogLevel(env.LOG_LEVEL); } // ...LOG_FORMAT, LOG_TIMESTAMP, LOG_COLOR } return config; }Clean. Reasonable. Correct. And
createLogger()never called it, so a user could setLOG_LEVEL=debug, restart their process, and get exactly the same output they got before.The tests found this function the same way a unit test finds anything: by importing it directly.
tests/config.test.tsreferencedloadConfigFromEnvironmenteleven times across eight assertions. Every one of them passed. Not a single one of them proved a user could reach it, because none of them went through the public entry point.That’s the failure mode. A unit test that imports the unit is testing the unit. It’s not testing whether the unit is wired up.
It wasn’t the only one
While I was in there I checked
LoggerConfig.transports, declared right in the public type atsrc/core/types.ts:70:transports?: TransportConfig[];TypeScript happily accepted a transports array. Autocomplete offered it. And
NodeLoggerignored it completely, because the Node adapter built its own hardcoded Winston transport list and never looked at the config field:const logger = winston.createLogger({ transports: [ new winston.transports.Console({ /* ... */ }), ], });So you could pass
transports: [{ type: 'file', options: { filename: 'app.log' } }], get no type error, get no runtime warning, and get no file. The type system confirmed your configuration was valid. The runtime threw it away.Two different features, same shape. Declared in the public surface, absent from the code path that runs.
What the fix actually cost
I wired it up in 2.0.0 and the breaking-change note is longer than most of the feature work:
LOG_LEVEL, LOG_FORMAT, LOG_TIMESTAMP and LOG_COLOR now take effect, so a process with any of them already set logs differently after upgrading without any code change, and they override configuration passed to createLogger().
Read that carefully. Turning on a documented feature is a breaking change when the feature has been off long enough. Somebody out there has
LOG_LEVEL=debugsitting in a.envfrom a project they set up last year, inherited by a service that has been logging atinfothis whole time because the variable did nothing. They upgrade a minor version, and now their logs are ten times bigger.There’s a nastier one buried in the same note. The old boolean parsing was
env.LOG_TIMESTAMP.toLowerCase() === 'true', soLOG_TIMESTAMP=1evaluated to false. Anybody who wrote1and expectedtruewas wrong twice over, first because the parse was strict about the literal string, and second because nothing read the result anyway. The new parser acceptstrue, 1, yes, onandfalse, 0, no, off, which meansLOG_TIMESTAMP=1flips from false to true across the upgrade.Nobody could have hit that in practice. The feature was dead. But the moment it goes live, every one of those latent misconfigurations wakes up at once.
How this happens
Four things worth stealing from this:
- Grep your own exports for call sites.
git grep <exportName> -- src/on any function you believe is load-bearing. If the only hit is the definition, you found one. - A unit test proves a unit works, not that it runs. At least one test per feature should enter through the same door a user does.
- A declared field on a public type is a promise. If the runtime ignores it, the type system is lying to your users with full IDE autocomplete support.
- Turning on a dead documented feature is a breaking change. Version it like one, and write the note that tells people their existing environment now means something.
The library is at 2.5.2 now and the environment variables work. That sentence should not have taken thirteen months to become true.
Sources
- logan-logger-ts — the repo, if you want the diffs
- The 2.0.0 commit — where the wiring finally happened, breaking-change note and all
- treering — the language-neutral spec and conformance fixtures that drive the library through its public API
- The Twelve-Factor App: Config — the case for environment-based configuration, which works better when you read the environment
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
- Grep your own exports for call sites.
-
Adding Types to JSON with Dhall
A few months ago I wrote a post asking whether there’s something better than JSON. Two configuration languages that sit above JSON kept coming up: CUE and Dhall. Both give you the things JSON lacks when you author config by hand, and both compile down to plain JSON, YAML, or whatever your services actually read. I spent more time with CUE back then and never gave Dhall a real look. This post is me going back for that second look, because the one feature I kept wanting was a type system over my config.
JSON is the universal language of API payloads and config files, and I don’t want that to change. But as a format for authoring configuration by hand, it’s rough:
- No comments.
- No variables or functions, so you copy-paste the same block ten times.
- No type system, so
"8080"and8080look equally valid. - No imports, which is how you end up with a 2,000-line monolith nobody wants to touch.
The usual escape hatch is a templating engine like Jinja or Helm, or a real programming language like Python or TypeScript that spits out JSON. That works, but you’ve traded one problem for a scarier one: your config generator is now Turing-complete. It can crash, hang in an infinite loop, or reach out and read some local environment variable, and it’ll do it at 2 AM when the pipeline runs.
This is where Dhall comes in.
What is Dhall?
The short version: Dhall is JSON plus types, plus functions, plus imports. It’s a programmable, strongly-typed configuration language.
The part I actually care about is what it doesn’t have. Dhall is not Turing-complete. No arbitrary recursion, no side effects. Every Dhall program is guaranteed to terminate. You get the abstraction power of a functional language like Haskell or Elm, with the guarantee that it will never hang your build. That’s a different trade than “just write a Python script.”
The problem, in JSON
Here’s a normal
config.jsonfor a microservice:{ "serviceName": "payment-api", "port": 8080, "environment": "production", "database": { "host": "db.internal.net", "maxConnections": 50 } }Three ways can be a problem in production: someone writes
"port": "8080"and the service won’t boot, someone typos"prodution"and it silently runs in debug mode, or someone forgetsmaxConnectionsentirely and you get a null blowup at runtime. Nothing catches any of it until it’s live.The same thing, typed
In Dhall you define the shape up front. Enums, record types, default values:
-- schema.dhall let Environment = < Local | Staging | Production > let Database = { Type = { host : Text, maxConnections : Natural } , default = { maxConnections = 20 } } let Config = { Type = { serviceName : Text , port : Natural , environment : Environment , database : Database.Type } , default = { port = 8080, environment = Environment.Local } } in { Environment, Database, Config }Now you author against that schema, and you get defaults and composition for free:
-- config.dhall let Schema = ./schema.dhall let myConfig : Schema.Config.Type = Schema.Config.default // { serviceName = "payment-api" , environment = Schema.Environment.Production , database = Schema.Database.default // { host = "db.internal.net", maxConnections = 50 } } in myConfigMisspell
Production, or pass"8080"as a string, and Dhall throws a type error before a single line of JSON is generated. Hopefully the benfit is now clear; adding a type safety layer to your config files.Compiling down to JSON
You don’t ship Dhall to your services. You ship the JSON they already understand:
brew install dhall-json dhall-to-json --file config.dhallOut comes clean, boring, standard JSON. Your services never know Dhall was involved. The part that I like is the safety lives at authoring time, and the runtime artifact stays dumb.
Two features worth knowing about
Hermetic imports with hash pinning. Dhall can import from a URL, so shared utilities live in one place instead of being copy-pasted across five repos. To keep someone from swapping the file out from under you, you pin the import to a SHA-256 hash of its normalized form:
let Prelude = https://prelude.dhall-lang.org/v22.0.0/package.dhall sha256:10db4c919c25e4d262db3ed0d1d6120da3e3906673f00e3012c1d14e1963976aIf the remote content changes, the hash won’t match and the build fails. The hash above is just an example, and each Prelude version has its own, so don’t copy it by hand.
dhall freeze --inplace config.dhallcomputes the correct hashes for whatever you’ve imported and pins them automatically.Exhaustive matching with
merge. When you map a union type to output, Dhall makes you handle every variant:let getLogPrefix = \(env : Environment) -> merge { Local = "[DEV] ", Staging = "[STAGE] ", Production = "[PROD] " } envAdd a
QAvariant later, and everymergeblock that touchedEnvironmentfails to compile until you deal with it. No forgottenswitchcase slipping into production. The compiler keeps a running list of everything you now owe it.Is it worth it?
Raw JSON Dhall Type safety None, fails at runtime Static, at compile time Comments & logic No Yes Termination N/A Guaranteed Dependency pinning No SHA-256 Output Consumed directly Compiles to JSON/YAML/TOML For a two-key config file, it doesn’t make sense, but once you’re staring down Kubernetes manifests, a pile of near-identical microservice configs, or anything where a typo takes down a service, the calculus changes. You keep clean static JSON as the thing your services actually read, and you move all the ways-to-get-it-wrong to a place where a compiler catches them first.
Sources
- Dhall Language Tutorial & Cheatsheet: records, union types, default overrides,
dhall-to-json, anddhall freeze. - Dhall language standard on GitHub: the non-Turing-complete design and the semantic integrity hash spec.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Is There Something Better Than JSON?
Have you ever looked at a JSON file and thought, “There has to be something better than this”? I have.
JSON has served us well. It works with everything, and it’s human readable. It’s a decent default, don’t get me wrong, but the more you use it, you’ll find its limitations to be quite painful. So before we answer the question of whether there’s anything better, we should describe what’s actually wrong with JSON.
The Problems with JSON
First, there’s no type system. No datetimes, no real integers, no structs, no unions, no tuples. If you need types, and you almost always do, you’re on your own.
Second, JSON is simple, which sounds like a feature until you try to store anything complicated in it. You end up inventing your own schema, and the schema tooling out there (JSON Schema, etc.) gets verbose fast. Because the spec is so loose, validation can be inconsistent across implementations.
There’s more: fields can be reordered, you have to receive the entire document before you can start verifying it, and there are no comments. You can’t leave a note for the next person explaining why a config value is set a certain way. That’s a real problem for anything that lives in version control.
The Machine-Readable Alternatives
Now, there are plenty of binary serialization formats that solve some of these issues. Protobuf, Cap’n Proto, CBOR, MessagePack, BSON. They’re all interesting and have their place. But they’re machine readable, not human readable. You can’t just open one up in your editor and make sense of it. So let’s set those aside.
The question I’m more interested in is: is there something better than JSON that you can still read and edit as a text file?
It turns out there are two solid options.
Dhall
Dhall is a programmable configuration language. Think of it as JSON with all the things you wish JSON had: functions, types, and imports. You can convert JSON to Dhall and back, and it’s just a text file you can open in any editor. The name comes from a character in an old video game, and the language itself is interesting enough that it’s worth your time to explore.
CUE
CUE stands for Configure, Unify, and Execute. It’s similar to Dhall in that it fills the gaps JSON leaves behind, like types, validation, and constraints, while staying human readable. Where CUE really pulls ahead is in its feature set. You can import Protobuf definitions, generate JSON Schema, validate existing configs, and a lot more. In terms of raw capabilities, CUE has more going on than Dhall.
JSON isn’t going anywhere. But if you’re looking for something interesting to explore, check out both of these. They make great fun little side projects.