Distributed-systems
-
`updated_at` Is Not a Conflict-Resolution Strategy
In the last post we talked about the problems with a distributed system, and touched on the fact that timestamps are not as reliable as you think they are.
If you have two
updated_atfields and you compare them, how do you decide which side is the correct one?The
updated_atfield only tells you that a write happened. It doesn’t tell you the meaning, or if it was intentional. Conflict resolution is fundamentally a question about causality. Did side A intend for this change to happen? A wall clock timestamp can’t tell you the answer to that.Two independent clocks can drift, and will drift. Yes, it will get corrected by NTP occasionally. But you can’t always rely on their NTP service working. Timestamps are a fine signal that something occurred, and they’re a reasonable way for a human to sort a list and answer roughly when we think a change occurred. But if you use them as a foundation to decide what data to keep, you’re gonna end up destroying and losing data.
Things That Actually Work
The good news is the alternatives are not exotic, and you don’t need all of them.
Content hashes. Hash the meaningful content and compare hashes instead of times. This kills the metadata-edit problem outright: if the hash matches, nothing changed, no matter what the timestamp claims. It’s the highest-value change on this list and usually the easiest, because it’s a pure function of data you already have.
Version counters. A monotonic integer per record, incremented on every meaningful write. Immune to clock skew entirely, because it isn’t a clock. The cost is that somebody has to own the increment, which is straightforward with a single authority and gets harder without one.
Sync checkpoints. Record what was confirmed at the last successful sync, not just when it happened. Then the question becomes “has this changed since the last agreed state,” which is answerable, instead of “is this newer,” which is a guess.
Operation logs. Store what happened rather than only the result. Heavier, but it’s the only option that lets you reconstruct intent after the fact, and it turns “which one wins” into a question you can actually audit.
You can get most of the benefit from the first one. Hash the content, and let the timestamp go back to being a display field.
When Last-Write-Wins Is Fine
I’m not gonna lie, last write wins is often the correct engineering choice, and replacing it with something more complicated can be its own mistake. Sometimes it’s fine. If a write gets lost and the data is recoverable, that’s a trade you can live with.
If it’s a simple tool without a ton of users, adding a lot of complexity is not the way to go.
If the data is just a cache or a projection, then who cares? You can rebuild it from the authoritative source anyway.
What I’d Actually Do
Keep
updated_at. It’s useful. Sort by it, display it, log it.Just stop letting it decide things. Add a content hash and check that first, so a no-op edit stays a no-op. If a field can be written from two sides independently, give it a version counter or an explicit authority rule, and write the rule down somewhere the next person will find it.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Databases Software-development Distributed-systems Local-first Data-modeling
-
The Moment You Add Sync, You Have a Distributed System
How do you keep two sets of data in sync? Like, by definition, you now have a distributed system.
It could be something simple, syncing files or talking with a remote service somewhere. Maybe it’s not a lot of code. Initially, it might not feel like a distributed system, because there’s no cluster or consensus protocol. There’s no leader election system. You have multiple leaders that need to stay in sync.
How do you maintain state between two independent systems, when their only connection is an over-the-network connection that’s allowed to fail?
Sync can be a verb that you apply to the data on one side, but it’s also describing the negotiation that happens between two distributed systems.
Here is the set of questions you have to answer if you are trying to build a distributed system that maintains sync.
- What’s new here that isn’t there?
- What’s new there that isn’t here?
- What changed in both places since we last talked?
- What happens if we get halfway through and the connection dies?
- If I retry, do I create a duplicate?
All of these sound like problems from a paper you read about replicated state machines. Congratulations, they’re now your problem too.
Just because the request succeeded doesn’t mean that the two systems now agree.
The problems that you’re going to run into are either caused by or solved by a timestamp.
Recovering from an error state is crucial for building a durable system.
Idempotency Is the Cheapest Insurance You Can Buy
It is a guarantee, or pretty much a guarantee, that if your sync can be interrupted, it will be. Idempotency is how you ensure that the same request can be retried safely. If you run the same request twice, you either need to produce the same result or no result.
Every item on both sides of the system needs its own stable identity that each side agrees on. The create needs to always be create-if-absent, basically an upsert.
How do you decide which copy of the data has authority?
When you start needing to do resolution logic, this is where your subtle data loss can occur. Your point-in-time recovery window is likely 30 days or less, and chances are you aren’t going to go back and check the old copy that is about to expire.
Last write wins is the default because it’s easy, and it’s what everybody assumes. It works when there aren’t a whole lot of writes and when one side is clearly the primary. It breaks down when the data can’t be replayed safely.
Do your timestamps actually mean what you think they mean? Can you trust time? It’s complicated to get correct. And if the difference between two timestamps is very small, and the drift is larger than the difference, problems occur.
Things to look into for later: CRDTs, vector clocks, operational transforms.
Chances are these are not the right answer for a personal tool that you’re building on the weekends. It’s good to have discipline and understand the solutions we’ve come up with for resolving synchronization problems. At the end of the day, you’re just gonna want something that works.
You have a distributed system. It has one user and it runs on a laptop, but it has all the failure modes, and it doesn’t care that you didn’t mean to build one.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Software-development Architecture Distributed-systems Local-first Apis
-
What Temporal Actually Does (And Why You'd Want It)
Building a multi-step process across microservices usually goes something like this. You wire up a message queue, add retry logic, build a state machine backed by a Postgres
statuscolumn, throw in some cron jobs, and pray. It sounds complicated because it is.Temporal is an open-source “durable execution” system that replaces all of that duct tape with a single, opinionated framework. Lets break it down.
Workflows and Activities
Temporal splits your application into two concepts:
- Workflows are your business logic, written in standard code (Go, Python, TypeScript) using a Temporal SDK. They must be deterministic. They define the order of operations, branching, loops, and error handling.
- Activities are the actual tasks your services perform. HTTP requests, database writes, external API calls. Activities are where the non-deterministic, real-world work happens.
When a workflow runs, it executes on your own worker services. Every time it schedules an activity, starts a timer, or completes a step, the Temporal Server records that event internally. If the worker crashes, another worker picks it up, replays the workflow’s event history to the exact point of failure, and resumes. No data loss. No half-finished state.
All of all the things that you would have to build yourself simplified Into A framework that handles it for you.
What It Replaces
Without something like Temporal, teams generally land in one of two camps:
- Choreography (event-driven): Services emit and listen to events through a message broker like Kafka or RabbitMQ. Highly decoupled, sure. But in practice it turns into a pinball machine. There’s no single place to understand the flow of a business transaction. Debugging becomes detective work across dozens of services and topics.
- Ad-hoc orchestration: You build a custom state machine with a database, message queues, background workers, and cron jobs. Then you write a ton of boilerplate for retries, dead-letter queues, and idempotency. Every team ends up building a slightly different version of this, and none of them are great.
Temporal gives you the reliability of a custom state machine without making you build and maintain one.
Why It’s Worth Looking At
A few things stand out:
- Durable sleep. A workflow can execute
sleep(30_DAYS). Temporal suspends the execution, frees the worker’s resources, and wakes it back up a month later exactly where it left off. Hard to do with a cron job. - Built-in resiliency. Exponential backoffs, timeouts, and retry policies are configured on the activity invocation. You’re not writing custom
whileloops andtry/catchblocks to handle network jitter. - Centralized observability. Instead of piecing together distributed traces or searching through logs to figure out why step 4 of 7 failed, the Temporal UI shows the exact execution state of every workflow. Inputs, outputs, errors, all in one place.
- Code over configuration. Unlike AWS Step Functions or YAML-heavy tools like Airflow, you write workflows in a real programming language. You can unit test them, store them in version control, and run them through your normal CI/CD pipeline.
That last point is worth reading and thinking through again. If your orchestration logic lives in code, it gets all the benefits code gets. Reviews, tests, refactoring, IDE support. Visual workflow builders look great in demos, but they don’t scale the way code does.
Should You Use It?
Temporal isn’t free in terms of operational complexity. You’re running the Temporal Server (or paying for Temporal Cloud), and your team needs to understand the replay model and determinism constraints. It’s not something you bolt on to a simple CRUD app.
But if you’re managing distributed transactions with queues, cron jobs, and hand-rolled state machines, Temporal is worth a serious look. It takes the hardest parts of that problem and makes them someone else’s. Durability, retries, observability. All handled.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].