How do you keep two sets of data in sync? Like, by definition, you now have a distributed system.

It could be something simple, syncing files or talking with a remote service somewhere. Maybe it’s not a lot of code. Initially, it might not feel like a distributed system, because there’s no cluster or consensus protocol. There’s no leader election system. You have multiple leaders that need to stay in sync.

How do you maintain state between two independent systems, when their only connection is an over-the-network connection that’s allowed to fail?

Sync can be a verb that you apply to the data on one side, but it’s also describing the negotiation that happens between two distributed systems.

Here is the set of questions you have to answer if you are trying to build a distributed system that maintains sync.

• What’s new here that isn’t there?
• What’s new there that isn’t here?
• What changed in both places since we last talked?
• What happens if we get halfway through and the connection dies?
• If I retry, do I create a duplicate?

All of these sound like problems from a paper you read about replicated state machines. Congratulations, they’re now your problem too.

Just because the request succeeded doesn’t mean that the two systems now agree.

The problems that you’re going to run into are either caused by or solved by a timestamp.

Recovering from an error state is crucial for building a durable system.

Idempotency Is the Cheapest Insurance You Can Buy

It is a guarantee, or pretty much a guarantee, that if your sync can be interrupted, it will be. Idempotency is how you ensure that the same request can be retried safely. If you run the same request twice, you either need to produce the same result or no result.

Every item on both sides of the system needs its own stable identity that each side agrees on. The create needs to always be create-if-absent, basically an upsert.

How do you decide which copy of the data has authority?

When you start needing to do resolution logic, this is where your subtle data loss can occur. Your point-in-time recovery window is likely 30 days or less, and chances are you aren’t going to go back and check the old copy that is about to expire.

Last write wins is the default because it’s easy, and it’s what everybody assumes. It works when there aren’t a whole lot of writes and when one side is clearly the primary. It breaks down when the data can’t be replayed safely.

Do your timestamps actually mean what you think they mean? Can you trust time? It’s complicated to get correct. And if the difference between two timestamps is very small, and the drift is larger than the difference, problems occur.

Things to look into for later: CRDTs, vector clocks, operational transforms.

Chances are these are not the right answer for a personal tool that you’re building on the weekends. It’s good to have discipline and understand the solutions we’ve come up with for resolving synchronization problems. At the end of the day, you’re just gonna want something that works.

You have a distributed system. It has one user and it runs on a laptop, but it has all the failure modes, and it doesn’t care that you didn’t mean to build one.

I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].