Architecture
-
The Moment You Add Sync, You Have a Distributed System
How do you keep two sets of data in sync? Like, by definition, you now have a distributed system.
It could be something simple, syncing files or talking with a remote service somewhere. Maybe it’s not a lot of code. Initially, it might not feel like a distributed system, because there’s no cluster or consensus protocol. There’s no leader election system. You have multiple leaders that need to stay in sync.
How do you maintain state between two independent systems, when their only connection is an over-the-network connection that’s allowed to fail?
Sync can be a verb that you apply to the data on one side, but it’s also describing the negotiation that happens between two distributed systems.
Here is the set of questions you have to answer if you are trying to build a distributed system that maintains sync.
- What’s new here that isn’t there?
- What’s new there that isn’t here?
- What changed in both places since we last talked?
- What happens if we get halfway through and the connection dies?
- If I retry, do I create a duplicate?
All of these sound like problems from a paper you read about replicated state machines. Congratulations, they’re now your problem too.
Just because the request succeeded doesn’t mean that the two systems now agree.
The problems that you’re going to run into are either caused by or solved by a timestamp.
Recovering from an error state is crucial for building a durable system.
Idempotency Is the Cheapest Insurance You Can Buy
It is a guarantee, or pretty much a guarantee, that if your sync can be interrupted, it will be. Idempotency is how you ensure that the same request can be retried safely. If you run the same request twice, you either need to produce the same result or no result.
Every item on both sides of the system needs its own stable identity that each side agrees on. The create needs to always be create-if-absent, basically an upsert.
How do you decide which copy of the data has authority?
When you start needing to do resolution logic, this is where your subtle data loss can occur. Your point-in-time recovery window is likely 30 days or less, and chances are you aren’t going to go back and check the old copy that is about to expire.
Last write wins is the default because it’s easy, and it’s what everybody assumes. It works when there aren’t a whole lot of writes and when one side is clearly the primary. It breaks down when the data can’t be replayed safely.
Do your timestamps actually mean what you think they mean? Can you trust time? It’s complicated to get correct. And if the difference between two timestamps is very small, and the drift is larger than the difference, problems occur.
Things to look into for later: CRDTs, vector clocks, operational transforms.
Chances are these are not the right answer for a personal tool that you’re building on the weekends. It’s good to have discipline and understand the solutions we’ve come up with for resolving synchronization problems. At the end of the day, you’re just gonna want something that works.
You have a distributed system. It has one user and it runs on a laptop, but it has all the failure modes, and it doesn’t care that you didn’t mean to build one.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Software-development Architecture Distributed-systems Local-first Apis
-
Your Local File Should Not Have to Argue With Your Database
Sync bugs usually all start the same way. Two copies of something, both of them mostly right, and no written rule about which one wins.
The problems occur when you don’t notice The bug. When the file says one thing and the database says another. It’s not a problem until it is. And then you have to spend time figuring the why and when’s of the drift.
So let’s talk about authority. Not storage, not sync, not “where does the data live.” Authority. Which copy is allowed to be right when two copies disagree.
The Question Nobody Writes Down
Most systems that hold the same data in two places never actually decide this. The decision gets made accidentally, by whichever code path happened to run last, and then it gets re-made differently by the next feature.
The failure is duplication without a stated rule.
Here’s a concrete version. I run a content pipeline for this blog. Posts are Markdown files with YAML frontmatter sitting in a directory. There’s also a Turso database holding metadata about those same posts. Two copies of what looks like the same information.
Ask the naive question, “which one is the source of truth,” and you get a bad answer, because the honest answer is neither, and both, depending on the field.
Split Authority by Field, Not by Store
You might try to pick an authoritative source based on store. Files win, or the database wins. But the useful granularity is usually the field.
In my pipeline it breaks down like this:
- Post content and tags: the Markdown file wins. The frontmatter is authoritative. If the database has a different tag list, the database is wrong, and it gets rebuilt from the file.
- Scheduling: the database wins. What time a post goes out, what slot it holds, whether it’s been claimed. The file does not get a vote.
Those are different answers for the same post, and that’s fine, because each one is written down and each one has a reason.
The content lives in the file because content is the thing I edit by hand, in an editor, with Git history behind it. I want
git logto be the real record of what changed. Putting that in a database would mean my writing history lives somewhere that is harder to access.The schedule lives in the database because scheduling is a coordination problem. It needs uniqueness constraints, it needs to answer “what’s in the 10am slot on Tuesday,” and it needs to do that without me parsing 241 files. A database is genuinely better at that. It just isn’t better at holding prose.
A Database Can Be Useful Without Being Authoritative
I think there’s a reflex where adding a database feels like promoting the data into it. You put the posts in Postgres and now Postgres is where posts are.
It doesn’t have to work that way. A database can be a query layer over data that lives somewhere else, and that’s a completely respectable job. Indexes, joins, counts, “show me every post tagged local-first published before June.” All of that is worth having, and none of it requires the database to be the authority.
The test I use: if I deleted the database right now, what would I lose forever?
For me, it would be the scheduling state because that’s what I put in the database. The important thing is I wouldn’t lose a single word that I’ve written. Every post would still be in a directory. This choice is deliberate.
It’s easy for the database to become a Cache and not an authoritative source.
What Should Happen When They Disagree
If you have documented your authoritative source, then the disagreements stops becoming a crisis, and it just is a routine. Resolution event
You should be able to rebuild it. There should be nothing to decide. The decision is documented and how you resolve conflicts. Just depends on. Which authoritative source owns Which s segment of your data?
In my case, there’s actually a third authoritative source, and that’s the remote blog system that hands back an ID every time I schedule a new post.
So, this is totally fine if you pick the authority at the field level and Document that decision to prevent trip-ups in the future.
Your files and your database shouldn’t be arguing, all it requires is a bit of planning.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Databases Software-development Architecture Local-first Data-modeling
-
Everything Is Eventually a Database Problem
I think there’s a saying that goes something like “code is ephemeral, but data is forever.” That’s never been more true than right now. Code is easier than ever to create, anyone can spin up a working app with an AI agent and minimal experience. But your data structure? That’s the thing that sticks around and haunts you.
Data modeling is one of those topics that doesn’t get enough attention, especially given how critical it is. You need to understand how your data is stored, how to structure it, and what tradeoffs you’re making in how you access it. Get it right early, and your code stays elegant and straightforward. Get it wrong, and your codebase becomes a forever series of workarounds…
Microservices Won’t Save You
For teams moving from a monolith to microservices, if the data stays tightly coupled, you don’t really have microservices; you have a distributed monolith with extra network hops.
Yes, data can be coupled just like code can be coupled. If all your different services are still hitting the same database with the same schema, you have a problem. You need separate data structures for your services, not a monolithic architecture hiding behind a microservices facade.
The Caching Trap
So what happens when you have a lot of data and your queries get slow? You’ve done all the easy stuff; optimized queries, added indexes, followed best practices. But things are still slow.
Every senior engineer’s first instinct is the same: “Let’s add Redis in front of it.” Or “more read replicas.” And sure, that works, but you have just added complexity and now you have to deal with cache invalidation.
What happens when you have stale data? How do you recache current data, and when does that happen?
Are you caching on the browser side too? Understanding where data can be cached and how to invalidate it is another genuinely difficult problem to solve. You’re just trading one set of problems for a different set of problems.
You Can’t Predict Every Future Question
If you’re selling things on the internet, chances are, you will care about event sourcing at some point. A lot of interesting business problems don’t care about the current state of a user, they care about the intent and history. So how you store intent and history is probably different from your ACID-compliant Postgres table that you’ve worked hard to normalize.
You can get your data structure perfect for displaying products and processing sales, then run into a completely new set of requirements that changes everything about how your data needs to be structured.
It’s genuinely hard to foresee all the potential questions you’ll need to answer in the future.
Why This Matters Now
Everything you do on a computer stores data somewhere, it’s just a matter of persistence.
Which is why; everything software-related is eventually a database problem.
Data modeling isn’t glamorous, but getting it right is the difference between a system that scales gracefully and one that fights you every step of the way.