Databases
-
`updated_at` Is Not a Conflict-Resolution Strategy
In the last post we talked about the problems with a distributed system, and touched on the fact that timestamps are not as reliable as you think they are.
If you have two
updated_atfields and you compare them, how do you decide which side is the correct one?The
updated_atfield only tells you that a write happened. It doesn’t tell you the meaning, or if it was intentional. Conflict resolution is fundamentally a question about causality. Did side A intend for this change to happen? A wall clock timestamp can’t tell you the answer to that.Two independent clocks can drift, and will drift. Yes, it will get corrected by NTP occasionally. But you can’t always rely on their NTP service working. Timestamps are a fine signal that something occurred, and they’re a reasonable way for a human to sort a list and answer roughly when we think a change occurred. But if you use them as a foundation to decide what data to keep, you’re gonna end up destroying and losing data.
Things That Actually Work
The good news is the alternatives are not exotic, and you don’t need all of them.
Content hashes. Hash the meaningful content and compare hashes instead of times. This kills the metadata-edit problem outright: if the hash matches, nothing changed, no matter what the timestamp claims. It’s the highest-value change on this list and usually the easiest, because it’s a pure function of data you already have.
Version counters. A monotonic integer per record, incremented on every meaningful write. Immune to clock skew entirely, because it isn’t a clock. The cost is that somebody has to own the increment, which is straightforward with a single authority and gets harder without one.
Sync checkpoints. Record what was confirmed at the last successful sync, not just when it happened. Then the question becomes “has this changed since the last agreed state,” which is answerable, instead of “is this newer,” which is a guess.
Operation logs. Store what happened rather than only the result. Heavier, but it’s the only option that lets you reconstruct intent after the fact, and it turns “which one wins” into a question you can actually audit.
You can get most of the benefit from the first one. Hash the content, and let the timestamp go back to being a display field.
When Last-Write-Wins Is Fine
I’m not gonna lie, last write wins is often the correct engineering choice, and replacing it with something more complicated can be its own mistake. Sometimes it’s fine. If a write gets lost and the data is recoverable, that’s a trade you can live with.
If it’s a simple tool without a ton of users, adding a lot of complexity is not the way to go.
If the data is just a cache or a projection, then who cares? You can rebuild it from the authoritative source anyway.
What I’d Actually Do
Keep
updated_at. It’s useful. Sort by it, display it, log it.Just stop letting it decide things. Add a content hash and check that first, so a no-op edit stays a no-op. If a field can be written from two sides independently, give it a version counter or an explicit authority rule, and write the rule down somewhere the next person will find it.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Databases Software-development Distributed-systems Local-first Data-modeling
-
Your Local File Should Not Have to Argue With Your Database
Sync bugs usually all start the same way. Two copies of something, both of them mostly right, and no written rule about which one wins.
The problems occur when you don’t notice The bug. When the file says one thing and the database says another. It’s not a problem until it is. And then you have to spend time figuring the why and when’s of the drift.
So let’s talk about authority. Not storage, not sync, not “where does the data live.” Authority. Which copy is allowed to be right when two copies disagree.
The Question Nobody Writes Down
Most systems that hold the same data in two places never actually decide this. The decision gets made accidentally, by whichever code path happened to run last, and then it gets re-made differently by the next feature.
The failure is duplication without a stated rule.
Here’s a concrete version. I run a content pipeline for this blog. Posts are Markdown files with YAML frontmatter sitting in a directory. There’s also a Turso database holding metadata about those same posts. Two copies of what looks like the same information.
Ask the naive question, “which one is the source of truth,” and you get a bad answer, because the honest answer is neither, and both, depending on the field.
Split Authority by Field, Not by Store
You might try to pick an authoritative source based on store. Files win, or the database wins. But the useful granularity is usually the field.
In my pipeline it breaks down like this:
- Post content and tags: the Markdown file wins. The frontmatter is authoritative. If the database has a different tag list, the database is wrong, and it gets rebuilt from the file.
- Scheduling: the database wins. What time a post goes out, what slot it holds, whether it’s been claimed. The file does not get a vote.
Those are different answers for the same post, and that’s fine, because each one is written down and each one has a reason.
The content lives in the file because content is the thing I edit by hand, in an editor, with Git history behind it. I want
git logto be the real record of what changed. Putting that in a database would mean my writing history lives somewhere that is harder to access.The schedule lives in the database because scheduling is a coordination problem. It needs uniqueness constraints, it needs to answer “what’s in the 10am slot on Tuesday,” and it needs to do that without me parsing 241 files. A database is genuinely better at that. It just isn’t better at holding prose.
A Database Can Be Useful Without Being Authoritative
I think there’s a reflex where adding a database feels like promoting the data into it. You put the posts in Postgres and now Postgres is where posts are.
It doesn’t have to work that way. A database can be a query layer over data that lives somewhere else, and that’s a completely respectable job. Indexes, joins, counts, “show me every post tagged local-first published before June.” All of that is worth having, and none of it requires the database to be the authority.
The test I use: if I deleted the database right now, what would I lose forever?
For me, it would be the scheduling state because that’s what I put in the database. The important thing is I wouldn’t lose a single word that I’ve written. Every post would still be in a directory. This choice is deliberate.
It’s easy for the database to become a Cache and not an authoritative source.
What Should Happen When They Disagree
If you have documented your authoritative source, then the disagreements stops becoming a crisis, and it just is a routine. Resolution event
You should be able to rebuild it. There should be nothing to decide. The decision is documented and how you resolve conflicts. Just depends on. Which authoritative source owns Which s segment of your data?
In my case, there’s actually a third authoritative source, and that’s the remote blog system that hands back an ID every time I schedule a new post.
So, this is totally fine if you pick the authority at the field level and Document that decision to prevent trip-ups in the future.
Your files and your database shouldn’t be arguing, all it requires is a bit of planning.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Databases Software-development Architecture Local-first Data-modeling
-
pgvector vs Pinecone: You Probably Don't Need a Separate Vector Database
Every time someone starts building a RAG pipeline, the same question will come up: do I need a “real” vector database like Pinecone, or can I just use pgvector with the Postgres I already have?
I can imagine teams agonizing over this decision for weeks. So maybe this will save you some time?
The Case for Staying Put
If you already have a PostgreSQL instance in your stack, adding
pgvectoris almost always the right first move.You manage one stateful service instead of two. Your existing backup strategy, monitoring, and security all stay the same. Your vector embeddings live next to your metadata, so you get ACID compliance and standard SQL joins. No syncing between two data stores. No eventual consistency headaches.
Performance? From what I found, for datasets under a few million vectors,
pgvectorwith HNSW indexes is fast. Really fast. It satisfies the latency requirements of most applications without breaking a sweat.And you’re not paying for another SaaS subscription…
When Pinecone Actually Makes Sense
Pinecone is a purpose-built vector database designed for high-dimensional data at massive scale. It’s serverless and fully managed.
If you’re dealing with hundreds of millions or billions of vectors, a specialized engine handles memory and disk I/O for similarity searches more efficiently than Postgres can. Pinecone also gives you native namespace support, metadata filtering optimized for vector search, and live index updates that are faster than re-indexing a large Postgres table.
Those are real advantages. At a certain scale.
The Decision Is Simpler Than You Think
Stay with Postgres + pgvector if:
- You want to minimize infra sprawl and moving parts
- Your vector dataset is under 5 to 10 million records
- You rely on relational joins between vectors and other business data
- You have existing observability and DBA expertise for Postgres
Consider Pinecone if:
- Your Postgres instance needs massive, expensive vertical scaling just to keep the vector index in memory
- You don’t want to tune HNSW parameters,
mmapsettings, or vacuuming schedules for large vector tables - You need sub-millisecond similarity search at a scale where Postgres starts to struggle
That is what I would use to make that decision.
Most teams are probably nowhere near the scale where Pinecone becomes necessary. They have a few hundred thousand vectors, maybe a million or two. Postgres handles that without flinching. Adding a separate managed vector database at that point is just adding operational complexity for no measurable benefit.
The trap is thinking you need to “plan ahead” for scale you don’t have yet. You can always migrate later if you actually hit the ceiling. Moving from pgvector to Pinecone is a well-documented path. But moving from two services back to one because you overengineered your stack? That’s a conversation nobody wants to have.
Start with what you have. Add complexity when the numbers force you to, not when a vendor’s marketing page makes you nervous.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
-
Everything Is Eventually a Database Problem
I think there’s a saying that goes something like “code is ephemeral, but data is forever.” That’s never been more true than right now. Code is easier than ever to create, anyone can spin up a working app with an AI agent and minimal experience. But your data structure? That’s the thing that sticks around and haunts you.
Data modeling is one of those topics that doesn’t get enough attention, especially given how critical it is. You need to understand how your data is stored, how to structure it, and what tradeoffs you’re making in how you access it. Get it right early, and your code stays elegant and straightforward. Get it wrong, and your codebase becomes a forever series of workarounds…
Microservices Won’t Save You
For teams moving from a monolith to microservices, if the data stays tightly coupled, you don’t really have microservices; you have a distributed monolith with extra network hops.
Yes, data can be coupled just like code can be coupled. If all your different services are still hitting the same database with the same schema, you have a problem. You need separate data structures for your services, not a monolithic architecture hiding behind a microservices facade.
The Caching Trap
So what happens when you have a lot of data and your queries get slow? You’ve done all the easy stuff; optimized queries, added indexes, followed best practices. But things are still slow.
Every senior engineer’s first instinct is the same: “Let’s add Redis in front of it.” Or “more read replicas.” And sure, that works, but you have just added complexity and now you have to deal with cache invalidation.
What happens when you have stale data? How do you recache current data, and when does that happen?
Are you caching on the browser side too? Understanding where data can be cached and how to invalidate it is another genuinely difficult problem to solve. You’re just trading one set of problems for a different set of problems.
You Can’t Predict Every Future Question
If you’re selling things on the internet, chances are, you will care about event sourcing at some point. A lot of interesting business problems don’t care about the current state of a user, they care about the intent and history. So how you store intent and history is probably different from your ACID-compliant Postgres table that you’ve worked hard to normalize.
You can get your data structure perfect for displaying products and processing sales, then run into a completely new set of requirements that changes everything about how your data needs to be structured.
It’s genuinely hard to foresee all the potential questions you’ll need to answer in the future.
Why This Matters Now
Everything you do on a computer stores data somewhere, it’s just a matter of persistence.
Which is why; everything software-related is eventually a database problem.
Data modeling isn’t glamorous, but getting it right is the difference between a system that scales gracefully and one that fights you every step of the way.
-
Just discovered DB Pro, a new desktop app for SQLite and LibSQL databases. Looks pretty promising. Meanwhile, I’m still waiting for DataGrip to get proper LibSQL support. Come on JetBrains, just give me Turso already!