Local-first
-
`updated_at` Is Not a Conflict-Resolution Strategy
In the last post we talked about the problems with a distributed system, and touched on the fact that timestamps are not as reliable as you think they are.
If you have two
updated_atfields and you compare them, how do you decide which side is the correct one?The
updated_atfield only tells you that a write happened. It doesn’t tell you the meaning, or if it was intentional. Conflict resolution is fundamentally a question about causality. Did side A intend for this change to happen? A wall clock timestamp can’t tell you the answer to that.Two independent clocks can drift, and will drift. Yes, it will get corrected by NTP occasionally. But you can’t always rely on their NTP service working. Timestamps are a fine signal that something occurred, and they’re a reasonable way for a human to sort a list and answer roughly when we think a change occurred. But if you use them as a foundation to decide what data to keep, you’re gonna end up destroying and losing data.
Things That Actually Work
The good news is the alternatives are not exotic, and you don’t need all of them.
Content hashes. Hash the meaningful content and compare hashes instead of times. This kills the metadata-edit problem outright: if the hash matches, nothing changed, no matter what the timestamp claims. It’s the highest-value change on this list and usually the easiest, because it’s a pure function of data you already have.
Version counters. A monotonic integer per record, incremented on every meaningful write. Immune to clock skew entirely, because it isn’t a clock. The cost is that somebody has to own the increment, which is straightforward with a single authority and gets harder without one.
Sync checkpoints. Record what was confirmed at the last successful sync, not just when it happened. Then the question becomes “has this changed since the last agreed state,” which is answerable, instead of “is this newer,” which is a guess.
Operation logs. Store what happened rather than only the result. Heavier, but it’s the only option that lets you reconstruct intent after the fact, and it turns “which one wins” into a question you can actually audit.
You can get most of the benefit from the first one. Hash the content, and let the timestamp go back to being a display field.
When Last-Write-Wins Is Fine
I’m not gonna lie, last write wins is often the correct engineering choice, and replacing it with something more complicated can be its own mistake. Sometimes it’s fine. If a write gets lost and the data is recoverable, that’s a trade you can live with.
If it’s a simple tool without a ton of users, adding a lot of complexity is not the way to go.
If the data is just a cache or a projection, then who cares? You can rebuild it from the authoritative source anyway.
What I’d Actually Do
Keep
updated_at. It’s useful. Sort by it, display it, log it.Just stop letting it decide things. Add a content hash and check that first, so a no-op edit stays a no-op. If a field can be written from two sides independently, give it a version counter or an explicit authority rule, and write the rule down somewhere the next person will find it.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Databases Software-development Distributed-systems Local-first Data-modeling
-
The Moment You Add Sync, You Have a Distributed System
How do you keep two sets of data in sync? Like, by definition, you now have a distributed system.
It could be something simple, syncing files or talking with a remote service somewhere. Maybe it’s not a lot of code. Initially, it might not feel like a distributed system, because there’s no cluster or consensus protocol. There’s no leader election system. You have multiple leaders that need to stay in sync.
How do you maintain state between two independent systems, when their only connection is an over-the-network connection that’s allowed to fail?
Sync can be a verb that you apply to the data on one side, but it’s also describing the negotiation that happens between two distributed systems.
Here is the set of questions you have to answer if you are trying to build a distributed system that maintains sync.
- What’s new here that isn’t there?
- What’s new there that isn’t here?
- What changed in both places since we last talked?
- What happens if we get halfway through and the connection dies?
- If I retry, do I create a duplicate?
All of these sound like problems from a paper you read about replicated state machines. Congratulations, they’re now your problem too.
Just because the request succeeded doesn’t mean that the two systems now agree.
The problems that you’re going to run into are either caused by or solved by a timestamp.
Recovering from an error state is crucial for building a durable system.
Idempotency Is the Cheapest Insurance You Can Buy
It is a guarantee, or pretty much a guarantee, that if your sync can be interrupted, it will be. Idempotency is how you ensure that the same request can be retried safely. If you run the same request twice, you either need to produce the same result or no result.
Every item on both sides of the system needs its own stable identity that each side agrees on. The create needs to always be create-if-absent, basically an upsert.
How do you decide which copy of the data has authority?
When you start needing to do resolution logic, this is where your subtle data loss can occur. Your point-in-time recovery window is likely 30 days or less, and chances are you aren’t going to go back and check the old copy that is about to expire.
Last write wins is the default because it’s easy, and it’s what everybody assumes. It works when there aren’t a whole lot of writes and when one side is clearly the primary. It breaks down when the data can’t be replayed safely.
Do your timestamps actually mean what you think they mean? Can you trust time? It’s complicated to get correct. And if the difference between two timestamps is very small, and the drift is larger than the difference, problems occur.
Things to look into for later: CRDTs, vector clocks, operational transforms.
Chances are these are not the right answer for a personal tool that you’re building on the weekends. It’s good to have discipline and understand the solutions we’ve come up with for resolving synchronization problems. At the end of the day, you’re just gonna want something that works.
You have a distributed system. It has one user and it runs on a laptop, but it has all the failure modes, and it doesn’t care that you didn’t mean to build one.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Software-development Architecture Distributed-systems Local-first Apis
-
Your Local File Should Not Have to Argue With Your Database
Sync bugs usually all start the same way. Two copies of something, both of them mostly right, and no written rule about which one wins.
The problems occur when you don’t notice The bug. When the file says one thing and the database says another. It’s not a problem until it is. And then you have to spend time figuring the why and when’s of the drift.
So let’s talk about authority. Not storage, not sync, not “where does the data live.” Authority. Which copy is allowed to be right when two copies disagree.
The Question Nobody Writes Down
Most systems that hold the same data in two places never actually decide this. The decision gets made accidentally, by whichever code path happened to run last, and then it gets re-made differently by the next feature.
The failure is duplication without a stated rule.
Here’s a concrete version. I run a content pipeline for this blog. Posts are Markdown files with YAML frontmatter sitting in a directory. There’s also a Turso database holding metadata about those same posts. Two copies of what looks like the same information.
Ask the naive question, “which one is the source of truth,” and you get a bad answer, because the honest answer is neither, and both, depending on the field.
Split Authority by Field, Not by Store
You might try to pick an authoritative source based on store. Files win, or the database wins. But the useful granularity is usually the field.
In my pipeline it breaks down like this:
- Post content and tags: the Markdown file wins. The frontmatter is authoritative. If the database has a different tag list, the database is wrong, and it gets rebuilt from the file.
- Scheduling: the database wins. What time a post goes out, what slot it holds, whether it’s been claimed. The file does not get a vote.
Those are different answers for the same post, and that’s fine, because each one is written down and each one has a reason.
The content lives in the file because content is the thing I edit by hand, in an editor, with Git history behind it. I want
git logto be the real record of what changed. Putting that in a database would mean my writing history lives somewhere that is harder to access.The schedule lives in the database because scheduling is a coordination problem. It needs uniqueness constraints, it needs to answer “what’s in the 10am slot on Tuesday,” and it needs to do that without me parsing 241 files. A database is genuinely better at that. It just isn’t better at holding prose.
A Database Can Be Useful Without Being Authoritative
I think there’s a reflex where adding a database feels like promoting the data into it. You put the posts in Postgres and now Postgres is where posts are.
It doesn’t have to work that way. A database can be a query layer over data that lives somewhere else, and that’s a completely respectable job. Indexes, joins, counts, “show me every post tagged local-first published before June.” All of that is worth having, and none of it requires the database to be the authority.
The test I use: if I deleted the database right now, what would I lose forever?
For me, it would be the scheduling state because that’s what I put in the database. The important thing is I wouldn’t lose a single word that I’ve written. Every post would still be in a directory. This choice is deliberate.
It’s easy for the database to become a Cache and not an authoritative source.
What Should Happen When They Disagree
If you have documented your authoritative source, then the disagreements stops becoming a crisis, and it just is a routine. Resolution event
You should be able to rebuild it. There should be nothing to decide. The decision is documented and how you resolve conflicts. Just depends on. Which authoritative source owns Which s segment of your data?
In my case, there’s actually a third authoritative source, and that’s the remote blog system that hands back an ID every time I schedule a new post.
So, this is totally fine if you pick the authority at the field level and Document that decision to prevent trip-ups in the future.
Your files and your database shouldn’t be arguing, all it requires is a bit of planning.
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
Databases Software-development Architecture Local-first Data-modeling
-
Run Your Whole Agent Stack on a $5 Box
I SSH’d into my home server this afternoon and ran
docker statson the memory layer that every one of my coding agent sessions talks to. Here’s what came back:mem0-qdrant 28.09MiB / 60.75GiB 2.13% mem0-neo4j 612.7MiB / 60.75GiB 0.77%640 megabytes. Vector store and graph store, both up for three weeks straight, serving every
rememberandrecallcall my agents make. The entire persistent memory for my AI tooling uses less RAM than one Chrome tab with Figma open.So let’s talk about why you’re paying a monthly subscription for this.
Local-First, Not Local-Only
I want to be precise here, because “self-hosted AI” has become a phrase people use to mean nine different things.
My setup is local-first, not local-only. The state lives on my hardware. The memories, the embeddings, the graph relationships, everything my agents have learned about my projects, all of it sits on a box I own, in a Docker volume I can
tarand carry away. Nobody can deprecate it, price-hike it, or sunset it.The inference does not. My embedder points at Mistral’s managed API. I’ll get to why, and how to swap it, but I’m not going to pretend otherwise in a post about self-hosting.
That embedder is the only thing that leaves my network, and only when something actually gets embedded, so when writing a memory and searching for one. Listing, deleting, and every graph operation are local with zero API calls.
State is what you can’t get back. Compute is a commodity you rent by the token. Losing access to an API means switching providers. Losing two years of accumulated project context means starting over.
The Four Pieces
Qdrant is semantic search. When I ask what it remembers about my package manager preferences, Qdrant turns that into a similarity query and hands back the relevant memories. It’s Rust, it’s fast, and at 28MB resident it’s essentially free to run. One gotcha: vector dimensions are fixed when the collection is created. Swap embedding models and you need a new collection, not a migration. I learned that the annoying way.
Neo4j is the graph store. Vectors are great at “find me things that sound like this” and bad at “what depends on what.” The graph holds explicit subject-predicate-object facts, so
project Xbuilt_withPython 3.13is a traversable edge instead of a fuzzy match. It’s the heavy one at 613MB, but it’s a JVM, so that’s mostly heap floor rather than working set. If you’re squeezing onto the smallest possible VPS, interrogate this one first.mem0 is the orchestration on top: what gets extracted from a conversation, what gets deduped against existing memories, what gets written where. That’s the difference between a database and a memory system.
The MCP server is what makes any of it useful. A small Go binary that speaks Model Context Protocol over stdio to Claude Code, exposing eight tools:
remember,recall,list_memories,forget,memory_stats,add_relation,recall_related,forget_relation.The topology is deliberately boring:
Mac Home server ┌──────────────┐ ┌──────────┐ ┌─────────────────┐ │ Claude Code │◄─►│ mem0-mcp │ LAN │ Qdrant + Neo4j │ │ │ │ (Go) │───────►│ │ └──────────────┘ └──────────┘ └─────────────────┘ stdio HTTP + boltClient binary on my laptop, containers on a box. No cloud in the middle, no account, no dashboard, no seat license.
The Ansible Role Is the Whole Argument
Anyone can
docker compose upa stack once. That’s a weekend, not infrastructure. What makes this real is that it’s a role in a repo, and rebuilding it on a fresh box is one command:ansible-playbook -i common_hosts home.yml --tags mem0That role does the unglamorous work:
- Installs a read-only deploy key scoped to exactly one repo, with an SSH
Hostalias so it can’t collide with my personal GitHub key - Clones and updates the source at a pinned branch
- Templates a
.envwith secrets pulled from Ansible Vault,no_log: trueso nothing leaks into terminal output on a--diffrun - Brings up the compose stack with
remove_orphans: true, so when I dropped a service upstream, the stale container went with it instead of lingering forever
Be careful with your secrets and how you are creating your .env files!
The Honest Part About the API Key
I self-host the state and rent the inference. Two reasons.
The first reason is speed. I ran embeddings locally before this, on CPU, and it was miserable: roughly 87 seconds to embed 32 memories, against about 2 seconds through a hosted API. That is a 45x difference on an operation sitting directly in the path of every
rememberandrecall. A good model on a CPU is still a slow model, and this was never a quality problem.The second is that embeddings have gotten cheap enough that not worth the time to setup your own embedding service. Mistral charges $0.10 per million tokens for
mistral-embed. Google’sgemini-embedding-001is $0.15 per million, halved on their batch API. Both are good models. Both bill you.Cloudflare is the worth knowing about if you’d rather not pay at all. Workers AI includes 10,000 neurons per day free, on the free plan as well as the paid one. Neurons are their normalized compute unit, and
bge-m3costs 1,075 of them per million input tokens — so that daily allowance is roughly nine million tokens a day, at no cost. Past it you’re at $0.012 per million, which is an order of magnitude under the paid competition. For a personal memory layer, nine million tokens a day is not a trial. It’s just free.One detail if you’re swapping:
bge-m3emits 1024-dimension vectors, the same asmistral-embed. Go back to that Qdrant gotcha — matching dimensions means your existing collection still works. Mismatched ones mean starting over.And the escape hatch is already built. The env vars in my role are
TEI_BASE_URL,TEI_MODEL,TEI_DIMENSIONS. Generic OpenAI-compatible embedder knobs, named after Text Embeddings Inference for historical reasons and pointed at Mistral today. Aim them at a self-hosted TEI container, at Ollama, at anything speaking that shape, and the rest of the stack doesn’t notice.That’s what local-first buys you. Not purity. Optionality.
So, the $5 Box
My server has 60GB of RAM, which is absurd overkill and exists because it does a dozen other things. The stack itself measured 640MB with three weeks of uptime, essentially zero CPU at idle.
That fits comfortably on a small cloud VPS in the few-dollars-a-month range. Check current pricing yourself rather than trusting a number in a blog post, but the shape is: a 2 vCPU / 4GB instance from Hetzner or similar costs less per month than one seat of most AI memory SaaS products, and you get to run everything else on it too.
Your real constraint is RAM, specifically Neo4j’s JVM floor. On a 1GB instance you’d be fighting it. At 2GB you’re fine. At 4GB you’ll forget it’s running.
Why I Care
The indie web ethos is about noticing that renting your identity from a platform means the platform decides what happens to it.
We’re about to make the same mistake with agent memory, except worse, because the thing being accumulated this time is a working model of how you think and what you’re building. Every “our AI remembers you across sessions” product is a proposal that you deposit that into someone else’s database and hope the pricing page stays reasonable.
Qdrant is Apache 2.0. Neo4j Community is GPL. Docker Compose is a YAML file. Ansible is idempotent YAML. Nothing in this stack is exotic. The barrier to owning your agent memory is an afternoon and 640 megabytes.
Not everyone needs this, and I’m not going to pretend a solo dev with three side projects is being exploited by a $20 subscription. But if you’re accumulating context you’d be genuinely sad to lose, the math changes. Own the state, rent the compute, and keep the role in version control so the whole thing is reproducible on a box you haven’t bought yet.
Moving the embedder onto Cloudflare’s free tier is next on my list, what’s on yours?
I’d appreciate a follow. You can subscribe with your email below. The emails go out once a week, or you can find me on Mastodon at @[email protected].
- Installs a read-only deploy key scoped to exactly one repo, with an SSH