Designing CRDT Systems: Replicas, Merges, Offline Edits, and Convergence

27 minute read

Published:

1. Why CRDTs Exist

A distributed system becomes difficult when more than one place can accept a write. If every write must go through one leader, ordering is easier. The system can say: this operation happened first, this operation happened second, and every replica should apply the same ordered log.

That leader-based model is often the right design. It is the model behind many databases, queues, and online collaborative editors. It also creates a hard limit: if a client is offline, far from the leader, or connected to a different region, it may not be able to write locally.

CRDTs solve a different problem. A Conflict-Free Replicated Data Type lets multiple replicas accept updates independently and later merge those updates without manual conflict resolution. The replicas may receive updates in different orders. Some replicas may be offline for minutes, hours, or days. If the CRDT is designed correctly, replicas that eventually receive the same updates converge to the same value.

That property is powerful, but it is not free. A CRDT moves complexity into data modeling, metadata, merge rules, deletion semantics, compaction, and product behavior. The important system-design question is not “Are CRDTs better than OT?” The better question is: “Does this product need independent writes, and can we afford the metadata and semantic complexity required to merge them?”

CRDT replicas accept local writes and converge after synchronization

What to remember: A CRDT is a data type whose replicas can be updated independently and merged deterministically. It is useful when availability, offline work, local latency, or multi-region writes matter more than strict single-leader ordering.


2. The Problem CRDTs Are Solving

Consider a shared note that currently says:

Meet at 10

Alice goes offline and changes it to:

Meet at 10 in room A

Bob is online in another region and changes it to:

Meet at 11

When Alice reconnects, the system has to combine two histories that were created without talking to each other. A last-write-wins design might discard one user’s work. A lock-based design might have prevented Alice from editing offline. A central sequencer could have ordered the writes if both users were online, but that assumption has already failed.

A CRDT design asks the application to represent edits so they can be merged. For a counter, merging can be simple. For a set, merging is harder because deletes need meaning. For a rich-text document, merging is much harder because the system must preserve order, formatting, comments, undo behavior, and user intent.

CRDTs are not magic conflict erasers. They remove one class of conflict: the system does not need a human or a single leader to decide how replicas converge at the data-structure level. The product can still have semantic conflicts. If two users edit the same paragraph in opposite ways, the CRDT can preserve both operations, but the result may still need product design around suggestions, comments, history, or review.


3. Strong Eventual Consistency

The core promise behind CRDTs is often called strong eventual consistency.

PropertyMeaning
Local updateA replica can accept a write without synchronously coordinating with every other replica.
Eventual deliveryIf replicas keep syncing, updates eventually reach the other replicas.
Deterministic mergeReceiving the same updates leads to the same state, even if delivery order differs.
ConvergenceReplicas that have seen the same updates end with equivalent values.

This is different from strong consistency. Strong consistency usually means a read observes a single, current value according to a global order or a leader’s order. CRDTs normally accept that replicas can temporarily disagree. The design goal is that disagreement is temporary and mechanically repairable.

A useful mental model is:

strong consistency:        coordinate before accepting the write
CRDT convergence:          accept locally, exchange information, merge later

That tradeoff explains where CRDTs fit. They are attractive for collaborative editors, offline-first apps, distributed databases with multi-master writes, local-first productivity tools, edge systems, shopping carts, counters, sets, presence-like structures, and replicated configuration where temporary divergence is acceptable.

They are a poor fit when every write must be globally validated before it is visible. Bank transfers, inventory with strict scarcity, uniqueness constraints, and access-control changes often need coordination somewhere in the design.


4. The Smallest CRDT Example: A Counter

A counter sounds trivial until two replicas increment it independently.

initial value = 0

replica A increments once     local value = 1
replica B increments twice    local value = 2

If the system stores only the final value, merging 1 and 2 is ambiguous. The correct merged value is 3, not 2. The CRDT counter fixes this by storing more structure than the user-visible value.

A grow-only counter keeps one count per replica:

ReplicaA componentB componentVisible value
A after local increment101
B after two local increments022
Both after merge123

The merge function takes the maximum value for each replica component:

merge({A: 1, B: 0}, {A: 0, B: 2}) = {A: 1, B: 2}
value = 1 + 2 = 3

This works because the per-replica component only grows. Replaying the same message does not double count it. Receiving messages in a different order does not change the result. Merging again does not change the result.

A grow-only counter stores one monotonic component per replica

This example reveals a general CRDT pattern: the internal representation is not always the same as the value shown to the user. The implementation stores enough metadata to make merging deterministic.


5. CRDTs Need Merge-Friendly Semantics

For a data type to behave like a CRDT, its updates and merges need properties that make ordering less important.

PropertyWhy it matters
CommutativeApplying independent updates in different orders should not change the final result.
AssociativeMerging grouped updates should not depend on grouping shape.
IdempotentReceiving the same update more than once should not duplicate its effect.
Monotonic metadataReplicas need a way to know that information has advanced.

These properties do not mean the application has no conflicts. They mean the data structure has a deterministic answer for combining concurrent information.

A shopping cart is a good example. Adding item X on one device and item Y on another device can merge naturally. Removing an item is harder. If one device adds an item while another removes it, the product must define the intended behavior. Should add win? Should remove win? Should the cart track each individual add event and remove only observed adds? Different CRDTs encode different answers.

That is why CRDT design starts with product semantics, not syntax. Before choosing a CRDT, decide what concurrent add, update, delete, move, and rename mean to users.


6. State-Based and Operation-Based CRDTs

CRDTs are commonly described in two families: state-based and operation-based. Both aim for convergence, but they synchronize different things.

ModelWhat replicas exchangeTypical merge ruleOperational requirement
State-based CRDTCurrent state or a delta of stateMerge states using a deterministic joinMessages may be duplicated or reordered.
Operation-based CRDTUpdate operationsApply operations once in a causally valid wayDelivery usually needs duplicate suppression and causal assumptions.

In a state-based CRDT, a replica can send its current CRDT state to another replica. The receiver merges that state into its own state. This is robust but can be expensive if the state is large.

In an operation-based CRDT, a replica sends operations such as “add element with ID x” or “insert character with ID c after ID p.” This can be bandwidth efficient, but the transport and storage layer must preserve enough information to avoid missing or duplicating important operations.

Many production systems mix the ideas. They exchange compact operation streams for normal sync, keep durable logs for recovery, send snapshots for fast load, and use state or delta-state transfer when a replica is far behind.

State-based CRDTs merge state while operation-based CRDTs exchange updates


7. Causality, Logical Time, and Version Vectors

CRDT systems need to reason about what a replica has already seen. Wall-clock time is not enough. Clocks can drift, messages can be delayed, and offline devices can create work long after the server’s current time has advanced.

Instead, CRDT systems commonly use logical metadata.

MetadataPurpose
Replica IDIdentifies the device, browser tab, region, or server that created an update.
Operation IDMakes each update uniquely identifiable and deduplicatable.
Lamport timestampGives a logical ordering signal without relying on wall-clock time.
Version vectorTracks the latest known counter per replica.
Causal contextDescribes which prior updates were visible when this update was created.

A version vector can answer questions like:

replica A has seen: A:7, B:3, C:0
replica B has seen: A:5, B:4, C:2

From this, the sync layer can infer that A needs B’s operations after B:3 and C’s operations after C:0. B needs A’s operations after A:5. Neither replica has a complete global picture.

Version vectors are not free. If there are millions of clients, a naive vector with one entry per client becomes too large. Production designs often scope replica IDs to active devices, compact old metadata, shard documents, garbage collect inactive actors, or use server-assigned epochs to bound the size of causal metadata.

Version vectors describe which operations each replica has observed


8. Sets: Why Deletes Are Hard

Adding to a replicated set is easy. If Alice adds A and Bob adds B, the merged set contains both values.

Deleting from a replicated set is harder because a delete needs to say what it is deleting. Suppose one replica removes X while another replica concurrently adds X. The system needs deterministic semantics.

Set designConcurrent add/remove behaviorCost
Grow-only setNo removesSimple but limited.
Two-phase setOnce removed, an item cannot be re-addedEasy to merge, awkward for users.
Last-write-wins setTimestamp decides winnerSimple, but clock behavior and lost intent are risks.
Observed-remove setRemove deletes the add events the remover has observedMore metadata, better semantics.

An observed-remove set stores unique tags for adds. Removing an item removes the tags the replica has seen. If another replica concurrently creates a new add tag, that add was not observed by the remove, so it can survive the merge.

This is a recurring CRDT idea: a delete is not just “remove value.” It often needs to reference the specific creation events or identifiers it has observed. That extra metadata is what makes concurrent behavior deterministic.


9. Sequence CRDTs for Text

Text collaboration is harder than counters and sets because order matters. Users do not only add values; they insert characters, delete ranges, move blocks, apply formatting, undo changes, and expect cursors to remain sensible.

Position-based operations are fragile in a multi-writer system:

insert "X" at index 5

Index 5 can mean different things on different replicas if those replicas have not seen the same prior edits. A sequence CRDT avoids relying only on current array indexes. It gives inserted items stable identifiers and orders those identifiers deterministically.

A sequence CRDT inserts text between stable identifiers instead of relying only on array indexes

At a simplified level, a text CRDT stores records like this:

FieldMeaning
Element IDGlobally unique ID for a character, token, or text chunk.
Parent or position IDLogical location where the element was inserted.
ValueCharacter, token, or formatting marker.
VisibilityWhether the element is visible or deleted.
Causal metadataWhat the actor had observed when creating the element.

If Alice and Bob both insert text at the same logical location, the CRDT uses a deterministic ordering rule over identifiers. Every replica that receives both inserts will render them in the same order.

This solves convergence. It does not automatically solve user intent. If two people insert words at the same sentence boundary, the merged sentence may be mechanically valid but awkward. Product design still matters. Systems often add comments, suggestions, conflict markers, history views, or paragraph-level semantics to help users understand concurrent edits.


10. A CRDT-Based Collaborative Document Architecture

A CRDT collaborative editor can accept local edits immediately, persist them locally, and synchronize them later. The server may still be important, but it does not need to be the only place where an edit can be created.

local edit first, sync second

The architecture usually separates local editing, sync transport, durable storage, merge/compaction work, and authorization.

ComponentResponsibility
Editor clientApplies local edits immediately and stores pending operations.
Local CRDT storeKeeps the document CRDT state, operation IDs, and version vector.
Sync clientExchanges missing updates with the server or peers.
Sync serviceReceives operations or deltas, deduplicates them, stores them, and forwards them.
Durable operation storeKeeps updates for replay, audit, and recovery.
Snapshot serviceCreates compact document snapshots to avoid replaying unbounded history.
Awareness serviceHandles ephemeral cursors, selections, and online status.
Authorization serviceDecides who can read, write, sync, share, or administer a document.

A CRDT document system separates local editing from synchronization and compaction

The server can still reject unauthorized writes, limit document size, enforce quotas, compact metadata, store snapshots, and provide a rendezvous point for devices. CRDT does not mean “no server.” It means the data model allows independent replicas to create mergeable updates.


11. The Write Path

When a user types in a CRDT editor, the experience is local first. The client does not wait for a server round trip before showing the edit.

  1. User types into the editor.
  2. Client converts the input into one or more CRDT operations.
  3. Client assigns stable operation IDs using its replica ID and local counter.
  4. Client applies the operations to its local CRDT state.
  5. Client stores the operations in a local pending log.
  6. Sync client sends operations to the server when connectivity is available.
  7. Server authenticates the session and validates write permission.
  8. Server deduplicates operations it has already seen.
  9. Server persists accepted operations.
  10. Server forwards missing operations to other replicas.
  11. Other replicas merge the operations into their local CRDT state.

The local-first path explains why CRDTs feel good in unreliable networks. The author sees their own edit immediately. The system can sync later.

The same path also explains the engineering cost. The client is no longer a thin view over server state. It owns local persistence, operation generation, deduplication, retry behavior, and recovery from partially synced work.


12. Reconnect and Sync

Reconnect in a CRDT system is not just “open a new WebSocket.” The transport is only the pipe. The important question is which operations each side is missing.

A reconnecting client usually sends:

FieldPurpose
Document IDIdentifies the replicated object.
User/session identityLets the server authenticate and authorize sync.
Replica IDIdentifies the local actor/device.
Version vector or sync tokenDescribes what the client has already seen.
Pending operationsSends local work that has not been acknowledged.

The server compares the client’s known version with its own durable history. Then it accepts missing client operations, sends missing server-side operations, and returns an updated sync token.

client: I have A:10, B:2 and pending A:11..A:14
server: I have A:10, B:7, C:3

server accepts A:11..A:14
server sends B:3..B:7 and C:1..C:3
both sides advance after merge

This is different from an OT design. In a central OT system, reconnect often means resuming from the last acknowledged document revision. In a CRDT system, there may be no single central revision that explains all local work. The sync protocol cares about missing operations and causal knowledge.


13. Authentication and Authorization

CRDT systems still need normal security controls. A common mistake is to assume that because data can merge, any replica can write anything. That is not a valid production design.

MomentRequired check
Document openUser can read the document.
Sync handshakeUser can participate in this document’s sync session.
Operation uploadUser can write the affected object or range.
Share changeUser can administer document permissions.
Snapshot readUser can read the snapshot and replay log.
Device revocationFuture sync from that device should be rejected or limited.

Long-lived local replicas create an important security problem. A user may create operations while they appear authorized, then upload them after their permission has changed. The product must define the rule.

PolicyBehavior
Check at upload timeReject operations uploaded after permission was removed.
Check at creation time with signed capabilityAccept operations created while a bounded write ticket was valid.
HybridAllow short offline windows, then require reauthorization.

The right answer depends on the product. A private note app may tolerate longer offline windows. A shared enterprise document may require tighter enforcement. The design should make this explicit instead of hiding it inside the sync implementation.


14. Metadata, Tombstones, and Compaction

CRDTs often store more metadata than centralized designs. That metadata is what allows deterministic merge, but it can grow over time.

MetadataWhy it existsProduction concern
Operation IDsDeduplication and replay safetyIDs must be unique and compact enough to store.
Replica IDsAttribute updates to actors/devicesInactive replicas need lifecycle management.
Version vectorsTrack what each replica has seenLarge actor sets can bloat sync metadata.
TombstonesPreserve delete knowledgeDeleted text can accumulate.
Causal contextInterpret concurrent operationsExpensive if not summarized.
SnapshotsSpeed up loadingNeed correct compaction boundaries and compatibility.

Text CRDTs are especially sensitive to tombstones. If every deleted character stays forever as hidden metadata, a heavily edited document can become much larger than the visible text.

Compaction is the process of removing or summarizing metadata without breaking future merges. It is difficult because a disconnected replica may later return with old operations. If the system has compacted away the context needed to interpret those operations, it needs a recovery path.

Practical systems usually combine several techniques:

TechniquePurpose
Snapshot at known sync frontierCreate a compact base once enough replicas have observed prior operations.
Actor retirementRemove or summarize metadata for inactive devices.
Server epochsRequire very old clients to resync from a newer snapshot.
Tombstone compactionRemove deleted elements only when no valid replica can reference them.
Chunked text representationStore runs or blocks instead of one record per character where possible.

Compaction is a principal-engineer-level design issue because it affects correctness, storage cost, migration, backwards compatibility, and disaster recovery. A CRDT prototype can ignore it. A production editor cannot.


15. Server-Side Management

Even in a CRDT system, servers do significant work.

Server responsibilityWhy it matters
AuthenticationIdentifies users, sessions, and devices.
AuthorizationPrevents unauthorized reads and writes.
Operation durabilityMakes synced edits recoverable.
DeduplicationMakes retries and reconnects safe.
FanoutDelivers new operations to connected replicas.
SnapshottingKeeps open and replay latency bounded.
CompactionControls metadata growth.
Abuse controlLimits operation size, rate, and document growth.
MigrationHandles CRDT format evolution.

The sync service should be idempotent. If a client retries the same operation after a timeout, the server should recognize the operation ID and avoid storing or broadcasting it twice.

The server should also separate durable document updates from ephemeral awareness. Cursor movement, typing indicators, and live selections can use best-effort sync with throttling. Text operations and comments need durable storage, replay, and authorization checks.


16. Multi-Region Design

CRDTs are attractive in multi-region systems because independent regions can accept writes and merge later. That does not mean every multi-region CRDT system is simple.

DesignBehaviorTradeoff
Single home regionCRDT logic exists mostly for offline clients; server writes still concentrate in one region.Simpler operations, higher latency for distant users.
Active-active regionsMultiple regions accept writes and replicate operations.Lower write latency, harder auth, quotas, ordering of side effects, and compaction.
Edge-assisted syncEdge nodes accept or buffer operations near users.Needs careful durability and replay guarantees.
Peer-to-peer syncClients exchange operations directly.Harder identity, security, moderation, and availability.

CRDTs handle the mergeable document state. They do not automatically solve global side effects. Notifications, billing events, audit logs, permission changes, search indexing, and export pipelines may still need ordered workflows or idempotent processing.

For many products, a good intermediate design is to keep CRDT document edits mergeable while using ordinary coordinated systems for non-mergeable side effects.


17. Failure Handling

A CRDT design should be explicit about what happens when parts of the system fail.

FailureExpected behavior
Client goes offlineLocal edits continue if policy allows; operations stay in local pending log.
Client reconnectsSync compares version vectors and exchanges missing operations.
Same operation is retriedOperation ID deduplication prevents duplicate effects.
Server crashes before persistingClient retries pending operation later.
Server persists but acknowledgement is lostClient retries; server deduplicates.
Replica returns after long absenceServer either syncs missing history or requires resync from a compacted snapshot.
Unauthorized client uploads operationsServer rejects operations according to authorization policy.
Compaction removed required contextClient must reload from a newer snapshot or run a migration path.

The failure model should separate user-visible guarantees from internal mechanics. “Your local edits are saved on this device” is not the same as “your edits have synced to other devices” or “your edits are durably stored on the server.” A good product makes those states visible enough that users understand risk during long offline periods.


18. Capacity Planning

CRDT capacity planning is not only request rate. The expensive parts are often metadata, replay, fanout, local storage, and compaction.

DimensionWhy it matters
Active documentsDetermines sync-service working set.
Replicas per documentDrives version-vector size and sync complexity.
Operations per secondDrives ingestion, deduplication, persistence, and fanout.
Operation sizeLarge inserts, formatting changes, and embedded object metadata can dominate bandwidth.
Replay lengthAffects document open, reconnect, and replica catch-up latency.
Tombstone volumeAffects storage, memory, and rendering performance.
Snapshot frequencyTrades write amplification for faster load and recovery.
Offline durationDetermines how much history must be retained for returning replicas.
Hot-document fanoutOne edit may need delivery to many connected clients.
Compaction throughputDetermines whether metadata growth remains bounded.

A system-design interview answer should estimate at least the active replica count, edit rate, operation size, retention window, snapshot size, and fanout cost. A production design should also track worst-case documents, not only averages. One heavily edited document with thousands of collaborators can be more dangerous than millions of quiet documents.


19. Observability

CRDT correctness issues can look like normal latency problems until users report missing edits or strange document order. The system needs metrics that distinguish sync delay, merge cost, replay pressure, and convergence failures.

AreaUseful signals
SyncSync handshake latency, pending operation count, missing operation count, resume success rate.
MergeMerge latency, operations applied per merge, duplicate operation rate, causal-gap count.
StorageOperation-log append latency, snapshot age, replay length, compaction backlog.
ClientLocal pending log size, local storage failures, render latency, offline duration.
FanoutConnected replicas per document, publish latency, dropped awareness messages.
SecurityAuthorization rejection rate, expired capability usage, revoked device upload attempts.
CorrectnessDivergence detection, checksum mismatch, invalid operation rejection, migration failure.

Some systems periodically compare document checksums between replicas that claim to have the same version frontier. A mismatch does not repair the document by itself, but it is a strong signal that the implementation has a merge, migration, or compaction bug.


20. CRDTs Versus OT

CRDTs and OT both appear in collaborative editing, but they optimize for different system shapes.

QuestionOT answerCRDT answer
Where are writes ordered?Usually by a central document sequencer.At replicas, then merged by deterministic rules.
Does offline editing fit naturally?Possible, but harder for long offline windows.Usually a core strength.
What does the server do?Orders and transforms operations.Stores, relays, authorizes, deduplicates, snapshots, and compacts.
What is the main correctness burden?Transformation rules and server ordering.Identifier design, causality, tombstones, merge semantics, and compaction.
What is easier to explain?Linear revision history.Independent replicas and eventual convergence.
What is harder operationally?Sequencer scaling and failover for hot documents.Metadata growth, old replicas, and semantic merges.

For a Google Docs-like always-online editor, OT with a central sequencer can be a practical and understandable design. For local-first editors, offline-first apps, peer collaboration, or active-active regions, CRDTs are often the better conceptual fit.

The design choice should come from requirements. Do not choose CRDTs because they sound more distributed. Choose them when independent writes and later merge are product requirements.


21. When Not to Use CRDTs

CRDTs are not the right abstraction for every distributed write problem.

NeedBetter fit
Strict uniquenessCoordinator, transaction, lease, or consensus-backed allocator.
Scarce inventoryReservation system with strong coordination.
Money movementLedger with transactional guarantees and audit controls.
Simple online collaborationCentral sequencer with OT may be easier.
Rare conflictsVersion checks and explicit conflict UI may be enough.
Small single-region productOrdinary database transactions may be simpler and safer.

The practical rule is simple: use CRDTs when the cost of coordination is higher than the cost of mergeable metadata and eventual convergence. If coordination is cheap and product semantics require one current answer, a coordinated design is usually better.


22. A Practical CRDT Design Checklist

By the end of a CRDT system design, the following boundaries should be clear.

BoundaryDesign decision
Replication scopeIs the CRDT per document, per paragraph, per object, or per field?
Replica identityWhat creates replica IDs, and how are old replicas retired?
Operation identityHow are operations deduplicated across retries and reconnects?
Merge semanticsWhat do concurrent add, delete, update, move, and format operations mean?
Sync protocolDoes sync exchange operations, state, deltas, snapshots, or a mix?
Offline policyHow long can a client write offline before reauthorization or resync?
AuthorizationAre permissions checked at operation creation, upload, or both?
CompactionWhen can tombstones and causal metadata be safely removed?
RecoveryWhat happens when a replica returns after compaction?
ObservabilityHow does the system detect divergence, causal gaps, and replay pressure?

If these questions are unanswered, the design is not finished. The CRDT data structure may converge in a demo, but the production system may still fail under old clients, revoked users, hot documents, large deletes, or long offline periods.


23. The Mental Model

A CRDT system is built around one idea: replicas do not need to ask permission from a single ordering server before every local update, but they must create updates that can be merged later.

That changes the shape of the system.

Centralized editing mindsetCRDT mindset
The server decides the next document revision.Replicas create operations with stable identities.
Clients wait for authoritative ordering.Clients apply local operations immediately.
Reconnect resumes from one revision number.Reconnect exchanges missing operations using causal metadata.
Deletes remove data from the current state.Deletes often create metadata that must be retained until safe.
Scaling focuses on sequencer ownership and fanout.Scaling also includes metadata, compaction, and old-replica recovery.

The beginner-level lesson is that CRDTs let independently edited replicas converge. The senior-level lesson is that convergence is a data-model property, not a complete product architecture. The principal-level lesson is that the hardest parts are often lifecycle problems: authorization over time, compaction without breaking old replicas, format migration, cost control, and explaining merged results to users.


24. References

  1. M. Shapiro et al., “Conflict-Free Replicated Data Types”
  2. M. Shapiro et al., “A Comprehensive Study of Convergent and Commutative Replicated Data Types”
  3. N. Preguica, J. M. Marques, M. Shapiro, and M. Letia, “A Commutative Replicated Data Type for Cooperative Editing”
  4. M. Kleppmann and A. R. Beresford, “A Conflict-Free Replicated JSON Datatype”

Leave a Comment