CAP, Consistency and Quorums

During a network partition a distributed store must choose between consistency and availability (CAP); consistency models range from strong to eventual, and quorum reads and writes (R + W > N) give tunable guarantees.

Once data lives on several machines, they can disagree. Consistency models describe what a reader is guaranteed to see.

flowchart LR
  W["Write x=2"] --> N1["Replica 1: x=2"]
  W --> N2["Replica 2: x=2"]
  W -.->|"partition"| N3["Replica 3: x=1"]
  R["Read x"] --> N3
  N3 --> CH{"Answer stale x=1 or refuse?"}
  CH -->|"AP"| STALE["Available, maybe stale"]
  CH -->|"CP"| ERR["Consistent, may error"]
  %% step W: A write reaches the replicas it can.
  %% step N3: A network partition cuts one replica off; it still holds the old value.
  %% step R: A client reads from the cut-off replica.
  %% step CH: The system must choose: answer with possibly stale data, or refuse until it knows.
  %% step STALE: AP systems stay available and reconcile later.
  %% step ERR: CP systems keep data correct but some requests fail during the partition.

CAP in one sentence

When a Partition happens, choose Consistency or Availability. Without a partition you can have both; the real everyday trade-off is often latency vs consistency (PACELC).

Consistency models

Model	Reader sees	Example
Strong (linearizable)	The latest write, always	Bank balance, inventory decrement, etcd, Spanner
Read-your-writes	Its own writes immediately	Profile edit then reload
Monotonic reads	Never goes back in time	Feed refreshes
Eventual	Latest value eventually	Like counters, DNS, S3 replication across regions

Quorums

With N replicas, a write waits for W acknowledgements and a read asks R replicas.

R + W > N → every read overlaps the latest write → strong-ish reads. N=3, W=2, R=2 is common (Cassandra QUORUM).
W=1, R=1 → fastest, eventually consistent.
W=N → every replica must be up to write (lower availability).

In managed services

Service	Default	Option
DynamoDB	Eventually consistent reads	`--consistent-read` (strong, double cost, same region)
S3	Strong read-after-write within a region	Cross-region replication is eventual
Cassandra	Per query `CONSISTENCY ONE/QUORUM/ALL`	Tune per use case
etcd / ZooKeeper	Strong (Raft/ZAB)	Needs a majority of members up

Choosing

Money, stock and unique usernames → strong. Likes, view counts, recommendations, analytics → eventual is fine and cheaper. Design the UI for it: optimistic updates, "processing" states, idempotent retries.

Traps: assuming replicas are always in sync; using eventual reads for a "do we still have stock?" check; a 2-member etcd/consensus cluster (losing one loses the majority); multi-region writes without conflict rules.