feat: holder sets and R < N placement (plan.md Phase 4) #19

Merged
sam merged 2 commits from feat/phase-4-holder-sets into main 2026-10-21 10:32:32 +00:00
Member

Implements Phase 4 of plan.md: holder sets and R < N.

What changed

Placement is derived, never configured. R = min(3, N); holders are the R lowest-rank live peers, so the leader (the lowest-rank live peer) is always the first holder and a deployment of three nodes or fewer keeps R = N. There is no WithReplicationFactor.

The holder set is leader-owned replicated state. The leader writes Holders{epoch, ids} on the stream's own command log (commandHolders). That epoch — never a fresh recomputation — is the set every write quorum and every promotion's read quorum are taken over, so two nodes whose live views disagree still count the same votes.

  • A created stream records epoch 1 before it accepts a write (ensureHolders).
  • A restarted node reads its epoch back from its command log before a promotion's read quorum (restoreHolders); no on-disk field was added.
  • If the record is missing, the quorum falls back to the configured set, never to the live view. A live view can only be smaller during failure detection, which is exactly the set a promotion must not read-quorum against. This fixed a real flake: a stream whose epoch was never written let a restarted, isolated node promote on a one-node quorum (TestPromotionFailsClosedWithoutAQuorum reproduced it).

Re-placement is resync, one stream at a time. A joining holder is admitted by an epoch only after its replica consumer has applied the leader's retained window (holdersCaughtUp reads the consumer watermark the replication path already maintains), so the epoch never names a node that cannot serve it and no moment has zero holders. The leader deletes a dropped holder's rep-<peerID> consumer in the same step, so the command log unpins. Server.Placements exposes the epoch, holders and in-progress set per loaded stream.

Reads on a non-holder forward. A holder answers a pull from its own copy — read availability is the holder set's, not the leader's — and a node with no copy forwards the pull, which the receiving node accepts as any holder (requireRelayTarget). This retires the shipped "record reads are never forwarded" rule: relayedStreamName now accepts a pull for any stream, and the loop guard is the receiving node's target check. Verified for a read after the leader is gone.

Quorum narrowing. replicaQuorum, the promotion read quorum, announceTerm and handoverTarget use the recorded holder set; confirmQuorum counts only a holder's ack (holderVotes), so a copy on a node the placement released cannot make an ack look durable on a majority of the set a promotion will ask.

Availability (added to README's resolved decisions)

Reads need at least one holder live; writes need at least quorum(R) holders live. N = 1 is the single-node server unchanged; N = 2 keeps durability and reads but loses writes when either node dies (quorum(2) = 2); N ≥ 3 (R = 3, quorum 2) tolerates one failure for writes and two for reads.

Tests

  • TestHoldersAreTheLowestRankedLivePeers (derivation, R = min(3, N) up to 16 members, permutation stability, leader first) and TestStreamHolderRecordProperties (rapid model of the applied epoch: monotone, no aliasing of the caller's slice).
  • TestCreationRecordsThePlacementEpoch, TestNonHolderKeepsNoCopy, TestReadOnANonHolderIsForwarded, TestReadOnANonHolderReachesALiveHolderAfterTheLeaderDies, TestHolderLossReplacesTheHolders, TestReturningHolderDisplacesTheTemporaryOne (a live node displaced by a returning one releases its copy and its consumer), TestBudgetEvictionWithNonUniformPlacement, TestTwoNodeMeshReadsFromTheSurvivorAndRefusesWrites, TestOnlyAHoldersConfirmationCounts.
  • Fuzz seeds for commandHolders (count/epoch/duplicate/bounds) and for a relayed user-stream pull; the extractor property now asserts the client and relayed paths name the same stream.
  • BenchmarkMeshForwardedRead (local vs forwarded) added to make bench-log: local 899 ns/op, forwarded 72 µs/op on mocknet, which prices the relay's dispatch, not a network.
  • TestPublishNeedsAQuorumNotJustOneReplica (four nodes, "quorum three") described R = N and is replaced by TestPublishNeedsAQuorumOfItsHolders; TestReplicaQuorumIsAMajorityOfTheConfiguredSet became ...OfTheHolderSet with the R = min(3, N) table plus the no-record fail-safe case. Both noted in plan.md's Phase 4 "As implemented".

Verification

go vet ./...                      # clean
go vet -tags rapid ./...          # clean
golangci-lint run                 # 0 issues
golangci-lint run --build-tags rapid  # 0 issues
go test -race ./...               # ok 40.8s
go test -race -tags rapid ./...   # ok 44.6s
gremlins unleash --dry-run        # 1440 runnable / 96 not covered / 93.75% (was 1369/97/93.38%)

coverage.log (88.9%, effectively unchanged) and bench.log (Phase 4 record) are updated. Phase 4 is marked shipped in plan.md, with an "As implemented" section recording the decisions the spec left open; doc.go and README gained the placement section, the availability table and the read-forwarding rule.

Implements Phase 4 of `plan.md`: holder sets and `R < N`. ## What changed **Placement is derived, never configured.** `R = min(3, N)`; holders are the `R` lowest-rank live peers, so the leader (the lowest-rank live peer) is always the first holder and a deployment of three nodes or fewer keeps `R = N`. There is no `WithReplicationFactor`. **The holder set is leader-owned replicated state.** The leader writes `Holders{epoch, ids}` on the stream's own command log (`commandHolders`). That epoch — never a fresh recomputation — is the set every write quorum and every promotion's read quorum are taken over, so two nodes whose live views disagree still count the same votes. - A created stream records epoch 1 before it accepts a write (`ensureHolders`). - A restarted node reads its epoch back from its command log before a promotion's read quorum (`restoreHolders`); no on-disk field was added. - If the record is missing, the quorum falls back to the **configured** set, never to the live view. A live view can only be smaller during failure detection, which is exactly the set a promotion must not read-quorum against. This fixed a real flake: a stream whose epoch was never written let a restarted, isolated node promote on a one-node quorum (`TestPromotionFailsClosedWithoutAQuorum` reproduced it). **Re-placement is resync, one stream at a time.** A joining holder is admitted by an epoch only after its replica consumer has applied the leader's retained window (`holdersCaughtUp` reads the consumer watermark the replication path already maintains), so the epoch never names a node that cannot serve it and no moment has zero holders. The leader deletes a dropped holder's `rep-<peerID>` consumer in the same step, so the command log unpins. `Server.Placements` exposes the epoch, holders and in-progress set per loaded stream. **Reads on a non-holder forward.** A holder answers a pull from its own copy — read availability is the holder set's, not the leader's — and a node with no copy forwards the pull, which the receiving node accepts as any holder (`requireRelayTarget`). This retires the shipped "record reads are never forwarded" rule: `relayedStreamName` now accepts a pull for any stream, and the loop guard is the receiving node's target check. Verified for a read after the leader is gone. **Quorum narrowing.** `replicaQuorum`, the promotion read quorum, `announceTerm` and `handoverTarget` use the recorded holder set; `confirmQuorum` counts only a holder's ack (`holderVotes`), so a copy on a node the placement released cannot make an ack look durable on a majority of the set a promotion will ask. ## Availability (added to README's resolved decisions) Reads need **at least one holder live**; writes need **at least `quorum(R)` holders live**. `N = 1` is the single-node server unchanged; `N = 2` keeps durability and reads but loses writes when either node dies (`quorum(2) = 2`); `N ≥ 3` (`R = 3`, quorum 2) tolerates one failure for writes and two for reads. ## Tests - `TestHoldersAreTheLowestRankedLivePeers` (derivation, `R = min(3, N)` up to 16 members, permutation stability, leader first) and `TestStreamHolderRecordProperties` (rapid model of the applied epoch: monotone, no aliasing of the caller's slice). - `TestCreationRecordsThePlacementEpoch`, `TestNonHolderKeepsNoCopy`, `TestReadOnANonHolderIsForwarded`, `TestReadOnANonHolderReachesALiveHolderAfterTheLeaderDies`, `TestHolderLossReplacesTheHolders`, `TestReturningHolderDisplacesTheTemporaryOne` (a live node displaced by a returning one releases its copy and its consumer), `TestBudgetEvictionWithNonUniformPlacement`, `TestTwoNodeMeshReadsFromTheSurvivorAndRefusesWrites`, `TestOnlyAHoldersConfirmationCounts`. - Fuzz seeds for `commandHolders` (count/epoch/duplicate/bounds) and for a relayed user-stream pull; the extractor property now asserts the client and relayed paths name the same stream. - `BenchmarkMeshForwardedRead` (local vs forwarded) added to `make bench-log`: local 899 ns/op, forwarded 72 µs/op on mocknet, which prices the relay's dispatch, not a network. - `TestPublishNeedsAQuorumNotJustOneReplica` (four nodes, "quorum three") described `R = N` and is replaced by `TestPublishNeedsAQuorumOfItsHolders`; `TestReplicaQuorumIsAMajorityOfTheConfiguredSet` became `...OfTheHolderSet` with the `R = min(3, N)` table plus the no-record fail-safe case. Both noted in `plan.md`'s Phase 4 "As implemented". ## Verification ``` go vet ./... # clean go vet -tags rapid ./... # clean golangci-lint run # 0 issues golangci-lint run --build-tags rapid # 0 issues go test -race ./... # ok 40.8s go test -race -tags rapid ./... # ok 44.6s gremlins unleash --dry-run # 1440 runnable / 96 not covered / 93.75% (was 1369/97/93.38%) ``` `coverage.log` (88.9%, effectively unchanged) and `bench.log` (Phase 4 record) are updated. Phase 4 is marked shipped in `plan.md`, with an "As implemented" section recording the decisions the spec left open; `doc.go` and README gained the placement section, the availability table and the read-forwarding rule.
feat: holder sets and R < N placement (plan.md Phase 4)
All checks were successful
code-review / Agent review (pull_request) Successful in 20s
9c0a788a72
A stream is now held by a derived placement set instead of by every mesh
node: R = min(3, N) holders, the R lowest-rank live peers, so the leader is
always the first holder and a deployment of three nodes or fewer is unchanged.

The holder set is leader-owned replicated state. The leader writes
Holders{epoch, ids} on the stream's own command log, and that epoch — never a
fresh recomputation of the live view — is the set every write quorum and every
promotion's read quorum is taken over, so two nodes whose live views disagree
still count the same votes. A created stream records epoch 1 before it accepts
a write; a restarted node reads its epoch back from the command log before a
promotion's read quorum. If the record is missing the quorum falls back to the
configured set, never to the live view, because a smaller set during failure
detection is exactly the set a promotion must not ask.

Re-placement is resync, one stream at a time. A joining holder is admitted by
an epoch only after its replica consumer has applied the leader's retained
window, so the epoch never names a node that cannot serve it and a placement
change never has a moment with no holder; the leader deletes a dropped
holder's rep-<peerID> consumer in the same step, so the command log unpins.
Server.Placements reports the epoch, holders and in-progress set per stream.

Reads: a holder answers a pull from its own copy, so read availability is the
holder set's rather than the leader's; a node with no copy forwards the pull,
and the receiving node accepts it as any holder (requireRelayTarget). This
retires "record reads are never forwarded" — relayedStreamName now accepts a
pull for any stream, and the loop guard is the receiving node's target check.

Quorum: replicaQuorum and the promotion read quorum use the recorded holder
set, and confirmQuorum counts only a holder's ack (holderVotes), so a copy on
a node the placement released cannot make an ack look durable on a majority.

Tests: holder derivation and the applied epoch's rapid model, forwarded reads
(including after the leader is gone), holder loss and the returning holder
displacing the temporary one, budget eviction with non-uniform placement, the
N = 2 read/write trade-off, the two-phase join, the placement codec's fuzz
seeds, and BenchmarkMeshForwardedRead for the forward-versus-local read cost.
TestPublishNeedsAQuorumNotJustOneReplica's four-node "quorum three" described
R = N and is replaced by the derived contract.
pi-reviewer left a comment

AI review of Phase 4 (holder sets, R < N) at head 9c0a788, run against a worktree at that commit with the container's Go 1.27.1 toolchain.

Verified in this session: go vet ./... clean, go build ./... clean, go test -race ./... ok (39.9s), golangci-lint run 0 issues, targeted placement/quorum tests ok. The diff is high quality: the wire codec is bounded before allocation (maxHolderRecord/maxPeerIDRecord, readBytes slices the input rather than allocating), holdersRecord/setHolders clone defensively, liveSet includes self so the leader is always derived into the set, the decode path rejects zero epochs/duplicate holders, and the test suite (including the rapid models and fuzz seeds) is genuinely strong.

No BLOCKING findings. Two IMPORTANT findings, both in the re-placement machinery:

  1. The promotion path records a placement epoch derived from the live view with no catch-up gate (promote.go:569 → publishHoldersLocked), while the reconcile path gates the same join on holdersCaughtUp. I reproduced the invariant violation with a throwaway mesh test (promotion over a shrunken live view named a node whose replica consumer had not applied the window).
  2. A placement plan that never converges (joining holder in the live view whose consumer never reaches target) stops advancePlacements at that stream on every reconcile pass — no expiry exists — so re-placement of every other stream this node leads stalls.

Plus two SUGGESTIONs: a doc/behavior mismatch in holdsStream (backfilling copies answer true, the comment says false) and a stale 1/2/3/5 holders row in the README's write-quorum table (R = min(3, N) caps at 3).

State is COMMENT because neither finding is a verified data-loss/race/security defect; both are availability/liveness gaps under specific conditions.

AI review of Phase 4 (holder sets, R < N) at head 9c0a788, run against a worktree at that commit with the container's Go 1.27.1 toolchain. Verified in this session: `go vet ./...` clean, `go build ./...` clean, `go test -race ./...` ok (39.9s), `golangci-lint run` 0 issues, targeted placement/quorum tests ok. The diff is high quality: the wire codec is bounded before allocation (`maxHolderRecord`/`maxPeerIDRecord`, `readBytes` slices the input rather than allocating), `holdersRecord`/`setHolders` clone defensively, `liveSet` includes self so the leader is always derived into the set, the decode path rejects zero epochs/duplicate holders, and the test suite (including the rapid models and fuzz seeds) is genuinely strong. No BLOCKING findings. Two IMPORTANT findings, both in the re-placement machinery: 1. The promotion path records a placement epoch derived from the live view with no catch-up gate (`promote.go:569` → `publishHoldersLocked`), while the reconcile path gates the same join on `holdersCaughtUp`. I reproduced the invariant violation with a throwaway mesh test (promotion over a shrunken live view named a node whose replica consumer had not applied the window). 2. A placement plan that never converges (joining holder in the live view whose consumer never reaches target) stops `advancePlacements` at that stream on every reconcile pass — no expiry exists — so re-placement of every other stream this node leads stalls. Plus two SUGGESTIONs: a doc/behavior mismatch in `holdsStream` (backfilling copies answer true, the comment says false) and a stale `1/2/3/5` holders row in the README's write-quorum table (R = min(3, N) caps at 3). State is COMMENT because neither finding is a verified data-loss/race/security defect; both are availability/liveness gaps under specific conditions.
README.md Outdated
@ -235,3 +254,3 @@
| Persistence | `WithStore(*os.Root)` makes the server durable. The stream is the unit of storage: each stream is a self-contained directory `<root>/streams/<name>/` holding a `stream.meta` (maxAge), a segmented record log under `log/` (files named by each segment's base sequence number, 20-digit zero-padded) and two fixed-stride consumer files (`consumers.meta`, `cursors.meta`). Names are 1..64 bytes of `[a-zA-Z0-9-]`, so a name is a safe path component with no encoding step. Without `WithStore` the server is purely in-memory, unchanged |
| Durable publish ack | a publish's record is fsync'd before its ack returns, so an acked record survives a crash, and fsyncs are **group-committed**: one fsync covers every record written before it started, publishers admitted while one is in flight are covered by the next one, and every writer in a batch is acked when that fsync returns. There is no interval to configure — an idle server pays exactly the one fsync per publish it always has, a busy one pays about one per batch, and there is no crash window to trade away. Appends to the segmented log are also the only writer of it, so on-disk order is sequence order |
| Write quorum | an acked publish is durable on a **quorum of the stream's holders**, `quorum(R) = R/2 + 1` with `R` the holder count (every mesh node today, `R = N`), so `1/2/3/5` holders need `1/2/2/3`. The quorum is derived from `R`, never configured: a promotion takes a read quorum of the same set, and the two majorities must intersect. The confirmation is the replica consumer's existing ack — a replica acks only after applying durably — so the leader waits on a channel the replication path already owns, and a relayed ack is counted only for the node that sent it (its consumer name is its own peer ID), so a vote cannot be cast for a node that never applied the record. The leader's own copy is the vote its append already made durable. A publish that cannot reach the quorum before `quorumTimeout` fails `ErrNoQuorum` — and so does a wait the caller cuts short, with the cancellation as its inner error — and is **not** undone: the record may be readable on the leader and may replicate later, so a retry that must not duplicate carries a `WithDeduplicationID` and reads the original sequence number back (a duplicate reply is an ack for the same sequence, so it waits on the same quorum). A node that relays a publish on a stream it does not lead **relays it on a connection of its own**: a node serves one application stream's requests one at a time, so a relayed publish waiting out its quorum on the shared per-peer connection would stall that node's own reconciliation and replication registration — the very work that produces the confirmation |
| Write quorum | an acked publish is durable on a **quorum of the stream's holders**, `quorum(R) = R/2 + 1` with `R = min(3, N)` the holder count, so `1/2/3/5` holders need `1/2/2/3`. The quorum is derived from `R`, never configured: a promotion takes a read quorum of the same set, and the two majorities must intersect. The confirmation is the replica consumer's existing ack — a replica acks only after applying durably — so the leader waits on a channel the replication path already owns, and a relayed ack is counted only for the node that sent it (its consumer name is its own peer ID), so a vote cannot be cast for a node that never applied the record. The leader's own copy is the vote its append already made durable. A publish that cannot reach the quorum before `quorumTimeout` fails `ErrNoQuorum` — and so does a wait the caller cuts short, with the cancellation as its inner error — and is **not** undone: the record may be readable on the leader and may replicate later, so a retry that must not duplicate carries a `WithDeduplicationID` and reads the original sequence number back (a duplicate reply is an ack for the same sequence, so it waits on the same quorum). A node that relays a publish on a stream it does not lead **relays it on a connection of its own**: a node serves one application stream's requests one at a time, so a relayed publish waiting out its quorum on the shared per-peer connection would stall that node's own reconciliation and replication registration — the very work that produces the confirmation |
Member

🟡 SUGGESTION — the example table "1/2/3/5 holders need 1/2/2/3" contradicts the new rule stated on the same line. With R = min(3, N) the holder count is capped at 3, so a 5-holder configuration cannot occur; the 5-column is a leftover from the R = N era. Suggest "1/2/3 holders need 1/2/2".

🟡 SUGGESTION — the example table "1/2/3/5 holders need 1/2/2/3" contradicts the new rule stated on the same line. With `R = min(3, N)` the holder count is capped at 3, so a 5-holder configuration cannot occur; the 5-column is a leftover from the R = N era. Suggest "1/2/3 holders need 1/2/2".
placement.go Outdated
@ -0,0 +56,4 @@
}
// holdsStream reports whether this node is a holder of the stream and so serves its
// record reads locally. A copy that has not finished backfilling answers false: its
Member

🟡 SUGGESTION — the doc comment contradicts the implementation. It says "A copy that has not finished backfilling answers false: its window may be short, and the read belongs on a node that has the whole one", but a joining holder (epoch 0, rule names it, copy resident/backfilling) reaches slices.Contains(holdersFor(stream, liveSet), self) → true, so reads are served locally from the short window. If that is intended (stale reads are P5-legal), fix the comment; if the comment is the contract, the implementation should answer false until the copy is caught up.

🟡 SUGGESTION — the doc comment contradicts the implementation. It says "A copy that has not finished backfilling answers false: its window may be short, and the read belongs on a node that has the whole one", but a joining holder (epoch 0, rule names it, copy resident/backfilling) reaches `slices.Contains(holdersFor(stream, liveSet), self)` → true, so reads are served locally from the short window. If that is intended (stale reads are P5-legal), fix the comment; if the comment is the contract, the implementation should answer false until the copy is caught up.
placement.go Outdated
@ -0,0 +254,4 @@
// The joining holders are still backfilling their copy: the epoch that admits them
// waits, which is what keeps a placement change from having a moment with no
// holder.
if !s.holdersCaughtUp(state.name, additions, plan.target) {
Member

🟠 IMPORTANT — a placement plan that never converges stalls re-placement of every other stream this node leads. advancePlacement returns true when holdersCaughtUp is false, and advancePlacements (placement.go:205) returns on the first true, so one in-progress plan stops the whole pass every reconcile cycle. Nothing expires a plan: it is cleared only by sameHolders (placement.go:228) or a successful publishHolders — there is no timeout, pass count, or deadline. Condition: a joining holder stays in the leader's live view (probes answered, e.g. an asymmetric partition where leader→node works but the node's replication pulls never advance, or a stalled replica loop) but its consumer never reaches target. Every stream this node leads with a dead holder then keeps its stale epoch and its write quorum stays unreachable, indefinitely. Fix: bound the plan's life — drop/re-derive the plan after N reconcile passes or a deadline (the joining node re-enters the derived set when it actually catches up), or skip past a plan older than a bound so other streams are re-placed.

🟠 IMPORTANT — a placement plan that never converges stalls re-placement of every other stream this node leads. `advancePlacement` returns true when `holdersCaughtUp` is false, and `advancePlacements` (placement.go:205) returns on the first true, so one in-progress plan stops the whole pass every reconcile cycle. Nothing expires a plan: it is cleared only by `sameHolders` (placement.go:228) or a successful `publishHolders` — there is no timeout, pass count, or deadline. Condition: a joining holder stays in the leader's live view (probes answered, e.g. an asymmetric partition where leader→node works but the node's replication pulls never advance, or a stalled replica loop) but its consumer never reaches `target`. Every stream this node leads with a dead holder then keeps its stale epoch and its write quorum stays unreachable, indefinitely. Fix: bound the plan's life — drop/re-derive the plan after N reconcile passes or a deadline (the joining node re-enters the derived set when it actually catches up), or skip past a plan older than a bound so other streams are re-placed.
promote.go Outdated
@ -569,0 +566,4 @@
// The new term owns the placement. The promotion repaired this copy from the read
// quorum and rebuilt the log, so the holders derived now are the set the next writes
// are quorumed over — and the next promotion asks.
if err := s.publishHoldersLocked(ctx, state); err != nil {
Member

🟠 IMPORTANT — the promotion path records a placement epoch without the two-phase join gate. rebuildCommandLog calls publishHoldersLocked, which writes holdersFor(stream, liveSet) — the full live-view derivation — in one step. The reconcile path gates the same join: advancePlacement (placement.go:257) waits for holdersCaughtUp before an epoch may name a new holder, and doc.go/plan.md promise "a joining holder is named by an epoch only after its replica consumer has applied the leader's retained window". Condition: a holder fails and then the leader fails before a reconcile pass re-places (leader death is itself the promotion trigger) — the promoted node's read quorum over the recorded set succeeds on the remaining live holders, and the new epoch immediately names the next-ranked live peer, which has no copy and no consumer. I reproduced it in this session with a 4-node mocknet mesh (throwaway test, removed after verification): go test -race -run TestReviewDemo failed with recorded [A B C] -> promoted epoch 2 names [B C QmcMwSaGi...] (additions [QmcMwSaGi...]) and holdersCaughtUp for the addition returned false — the epoch named a node that had not applied the window. If the remaining live recorded holders then fail before the new holder backfills, reads forward to a named holder that cannot serve (ErrForwardLoop / stale), breaking the README availability table's "reads need at least one holder live". Fix: at promotion record the epoch over the live recorded holders only (the same "pure removal lands in one step" rule advancePlacement applies to len(additions) == 0), and let the reconcile path add the new member through holdersCaughtUp; or run the additions through holdersCaughtUp before the epoch lands.

🟠 IMPORTANT — the promotion path records a placement epoch without the two-phase join gate. `rebuildCommandLog` calls `publishHoldersLocked`, which writes `holdersFor(stream, liveSet)` — the full live-view derivation — in one step. The reconcile path gates the same join: `advancePlacement` (placement.go:257) waits for `holdersCaughtUp` before an epoch may name a new holder, and doc.go/plan.md promise "a joining holder is named by an epoch only after its replica consumer has applied the leader's retained window". Condition: a holder fails and then the leader fails before a reconcile pass re-places (leader death is itself the promotion trigger) — the promoted node's read quorum over the recorded set succeeds on the remaining live holders, and the new epoch immediately names the next-ranked live peer, which has no copy and no consumer. I reproduced it in this session with a 4-node mocknet mesh (throwaway test, removed after verification): `go test -race -run TestReviewDemo` failed with `recorded [A B C] -> promoted epoch 2 names [B C QmcMwSaGi...] (additions [QmcMwSaGi...])` and `holdersCaughtUp` for the addition returned false — the epoch named a node that had not applied the window. If the remaining live recorded holders then fail before the new holder backfills, reads forward to a named holder that cannot serve (ErrForwardLoop / stale), breaking the README availability table's "reads need at least one holder live". Fix: at promotion record the epoch over the live *recorded* holders only (the same "pure removal lands in one step" rule `advancePlacement` applies to `len(additions) == 0`), and let the reconcile path add the new member through `holdersCaughtUp`; or run the additions through `holdersCaughtUp` before the epoch lands.
Review follow-up on the holder-set placement (PR #19). Two availability
findings and two documentation ones.

1. The promotion path recorded a placement epoch from the full live-view
   derivation while the reconcile path gated the same join on
   holdersCaughtUp. A promotion happens exactly when the live view has just
   shrunk, so the derived set names the next-ranked node — which has no copy
   and no consumer — and an epoch naming it sends reads to a holder that
   cannot serve them if the remaining recorded holders also fail.
   promotionHolders now records the recorded holders the live view still
   has, plus itself, and never a fresh derivation; the joiner is admitted
   later through holdersCaughtUp, so a promotion and a reconcile cannot
   disagree about what a holder is.

2. A plan that never converged stopped advancePlacements at that stream on
   every pass, indefinitely. A plan now holds the churn slot for
   placementPark (10s) and then yields, and the catch-up check runs before
   the plan's bounds so a join that completes during the park still lands on
   the next pass.

Both fixes rest on one floor: a holder set never shrinks below R =
min(3, N). A dead holder is replaced by the next-ranked live peer, never
simply dropped — dropping it would shrink the write quorum with it, and a
two-node stream whose survivor recorded itself would ack on one copy. With
fewer live nodes than the member count calls for, the stream keeps the
recorded set and writes fail on the quorum, which is what the N = 2 contract
already promised.

3. holdsStream's comment now states the actual contract: a copy still
   backfilling is a holder and serves its own window (a read above its
   applied position blocks, which is stale, not a hole); only a node the
   applied epoch does not name, or one with no copy, forwards.

4. README's write-quorum row said "1/2/3/5 holders need 1/2/2/3" under
   R = min(3, N). Five holders cannot occur; it is 1/2/3 -> 1/2/2, and the
   placement row now states the no-shrink floor.

Tests: TestPromotionDoesNotNameAnUncaughtUpHolder (kill the leader before
any reconcile pass, promote, assert the epoch excludes the joiner and that
it is admitted afterwards and holds the window) and
TestAStuckPlacementYieldsThePass (a fresh plan holds the pass and a second
plan is refused; past the deadline the pass moves on and the slot is free).
sam merged commit 2a1c05a541 into main 2026-10-21 10:32:32 +00:00
sam deleted branch feat/phase-4-holder-sets 2026-10-21 10:32:32 +00:00
Sign in to join this conversation.
No reviewers
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
norobo/webstream!19
No description provided.