feat: holder sets and R < N placement (plan.md Phase 4) #19
No reviewers
Labels
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
norobo/webstream!19
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/phase-4-holder-sets"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Implements Phase 4 of
plan.md: holder sets andR < N.What changed
Placement is derived, never configured.
R = min(3, N); holders are theRlowest-rank live peers, so the leader (the lowest-rank live peer) is always the first holder and a deployment of three nodes or fewer keepsR = N. There is noWithReplicationFactor.The holder set is leader-owned replicated state. The leader writes
Holders{epoch, ids}on the stream's own command log (commandHolders). That epoch — never a fresh recomputation — is the set every write quorum and every promotion's read quorum are taken over, so two nodes whose live views disagree still count the same votes.ensureHolders).restoreHolders); no on-disk field was added.TestPromotionFailsClosedWithoutAQuorumreproduced it).Re-placement is resync, one stream at a time. A joining holder is admitted by an epoch only after its replica consumer has applied the leader's retained window (
holdersCaughtUpreads the consumer watermark the replication path already maintains), so the epoch never names a node that cannot serve it and no moment has zero holders. The leader deletes a dropped holder'srep-<peerID>consumer in the same step, so the command log unpins.Server.Placementsexposes the epoch, holders and in-progress set per loaded stream.Reads on a non-holder forward. A holder answers a pull from its own copy — read availability is the holder set's, not the leader's — and a node with no copy forwards the pull, which the receiving node accepts as any holder (
requireRelayTarget). This retires the shipped "record reads are never forwarded" rule:relayedStreamNamenow accepts a pull for any stream, and the loop guard is the receiving node's target check. Verified for a read after the leader is gone.Quorum narrowing.
replicaQuorum, the promotion read quorum,announceTermandhandoverTargetuse the recorded holder set;confirmQuorumcounts only a holder's ack (holderVotes), so a copy on a node the placement released cannot make an ack look durable on a majority of the set a promotion will ask.Availability (added to README's resolved decisions)
Reads need at least one holder live; writes need at least
quorum(R)holders live.N = 1is the single-node server unchanged;N = 2keeps durability and reads but loses writes when either node dies (quorum(2) = 2);N ≥ 3(R = 3, quorum 2) tolerates one failure for writes and two for reads.Tests
TestHoldersAreTheLowestRankedLivePeers(derivation,R = min(3, N)up to 16 members, permutation stability, leader first) andTestStreamHolderRecordProperties(rapid model of the applied epoch: monotone, no aliasing of the caller's slice).TestCreationRecordsThePlacementEpoch,TestNonHolderKeepsNoCopy,TestReadOnANonHolderIsForwarded,TestReadOnANonHolderReachesALiveHolderAfterTheLeaderDies,TestHolderLossReplacesTheHolders,TestReturningHolderDisplacesTheTemporaryOne(a live node displaced by a returning one releases its copy and its consumer),TestBudgetEvictionWithNonUniformPlacement,TestTwoNodeMeshReadsFromTheSurvivorAndRefusesWrites,TestOnlyAHoldersConfirmationCounts.commandHolders(count/epoch/duplicate/bounds) and for a relayed user-stream pull; the extractor property now asserts the client and relayed paths name the same stream.BenchmarkMeshForwardedRead(local vs forwarded) added tomake bench-log: local 899 ns/op, forwarded 72 µs/op on mocknet, which prices the relay's dispatch, not a network.TestPublishNeedsAQuorumNotJustOneReplica(four nodes, "quorum three") describedR = Nand is replaced byTestPublishNeedsAQuorumOfItsHolders;TestReplicaQuorumIsAMajorityOfTheConfiguredSetbecame...OfTheHolderSetwith theR = min(3, N)table plus the no-record fail-safe case. Both noted inplan.md's Phase 4 "As implemented".Verification
coverage.log(88.9%, effectively unchanged) andbench.log(Phase 4 record) are updated. Phase 4 is marked shipped inplan.md, with an "As implemented" section recording the decisions the spec left open;doc.goand README gained the placement section, the availability table and the read-forwarding rule.A stream is now held by a derived placement set instead of by every mesh node: R = min(3, N) holders, the R lowest-rank live peers, so the leader is always the first holder and a deployment of three nodes or fewer is unchanged. The holder set is leader-owned replicated state. The leader writes Holders{epoch, ids} on the stream's own command log, and that epoch — never a fresh recomputation of the live view — is the set every write quorum and every promotion's read quorum is taken over, so two nodes whose live views disagree still count the same votes. A created stream records epoch 1 before it accepts a write; a restarted node reads its epoch back from the command log before a promotion's read quorum. If the record is missing the quorum falls back to the configured set, never to the live view, because a smaller set during failure detection is exactly the set a promotion must not ask. Re-placement is resync, one stream at a time. A joining holder is admitted by an epoch only after its replica consumer has applied the leader's retained window, so the epoch never names a node that cannot serve it and a placement change never has a moment with no holder; the leader deletes a dropped holder's rep-<peerID> consumer in the same step, so the command log unpins. Server.Placements reports the epoch, holders and in-progress set per stream. Reads: a holder answers a pull from its own copy, so read availability is the holder set's rather than the leader's; a node with no copy forwards the pull, and the receiving node accepts it as any holder (requireRelayTarget). This retires "record reads are never forwarded" — relayedStreamName now accepts a pull for any stream, and the loop guard is the receiving node's target check. Quorum: replicaQuorum and the promotion read quorum use the recorded holder set, and confirmQuorum counts only a holder's ack (holderVotes), so a copy on a node the placement released cannot make an ack look durable on a majority. Tests: holder derivation and the applied epoch's rapid model, forwarded reads (including after the leader is gone), holder loss and the returning holder displacing the temporary one, budget eviction with non-uniform placement, the N = 2 read/write trade-off, the two-phase join, the placement codec's fuzz seeds, and BenchmarkMeshForwardedRead for the forward-versus-local read cost. TestPublishNeedsAQuorumNotJustOneReplica's four-node "quorum three" described R = N and is replaced by the derived contract.AI review of Phase 4 (holder sets, R < N) at head
9c0a788, run against a worktree at that commit with the container's Go 1.27.1 toolchain.Verified in this session:
go vet ./...clean,go build ./...clean,go test -race ./...ok (39.9s),golangci-lint run0 issues, targeted placement/quorum tests ok. The diff is high quality: the wire codec is bounded before allocation (maxHolderRecord/maxPeerIDRecord,readBytesslices the input rather than allocating),holdersRecord/setHoldersclone defensively,liveSetincludes self so the leader is always derived into the set, the decode path rejects zero epochs/duplicate holders, and the test suite (including the rapid models and fuzz seeds) is genuinely strong.No BLOCKING findings. Two IMPORTANT findings, both in the re-placement machinery:
promote.go:569→publishHoldersLocked), while the reconcile path gates the same join onholdersCaughtUp. I reproduced the invariant violation with a throwaway mesh test (promotion over a shrunken live view named a node whose replica consumer had not applied the window).advancePlacementsat that stream on every reconcile pass — no expiry exists — so re-placement of every other stream this node leads stalls.Plus two SUGGESTIONs: a doc/behavior mismatch in
holdsStream(backfilling copies answer true, the comment says false) and a stale1/2/3/5holders row in the README's write-quorum table (R = min(3, N) caps at 3).State is COMMENT because neither finding is a verified data-loss/race/security defect; both are availability/liveness gaps under specific conditions.
@ -235,3 +254,3 @@| Persistence | `WithStore(*os.Root)` makes the server durable. The stream is the unit of storage: each stream is a self-contained directory `<root>/streams/<name>/` holding a `stream.meta` (maxAge), a segmented record log under `log/` (files named by each segment's base sequence number, 20-digit zero-padded) and two fixed-stride consumer files (`consumers.meta`, `cursors.meta`). Names are 1..64 bytes of `[a-zA-Z0-9-]`, so a name is a safe path component with no encoding step. Without `WithStore` the server is purely in-memory, unchanged || Durable publish ack | a publish's record is fsync'd before its ack returns, so an acked record survives a crash, and fsyncs are **group-committed**: one fsync covers every record written before it started, publishers admitted while one is in flight are covered by the next one, and every writer in a batch is acked when that fsync returns. There is no interval to configure — an idle server pays exactly the one fsync per publish it always has, a busy one pays about one per batch, and there is no crash window to trade away. Appends to the segmented log are also the only writer of it, so on-disk order is sequence order || Write quorum | an acked publish is durable on a **quorum of the stream's holders**, `quorum(R) = R/2 + 1` with `R` the holder count (every mesh node today, `R = N`), so `1/2/3/5` holders need `1/2/2/3`. The quorum is derived from `R`, never configured: a promotion takes a read quorum of the same set, and the two majorities must intersect. The confirmation is the replica consumer's existing ack — a replica acks only after applying durably — so the leader waits on a channel the replication path already owns, and a relayed ack is counted only for the node that sent it (its consumer name is its own peer ID), so a vote cannot be cast for a node that never applied the record. The leader's own copy is the vote its append already made durable. A publish that cannot reach the quorum before `quorumTimeout` fails `ErrNoQuorum` — and so does a wait the caller cuts short, with the cancellation as its inner error — and is **not** undone: the record may be readable on the leader and may replicate later, so a retry that must not duplicate carries a `WithDeduplicationID` and reads the original sequence number back (a duplicate reply is an ack for the same sequence, so it waits on the same quorum). A node that relays a publish on a stream it does not lead **relays it on a connection of its own**: a node serves one application stream's requests one at a time, so a relayed publish waiting out its quorum on the shared per-peer connection would stall that node's own reconciliation and replication registration — the very work that produces the confirmation || Write quorum | an acked publish is durable on a **quorum of the stream's holders**, `quorum(R) = R/2 + 1` with `R = min(3, N)` the holder count, so `1/2/3/5` holders need `1/2/2/3`. The quorum is derived from `R`, never configured: a promotion takes a read quorum of the same set, and the two majorities must intersect. The confirmation is the replica consumer's existing ack — a replica acks only after applying durably — so the leader waits on a channel the replication path already owns, and a relayed ack is counted only for the node that sent it (its consumer name is its own peer ID), so a vote cannot be cast for a node that never applied the record. The leader's own copy is the vote its append already made durable. A publish that cannot reach the quorum before `quorumTimeout` fails `ErrNoQuorum` — and so does a wait the caller cuts short, with the cancellation as its inner error — and is **not** undone: the record may be readable on the leader and may replicate later, so a retry that must not duplicate carries a `WithDeduplicationID` and reads the original sequence number back (a duplicate reply is an ack for the same sequence, so it waits on the same quorum). A node that relays a publish on a stream it does not lead **relays it on a connection of its own**: a node serves one application stream's requests one at a time, so a relayed publish waiting out its quorum on the shared per-peer connection would stall that node's own reconciliation and replication registration — the very work that produces the confirmation |🟡 SUGGESTION — the example table "1/2/3/5 holders need 1/2/2/3" contradicts the new rule stated on the same line. With
R = min(3, N)the holder count is capped at 3, so a 5-holder configuration cannot occur; the 5-column is a leftover from the R = N era. Suggest "1/2/3 holders need 1/2/2".@ -0,0 +56,4 @@}// holdsStream reports whether this node is a holder of the stream and so serves its// record reads locally. A copy that has not finished backfilling answers false: its🟡 SUGGESTION — the doc comment contradicts the implementation. It says "A copy that has not finished backfilling answers false: its window may be short, and the read belongs on a node that has the whole one", but a joining holder (epoch 0, rule names it, copy resident/backfilling) reaches
slices.Contains(holdersFor(stream, liveSet), self)→ true, so reads are served locally from the short window. If that is intended (stale reads are P5-legal), fix the comment; if the comment is the contract, the implementation should answer false until the copy is caught up.@ -0,0 +254,4 @@// The joining holders are still backfilling their copy: the epoch that admits them// waits, which is what keeps a placement change from having a moment with no// holder.if !s.holdersCaughtUp(state.name, additions, plan.target) {🟠 IMPORTANT — a placement plan that never converges stalls re-placement of every other stream this node leads.
advancePlacementreturns true whenholdersCaughtUpis false, andadvancePlacements(placement.go:205) returns on the first true, so one in-progress plan stops the whole pass every reconcile cycle. Nothing expires a plan: it is cleared only bysameHolders(placement.go:228) or a successfulpublishHolders— there is no timeout, pass count, or deadline. Condition: a joining holder stays in the leader's live view (probes answered, e.g. an asymmetric partition where leader→node works but the node's replication pulls never advance, or a stalled replica loop) but its consumer never reachestarget. Every stream this node leads with a dead holder then keeps its stale epoch and its write quorum stays unreachable, indefinitely. Fix: bound the plan's life — drop/re-derive the plan after N reconcile passes or a deadline (the joining node re-enters the derived set when it actually catches up), or skip past a plan older than a bound so other streams are re-placed.@ -569,0 +566,4 @@// The new term owns the placement. The promotion repaired this copy from the read// quorum and rebuilt the log, so the holders derived now are the set the next writes// are quorumed over — and the next promotion asks.if err := s.publishHoldersLocked(ctx, state); err != nil {🟠 IMPORTANT — the promotion path records a placement epoch without the two-phase join gate.
rebuildCommandLogcallspublishHoldersLocked, which writesholdersFor(stream, liveSet)— the full live-view derivation — in one step. The reconcile path gates the same join:advancePlacement(placement.go:257) waits forholdersCaughtUpbefore an epoch may name a new holder, and doc.go/plan.md promise "a joining holder is named by an epoch only after its replica consumer has applied the leader's retained window". Condition: a holder fails and then the leader fails before a reconcile pass re-places (leader death is itself the promotion trigger) — the promoted node's read quorum over the recorded set succeeds on the remaining live holders, and the new epoch immediately names the next-ranked live peer, which has no copy and no consumer. I reproduced it in this session with a 4-node mocknet mesh (throwaway test, removed after verification):go test -race -run TestReviewDemofailed withrecorded [A B C] -> promoted epoch 2 names [B C QmcMwSaGi...] (additions [QmcMwSaGi...])andholdersCaughtUpfor the addition returned false — the epoch named a node that had not applied the window. If the remaining live recorded holders then fail before the new holder backfills, reads forward to a named holder that cannot serve (ErrForwardLoop / stale), breaking the README availability table's "reads need at least one holder live". Fix: at promotion record the epoch over the live recorded holders only (the same "pure removal lands in one step" ruleadvancePlacementapplies tolen(additions) == 0), and let the reconcile path add the new member throughholdersCaughtUp; or run the additions throughholdersCaughtUpbefore the epoch lands.