feat: failure detection, promotion and re-election (plan.md Phase 3) #18
No reviewers
Labels
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
norobo/webstream!18
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/phase-3-failure-detection-and-promotion"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Implements Phase 3 of
plan.md: a leader loss costs a bounded unavailability window instead of stopping the stream's writes forever (P3 in the plan's promise table).What lands
Failure detection (
promote.go). A heartbeat RPC on the control connection the mesh already holds, once per 500ms per peer, with a 750ms lease (three intervals). A broken Conn is dropped and the peer is dead at once (stream-open failure detection), and any answered request counts as liveness, so the oracle and the data path never disagree. The leader rule ranks the live view instead of the configured set.Promotion is permission to serve, not placement. A node accepts a stream's mutations only at a term it announced itself:
Server.Promotetakes a read quorum of the stream's recorded holder set (the failed leader included, never a live recomputation), and reads the term in force. A higher term whose leader is still reachable turns the attempt into a rejoin: the node adopts the term, follows that leader and refuses.Term{term, leader}).frameFetchRecords), resyncing onto a holder's base when the gap is below it.Reset{base}, oneAppendper retained record, oneConsumerUpsertper consumer. A predecessor's log cannot be replayed, and the Reset is what truncates a follower holding an uncommitted tail.A promotion that cannot reach a majority fails with
ErrReadQuorum(which matchesErrNoQuorum) and serves nothing. A write that reaches the placement rule's candidate promotes it first; a superseded node refuses withErrStaleTermand follows. A reloaded copy is not serving until it promotes.Server.TransferLeadershiphands a stream to the next live holder for planned work. The new leader announces its term once and again to a peer that comes back, and routing follows the announcement while its node is alive.Deliberate choices, documented in
plan.md("As implemented")R = N) until Phase 4'sHoldersepoch.Server.Promote; a timer-driven promotion on partition detection is not part of this landing.Verification
go vet ./...,go vet -tags rapid ./...,golangci-lint runandgolangci-lint run --build-tags rapid: clean.go test -race ./...: ok (~39s).go test -race -tags rapid ./...: ok.gremlins unleash --dry-run: 1350 runnable / 96 not covered / 93.36% (was 1171/90/92.86%). The mutants this change adds are all in the documented classes — the const arithmetic of the new intervals (7) andprobeTerm'sadoptTermfailure branch (1, a seam-needing defensive path); two previously uncovered mutants are now covered. The new comparators usebytes.Compare/cmp.CompareandsummarizeQuorumuses if/else, so they avoid the case-guard class.coverage.logandbench.logupdated.FuzzDecodeCommand,FuzzDecodeResponse, the newFuzzDecodeRecordPage.residentStreamreadrs.streamoutside the lockfinishLoadwrites it under, andpublishResidentTrimstook a nested read lock while a writer could be waiting.TestLeaderOutageServesReadsEverywherenow asserts the failover contract (reads everywhere, writes resume on three nodes and fail withErrNoQuorumon two) instead of the pre-Phase-3ErrLeaderUnavailable.Notes for review
promote.gois the new file;promotion_test.goholds the Phase 3 tests and the mocknet partition helpers (DisconnectPeers+UnlinkPeers).Termcommand gained the leader's peer ID. There is no migration path by design (no deployed store exists); a command log from an earlier revision is refused rather than misread.BenchmarkPromotionmeasures the RTO per mesh size on mocknet: ~6.1ms p50 at three nodes, ~9.1ms p50 at five, which excludes transport latency by construction.A leader loss costs a bounded unavailability window instead of stopping the stream's writes forever. Failure detection is per peer, on the control connection the mesh already holds: a heartbeat probe every 500ms with a 750ms lease (three intervals), stream-open failure detection (a broken Conn is dropped and the peer is dead until it answers again), and any answered request counted as liveness, so the oracle and the data path cannot disagree. The leader rule now ranks the live view: `hashLeader` = lowest-ranked live peer. Serving is permission, not placement. A node accepts a stream's mutations only at a term it announced itself: - `Server.Promote` takes a read quorum of the stream's recorded holder set (the failed leader included, never a live recomputation), reads the term in force, durably raises the stream's term above every term it can observe, announces itself (`Term{term, leader}`), copies the committed records its copy lacks, rebuilds the command log as `Reset{base}` + the retained window + consumer watermarks so replicas converge on what it holds, and only then serves. A promotion that cannot reach a majority fails with `ErrReadQuorum` (which matches `ErrNoQuorum`) and serves nothing. - A write that reaches the placement rule's candidate promotes it first (`ensureServing` on the mutation path); a superseded node refuses with `ErrStaleTerm` and follows the leader its read quorum named. - A reloaded copy is not serving: `served` is per process, so a restarted node announces its term again before it serves. - `Server.TransferLeadership` hands a stream to the next live holder for planned work. - The promoted leader announces its term once, and again to a peer that comes back, so a node that was partitioned across a promotion steps down. Routing follows the announcement while its node is alive, which is what makes a planned transfer and a returning old leader behave like the decisions say. Replicas get the new wire kinds (`frameHeartbeat`, `frameQuorumRead`, `frameFetchRecords`, `framePromote`, `frameAnnounceTerm`), the `Leader` field of a Term command, the `commandReset` command and the `stream.meta` term they already had; `pinFloor` keeps a rebuilt window's Reset until the window it describes holds no retained record. Tests: promotion/fencing/re-election/transfer on mocknet (with the partition helpers `UnlinkPeers`/`DisconnectPeers` the phase needs), the symmetric partition and minority cases, a term/serving rapid model, a quorum-summary model, rank stability, `FuzzDecodeRecordPage` and the extended command/response seeds, and `BenchmarkPromotion` (RTO per mesh size) in `bench.log`. Two pre-existing races the new hot path exposed are fixed: `residentStream` read `rs.stream` outside the lock `finishLoad` writes it under, and `publishResidentTrims` took a nested read lock while a writer could be waiting. BREAKING CHANGE: the on-disk command log's Term command carries the leader's peer ID, so a command log written by an earlier revision is refused rather than misread (no deployed store exists to migrate).Review: feat/phase-3-failure-detection-and-promotion
Verified in a worktree at head
fda6e172in this session:go vet ./...andgo vet -tags rapid ./...clean,gofmtclean,golangci-lint run(both tag sets) 0 issues,go test -race -count=1 ./...ok (34.7s, including all new promotion/rapid/fuzz tests), no new modules vs base (go list -m alldiff empty),govulncheck ./...finds only the pre-existing Windows-only stdlib GO-2026-6604 (os@go1.27.1, fixed in 1.27.2) — not introduced here.Overall: this is careful, well-documented work. The term/read-quorum/rebuild design holds together under the race detector, the quorum-intersection argument for at-most-one-ack is sound, the page codec is bounded (malicious counts terminate:
readRecordPagebreaks on the reader error after at most two iterations), and the test suite (mocknet partitions, synctest, rapid models, fuzz seeds) is exemplary.sync.WaitGroup.Gocompiles against the module's go 1.27.1 toolchain (confirmed viago doc sync.WaitGroup).No BLOCKING issues. Two IMPORTANT findings on the new failure-detection machinery (both are availability/placement-churn hazards, not safety: the term/quorum machinery fences every scenario they produce) and two SUGGESTIONs. Given the review policy (REQUEST_CHANGES only for verified blocking issues) this is a COMMENT; I would still address #1 and #2 before relying on this in production, since both are properties of the core liveness path this PR exists to provide.
@ -0,0 +214,4 @@}// heartbeatOnce probes every peer but this node.func (s *Server) heartbeatOnce(ctx context.Context) {🟠 IMPORTANT — the per-peer lease is violated by the serial probe loop at mesh sizes ≥ 4.
heartbeatOnceprobes every peer sequentially (for _, id := range s.mesh.set { s.probePeer(ctx, id) }, lines 217-225), each probe bounded byheartbeatInterval= 500ms (line 236), whilepeerLease= 3 × heartbeatInterval = 1.5s (line 40). A full cycle therefore takes up to (N-1) × 500ms, and the ticker drops ticks while it runs, so the cycle is also the effective per-peer probe interval. When the cycle exceeds 1.5s — any mesh with N ≥ 4 whose probe round-trips approach the 500ms deadline, or with one stalled peer delaying every later peer — a healthy peer probed early in the cycle is dropped from the live view (alive()at line 119:time.Since(seen) <= peerLease) before its next probe, every cycle. The live view oscillates,forwardTofalls back to the placement rule, and the stream's leader can be considered dead while it is answering (spurious promotion attempts, misrouted relays, ErrLeaderUnavailable to clients). Evidence (grep of promote.go):34: heartbeatInterval = 500 * time.Millisecond,40: peerLease = 3 * heartbeatInterval,236: probeCtx, cancel := context.WithTimeout(ctx, heartbeatInterval). Fix: probe peers concurrently per tick (or stagger per-peer tickers), or derive the lease from the cycle:peerLease >= N * heartbeatInterval.@ -0,0 +644,4 @@return fmt.Errorf("announcing the term on %q: %w", stream, err)}base, next := state.base(), state.nextSeq()🟡 SUGGESTION — the Reset's base and the window snapshot are separate locked reads while the age loop can run.
state.makeActive(ctx)(line 515) restarts the age trim beforerebuildCommandLog(634), andtrimExpiredadvances the base unders.lock;base, next := state.base(), state.nextSeq()(647) andwindow := state.windowRecords()(652) are separate critical sections, so a trim can land between them — the comment at 647 ('read under the stream's lock in one go') holds for the window but not for the Reset's base. The resultingReset{oldBase}+ Appends-from-newBase is only repaired downstream becauseapplyAppendresyncs on the record gap (streamInfo + resetToLocked) — correct, but it costs every replaying replica a full resync and keeps the stale base alive longer viapinFloor/windowFloor. Fix: take base+window in one exclusive read, or rebuild beforemakeActive.@ -0,0 +855,4 @@return quorumReply{}, false}mesh.drop(ctx, id, conn)🟠 IMPORTANT — a transient error on the promotion's read quorum deletes the holder from the local live view, and the deliberate retry cannot undo it.
mesh.drop(server.go:818) callsm.markDead(id)(server.go:834) whenever the failed request was on the cached control Conn.askHolderOncedrops the Conn precisely in the transient case the retry exists for ('one transient transport error must not fail a read quorum', askHolder doc) — the retry dials a fresh Conn and succeeds — but the success path returns the reply without anymarkAlive. The only sender-sidemarkAlivein the codebase isprobePeer(promote.go:266);mesh.forward's relay success also never marks alive. So the holder stays dead in this node's live view until the next heartbeat (≤500ms). In that windowforwardToskips the announced leader (s.mesh.alive(leader)false) and routes to the placement candidate — which is not serving — so the stream's requests are refused (codeFor maps ErrNotLeader to codeLeaderUnavailable), and if that candidate is this node it attempts a promotion whose rejoin check is skipped becausetop == termValue(equal term), promoting over a live announced leader. One transient error can thus migrate a stream's leadership spuriously. Evidence:grep -n 'markAlive|markDead' *.goshows sender-side markAlive only at promote.go:266; markDead at server.go:834 (inside drop) and promote.go:246/259 (probePeer); the askHolderOnce source at 829-866. Fix:markAlive(id)on the retry's success (and onmesh.forwardsuccess), or re-arm the peer when a fresh Conn round-trip succeeds.@ -0,0 +1101,4 @@break}size := len(encodeRecordFrame(rec)) + wireUint32Size🟡 SUGGESTION — each record is encoded twice on the copy path.
size := len(encodeRecordFrame(rec)) + wireUint32Sizebuilds a full encoded frame per record purely for the byte budget, andwriteRecordPagethen encodes the same record again on the wire. For a full 256-record page of ~1KB records that is ~512KB of throwaway encode/allocations per page while copying a backlog. A field-based size estimate (seq 8 + timestamp 8 + dedupID 4+len + data 4+len + frame prefixes) avoids the discarded encode.Findings resolved in
eb7e4f4All four addressed; each fix has a test that fails without it.
#1 (IMPORTANT, serial probe loop vs the lease) — fixed.
heartbeatOnceprobes every peer concurrently viasync.WaitGroup.Go, so one cycle costs oneheartbeatIntervalregardless of mesh size or one stalled peer, which is what the 3× lease is derived from.TestHeartbeatCycleCostsOneInterval(synctest) installs a handler on two peers that answers the protocol negotiation and then never replies, measures the sequential cost first (asserts it exceeds one interval, so the test cannot pass vacuously), then asserts one cycle fits inside one interval.#2 (IMPORTANT, no
markAliveon a successful request) — fixed, and it was wider than the retry. Every answered round trip now records liveness:mesh.forward,forwardOwnConn,askHolderOnce,sendTermAnnouncementandcopyFrom. A refusal counts as an answer (refusedByPeerdistinguishes a peer's own reply from a transport failure), and a refusal no longer drops the Conn it arrived on. A dial is deliberately not evidence: in this review round I first addedmarkAliveto the successful-dial paths too, andTestRepeatedPromotionsAdvanceTheTermcaught it — a stopped server's host still negotiates the protocol, so only the request proves the node serves.TestAnsweredRequestRestoresALivePeercovers three cases: a relayed refusal (resp.err() == ErrConsumerNotFound; noteConn.forwardreturns an inner refusal as a successful relay, which the test pins), an answered relay, and a loop-guard refusal (ErrForwardLoop).#3 (SUGGESTION, base and window in separate critical sections) — fixed both ways.
takeOvernow rebuilds the command log beforestate.makeActiverestarts the age loop, andwindowSnapshot()returns the Reset's base and the retained records under one hold, so no trim can land between them. The Reset and the window it describes therefore always agree, and a replaying replica no longer pays a resync for it.#4 (SUGGESTION, double encode on the copy path) — fixed.
recordPageSize/recordFrameSize(store.go, next toencodeRecordFrame) compute the wire size from the fields;serveFetchRecordsno longer builds a throwaway frame per record.TestRecordPageSizeMatchesTheFramepins the arithmetic toencodeRecordFrameacross empty, typical and 1 KiB records.Extra: the review's own test surfaced a flake in mine —
TestConcurrentPromotionOfTheCandidateIsIdempotentpromoted before a survivor held the stream, so that holder's "stream not found" counted as a zero reply to the read quorum. It now waits for both survivors to hold the record first.Verification at
eb7e4f4:go vet(+-tags rapid),golangci-lint run(+--build-tags rapid) 0 issues;go test -race ./...green 14 consecutive runs (8 plain, 6 with-coverprofile) and-tags rapidgreen; fuzz smoke runs unchanged. Coverage 89.0 → 89.1 %, gremlins 1350/96/93.36 % → 1369/97/93.38 %; the only uncovered mutants in the new code remain the const arithmetic of the three intervals and the seam-needingprobeTerm→adoptTermfailure branch (coverage.logrecords it).Comment pass (
3941dea)Done, to the standard PR #17 sets: an exported identifier keeps one short godoc line opening with its name; every other comment either summarises a constraint the code cannot show (locking/ordering, failure mode, on-disk/wire invariant) or is gone.
*_test.gonow carries no comments at all.promote.godoc commentspromotion_test.gocomments*_test.go(whole repo)//go:build rapid,//go:noinline)The removals were mostly narration that restated the following statement ("Every acked sequence is on the new leader: …", "It serves, and the survivors converge…", "Highest position first: it can fill the most", the step-by-step lists inside test bodies). What stayed, in one to three lines each, is the set of facts a reader cannot recover locally: the lease's relation to the probe cycle, the write quorum intersecting a promotion's read quorum, the recorded holder set versus the live view, why a dial is not liveness evidence but a refusal is, the Reset's place in a rebuilt log and the floor that keeps it alive, the writeLock that makes
appendCommandPosition's read-back position exact, and theframeForward-only path that makes an internal name unreachable to a client.doc.gowent 148 → 141 lines, keeping the failover architecture it is the only home for.No code changed:
go/scannertoken streams (comments excluded) of all 14 changed files against the parent commit are identical.gofmt -lclean,go vet(±rapid) clean,golangci-lint(±--build-tags rapid) 0 issues,go test -race ./...and-race -tags rapid ./...green.