feat: failure detection, promotion and re-election (plan.md Phase 3) #18

Merged
sam merged 3 commits from feat/phase-3-failure-detection-and-promotion into main 2026-10-20 15:32:24 +00:00
Member

Implements Phase 3 of plan.md: a leader loss costs a bounded unavailability window instead of stopping the stream's writes forever (P3 in the plan's promise table).

What lands

Failure detection (promote.go). A heartbeat RPC on the control connection the mesh already holds, once per 500ms per peer, with a 750ms lease (three intervals). A broken Conn is dropped and the peer is dead at once (stream-open failure detection), and any answered request counts as liveness, so the oracle and the data path never disagree. The leader rule ranks the live view instead of the configured set.

Promotion is permission to serve, not placement. A node accepts a stream's mutations only at a term it announced itself:

  1. Server.Promote takes a read quorum of the stream's recorded holder set (the failed leader included, never a live recomputation), and reads the term in force. A higher term whose leader is still reachable turns the attempt into a rejoin: the node adopts the term, follows that leader and refuses.
  2. It durably raises the term above every term it observed and announces itself (Term{term, leader}).
  3. It replays its own log and copies the records the quorum's applied position has and it lacks (frameFetchRecords), resyncing onto a holder's base when the gap is below it.
  4. It rebuilds the command log: Reset{base}, one Append per retained record, one ConsumerUpsert per consumer. A predecessor's log cannot be replayed, and the Reset is what truncates a follower holding an uncommitted tail.
  5. Only then does it serve.

A promotion that cannot reach a majority fails with ErrReadQuorum (which matches ErrNoQuorum) and serves nothing. A write that reaches the placement rule's candidate promotes it first; a superseded node refuses with ErrStaleTerm and follows. A reloaded copy is not serving until it promotes. Server.TransferLeadership hands a stream to the next live holder for planned work. The new leader announces its term once and again to a peer that comes back, and routing follows the announcement while its node is alive.

Deliberate choices, documented in plan.md ("As implemented")

  • The term announcement names the leader, and routing follows it. Two things the phase owes need it: a transfer to a node the rule would not pick, and an old leader that must rejoin as a follower rather than take the stream back on every restart.
  • The read quorum comes before the term bump: nothing is written unless a majority answers, and the read is what tells the promoter which term it has to beat.
  • The recorded holder set is the configured node set (R = N) until Phase 4's Holders epoch.
  • Promotion is demand-driven (a mutation reaching the candidate) plus Server.Promote; a timer-driven promotion on partition detection is not part of this landing.

Verification

  • go vet ./..., go vet -tags rapid ./..., golangci-lint run and golangci-lint run --build-tags rapid: clean.
  • go test -race ./...: ok (~39s). go test -race -tags rapid ./...: ok.
  • gremlins unleash --dry-run: 1350 runnable / 96 not covered / 93.36% (was 1171/90/92.86%). The mutants this change adds are all in the documented classes — the const arithmetic of the new intervals (7) and probeTerm's adoptTerm failure branch (1, a seam-needing defensive path); two previously uncovered mutants are now covered. The new comparators use bytes.Compare/cmp.Compare and summarizeQuorum uses if/else, so they avoid the case-guard class.
  • Coverage 89.0% (was 88.6%); coverage.log and bench.log updated.
  • Fuzz targets smoke-run: FuzzDecodeCommand, FuzzDecodeResponse, the new FuzzDecodeRecordPage.
  • Two pre-existing races the new hot path exposed are fixed: residentStream read rs.stream outside the lock finishLoad writes it under, and publishResidentTrims took a nested read lock while a writer could be waiting.

TestLeaderOutageServesReadsEverywhere now asserts the failover contract (reads everywhere, writes resume on three nodes and fail with ErrNoQuorum on two) instead of the pre-Phase-3 ErrLeaderUnavailable.

Notes for review

  • promote.go is the new file; promotion_test.go holds the Phase 3 tests and the mocknet partition helpers (DisconnectPeers + UnlinkPeers).
  • The command log's Term command gained the leader's peer ID. There is no migration path by design (no deployed store exists); a command log from an earlier revision is refused rather than misread.
  • BenchmarkPromotion measures the RTO per mesh size on mocknet: ~6.1ms p50 at three nodes, ~9.1ms p50 at five, which excludes transport latency by construction.
Implements Phase 3 of `plan.md`: a leader loss costs a bounded unavailability window instead of stopping the stream's writes forever (P3 in the plan's promise table). ## What lands **Failure detection** (`promote.go`). A heartbeat RPC on the control connection the mesh already holds, once per 500ms per peer, with a 750ms lease (three intervals). A broken Conn is dropped and the peer is dead at once (stream-open failure detection), and any answered request counts as liveness, so the oracle and the data path never disagree. The leader rule ranks the live view instead of the configured set. **Promotion is permission to serve, not placement.** A node accepts a stream's mutations only at a term it announced itself: 1. `Server.Promote` takes a read quorum of the stream's *recorded* holder set (the failed leader included, never a live recomputation), and reads the term in force. A higher term whose leader is still reachable turns the attempt into a rejoin: the node adopts the term, follows that leader and refuses. 2. It durably raises the term above every term it observed and announces itself (`Term{term, leader}`). 3. It replays its own log and copies the records the quorum's applied position has and it lacks (`frameFetchRecords`), resyncing onto a holder's base when the gap is below it. 4. It rebuilds the command log: `Reset{base}`, one `Append` per retained record, one `ConsumerUpsert` per consumer. A predecessor's log cannot be replayed, and the Reset is what truncates a follower holding an uncommitted tail. 5. Only then does it serve. A promotion that cannot reach a majority fails with `ErrReadQuorum` (which matches `ErrNoQuorum`) and serves nothing. A write that reaches the placement rule's candidate promotes it first; a superseded node refuses with `ErrStaleTerm` and follows. A reloaded copy is not serving until it promotes. `Server.TransferLeadership` hands a stream to the next live holder for planned work. The new leader announces its term once and again to a peer that comes back, and routing follows the announcement while its node is alive. ## Deliberate choices, documented in `plan.md` ("As implemented") - The term announcement names the leader, and routing follows it. Two things the phase owes need it: a transfer to a node the rule would not pick, and an old leader that must rejoin as a follower rather than take the stream back on every restart. - The read quorum comes *before* the term bump: nothing is written unless a majority answers, and the read is what tells the promoter which term it has to beat. - The recorded holder set is the configured node set (`R = N`) until Phase 4's `Holders` epoch. - Promotion is demand-driven (a mutation reaching the candidate) plus `Server.Promote`; a timer-driven promotion on partition detection is not part of this landing. ## Verification - `go vet ./...`, `go vet -tags rapid ./...`, `golangci-lint run` and `golangci-lint run --build-tags rapid`: clean. - `go test -race ./...`: ok (~39s). `go test -race -tags rapid ./...`: ok. - `gremlins unleash --dry-run`: 1350 runnable / 96 not covered / 93.36% (was 1171/90/92.86%). The mutants this change adds are all in the documented classes — the const arithmetic of the new intervals (7) and `probeTerm`'s `adoptTerm` failure branch (1, a seam-needing defensive path); two previously uncovered mutants are now covered. The new comparators use `bytes.Compare`/`cmp.Compare` and `summarizeQuorum` uses if/else, so they avoid the case-guard class. - Coverage 89.0% (was 88.6%); `coverage.log` and `bench.log` updated. - Fuzz targets smoke-run: `FuzzDecodeCommand`, `FuzzDecodeResponse`, the new `FuzzDecodeRecordPage`. - Two pre-existing races the new hot path exposed are fixed: `residentStream` read `rs.stream` outside the lock `finishLoad` writes it under, and `publishResidentTrims` took a nested read lock while a writer could be waiting. `TestLeaderOutageServesReadsEverywhere` now asserts the failover contract (reads everywhere, writes resume on three nodes and fail with `ErrNoQuorum` on two) instead of the pre-Phase-3 `ErrLeaderUnavailable`. ## Notes for review - `promote.go` is the new file; `promotion_test.go` holds the Phase 3 tests and the mocknet partition helpers (`DisconnectPeers` + `UnlinkPeers`). - The command log's `Term` command gained the leader's peer ID. There is no migration path by design (no deployed store exists); a command log from an earlier revision is refused rather than misread. - `BenchmarkPromotion` measures the RTO per mesh size on mocknet: ~6.1ms p50 at three nodes, ~9.1ms p50 at five, which excludes transport latency by construction.
feat: failure detection, promotion and re-election (plan.md Phase 3)
All checks were successful
code-review / Agent review (pull_request) Successful in 8s
fda6e1722e
A leader loss costs a bounded unavailability window instead of stopping the
stream's writes forever.

Failure detection is per peer, on the control connection the mesh already
holds: a heartbeat probe every 500ms with a 750ms lease (three intervals),
stream-open failure detection (a broken Conn is dropped and the peer is dead
until it answers again), and any answered request counted as liveness, so the
oracle and the data path cannot disagree. The leader rule now ranks the live
view: `hashLeader` = lowest-ranked live peer.

Serving is permission, not placement. A node accepts a stream's mutations only
at a term it announced itself:

- `Server.Promote` takes a read quorum of the stream's recorded holder set (the
  failed leader included, never a live recomputation), reads the term in force,
  durably raises the stream's term above every term it can observe, announces
  itself (`Term{term, leader}`), copies the committed records its copy lacks,
  rebuilds the command log as `Reset{base}` + the retained window + consumer
  watermarks so replicas converge on what it holds, and only then serves. A
  promotion that cannot reach a majority fails with `ErrReadQuorum` (which
  matches `ErrNoQuorum`) and serves nothing.
- A write that reaches the placement rule's candidate promotes it first
  (`ensureServing` on the mutation path); a superseded node refuses with
  `ErrStaleTerm` and follows the leader its read quorum named.
- A reloaded copy is not serving: `served` is per process, so a restarted node
  announces its term again before it serves.
- `Server.TransferLeadership` hands a stream to the next live holder for
  planned work.
- The promoted leader announces its term once, and again to a peer that comes
  back, so a node that was partitioned across a promotion steps down. Routing
  follows the announcement while its node is alive, which is what makes a
  planned transfer and a returning old leader behave like the decisions say.

Replicas get the new wire kinds (`frameHeartbeat`, `frameQuorumRead`,
`frameFetchRecords`, `framePromote`, `frameAnnounceTerm`), the `Leader` field of
a Term command, the `commandReset` command and the `stream.meta` term they
already had; `pinFloor` keeps a rebuilt window's Reset until the window it
describes holds no retained record.

Tests: promotion/fencing/re-election/transfer on mocknet (with the partition
helpers `UnlinkPeers`/`DisconnectPeers` the phase needs), the symmetric
partition and minority cases, a term/serving rapid model, a quorum-summary
model, rank stability, `FuzzDecodeRecordPage` and the extended command/response
seeds, and `BenchmarkPromotion` (RTO per mesh size) in `bench.log`. Two
pre-existing races the new hot path exposed are fixed: `residentStream` read
`rs.stream` outside the lock `finishLoad` writes it under, and
`publishResidentTrims` took a nested read lock while a writer could be waiting.

BREAKING CHANGE: the on-disk command log's Term command carries the leader's
peer ID, so a command log written by an earlier revision is refused rather
than misread (no deployed store exists to migrate).
pi-reviewer left a comment

Review: feat/phase-3-failure-detection-and-promotion

Verified in a worktree at head fda6e172 in this session: go vet ./... and go vet -tags rapid ./... clean, gofmt clean, golangci-lint run (both tag sets) 0 issues, go test -race -count=1 ./... ok (34.7s, including all new promotion/rapid/fuzz tests), no new modules vs base (go list -m all diff empty), govulncheck ./... finds only the pre-existing Windows-only stdlib GO-2026-6604 (os@go1.27.1, fixed in 1.27.2) — not introduced here.

Overall: this is careful, well-documented work. The term/read-quorum/rebuild design holds together under the race detector, the quorum-intersection argument for at-most-one-ack is sound, the page codec is bounded (malicious counts terminate: readRecordPage breaks on the reader error after at most two iterations), and the test suite (mocknet partitions, synctest, rapid models, fuzz seeds) is exemplary. sync.WaitGroup.Go compiles against the module's go 1.27.1 toolchain (confirmed via go doc sync.WaitGroup).

No BLOCKING issues. Two IMPORTANT findings on the new failure-detection machinery (both are availability/placement-churn hazards, not safety: the term/quorum machinery fences every scenario they produce) and two SUGGESTIONs. Given the review policy (REQUEST_CHANGES only for verified blocking issues) this is a COMMENT; I would still address #1 and #2 before relying on this in production, since both are properties of the core liveness path this PR exists to provide.

## Review: feat/phase-3-failure-detection-and-promotion Verified in a worktree at head fda6e172 in this session: `go vet ./...` and `go vet -tags rapid ./...` clean, `gofmt` clean, `golangci-lint run` (both tag sets) 0 issues, `go test -race -count=1 ./...` ok (34.7s, including all new promotion/rapid/fuzz tests), no new modules vs base (`go list -m all` diff empty), `govulncheck ./...` finds only the pre-existing Windows-only stdlib GO-2026-6604 (os@go1.27.1, fixed in 1.27.2) — not introduced here. Overall: this is careful, well-documented work. The term/read-quorum/rebuild design holds together under the race detector, the quorum-intersection argument for at-most-one-ack is sound, the page codec is bounded (malicious counts terminate: `readRecordPage` breaks on the reader error after at most two iterations), and the test suite (mocknet partitions, synctest, rapid models, fuzz seeds) is exemplary. `sync.WaitGroup.Go` compiles against the module's go 1.27.1 toolchain (confirmed via `go doc sync.WaitGroup`). No BLOCKING issues. Two IMPORTANT findings on the new failure-detection machinery (both are availability/placement-churn hazards, not safety: the term/quorum machinery fences every scenario they produce) and two SUGGESTIONs. Given the review policy (REQUEST_CHANGES only for verified blocking issues) this is a COMMENT; I would still address #1 and #2 before relying on this in production, since both are properties of the core liveness path this PR exists to provide.
@ -0,0 +214,4 @@
}
// heartbeatOnce probes every peer but this node.
func (s *Server) heartbeatOnce(ctx context.Context) {
Member

🟠 IMPORTANT — the per-peer lease is violated by the serial probe loop at mesh sizes ≥ 4. heartbeatOnce probes every peer sequentially (for _, id := range s.mesh.set { s.probePeer(ctx, id) }, lines 217-225), each probe bounded by heartbeatInterval = 500ms (line 236), while peerLease = 3 × heartbeatInterval = 1.5s (line 40). A full cycle therefore takes up to (N-1) × 500ms, and the ticker drops ticks while it runs, so the cycle is also the effective per-peer probe interval. When the cycle exceeds 1.5s — any mesh with N ≥ 4 whose probe round-trips approach the 500ms deadline, or with one stalled peer delaying every later peer — a healthy peer probed early in the cycle is dropped from the live view (alive() at line 119: time.Since(seen) <= peerLease) before its next probe, every cycle. The live view oscillates, forwardTo falls back to the placement rule, and the stream's leader can be considered dead while it is answering (spurious promotion attempts, misrouted relays, ErrLeaderUnavailable to clients). Evidence (grep of promote.go): 34: heartbeatInterval = 500 * time.Millisecond, 40: peerLease = 3 * heartbeatInterval, 236: probeCtx, cancel := context.WithTimeout(ctx, heartbeatInterval). Fix: probe peers concurrently per tick (or stagger per-peer tickers), or derive the lease from the cycle: peerLease >= N * heartbeatInterval.

🟠 IMPORTANT — the per-peer lease is violated by the serial probe loop at mesh sizes ≥ 4. `heartbeatOnce` probes every peer sequentially (`for _, id := range s.mesh.set { s.probePeer(ctx, id) }`, lines 217-225), each probe bounded by `heartbeatInterval` = 500ms (line 236), while `peerLease` = 3 × heartbeatInterval = 1.5s (line 40). A full cycle therefore takes up to (N-1) × 500ms, and the ticker drops ticks while it runs, so the cycle is also the *effective* per-peer probe interval. When the cycle exceeds 1.5s — any mesh with N ≥ 4 whose probe round-trips approach the 500ms deadline, or with one stalled peer delaying every later peer — a healthy peer probed early in the cycle is dropped from the live view (`alive()` at line 119: `time.Since(seen) <= peerLease`) before its next probe, every cycle. The live view oscillates, `forwardTo` falls back to the placement rule, and the stream's leader can be considered dead while it is answering (spurious promotion attempts, misrouted relays, ErrLeaderUnavailable to clients). Evidence (grep of promote.go): `34: heartbeatInterval = 500 * time.Millisecond`, `40: peerLease = 3 * heartbeatInterval`, `236: probeCtx, cancel := context.WithTimeout(ctx, heartbeatInterval)`. Fix: probe peers concurrently per tick (or stagger per-peer tickers), or derive the lease from the cycle: `peerLease >= N * heartbeatInterval`.
promote.go Outdated
@ -0,0 +644,4 @@
return fmt.Errorf("announcing the term on %q: %w", stream, err)
}
base, next := state.base(), state.nextSeq()
Member

🟡 SUGGESTION — the Reset's base and the window snapshot are separate locked reads while the age loop can run. state.makeActive(ctx) (line 515) restarts the age trim before rebuildCommandLog (634), and trimExpired advances the base under s.lock; base, next := state.base(), state.nextSeq() (647) and window := state.windowRecords() (652) are separate critical sections, so a trim can land between them — the comment at 647 ('read under the stream's lock in one go') holds for the window but not for the Reset's base. The resulting Reset{oldBase} + Appends-from-newBase is only repaired downstream because applyAppend resyncs on the record gap (streamInfo + resetToLocked) — correct, but it costs every replaying replica a full resync and keeps the stale base alive longer via pinFloor/windowFloor. Fix: take base+window in one exclusive read, or rebuild before makeActive.

🟡 SUGGESTION — the Reset's base and the window snapshot are separate locked reads while the age loop can run. `state.makeActive(ctx)` (line 515) restarts the age trim before `rebuildCommandLog` (634), and `trimExpired` advances the base under `s.lock`; `base, next := state.base(), state.nextSeq()` (647) and `window := state.windowRecords()` (652) are separate critical sections, so a trim can land between them — the comment at 647 ('read under the stream's lock in one go') holds for the window but not for the Reset's base. The resulting `Reset{oldBase}` + Appends-from-newBase is only repaired downstream because `applyAppend` resyncs on the record gap (streamInfo + resetToLocked) — correct, but it costs every replaying replica a full resync and keeps the stale base alive longer via `pinFloor`/`windowFloor`. Fix: take base+window in one exclusive read, or rebuild before `makeActive`.
@ -0,0 +855,4 @@
return quorumReply{}, false
}
mesh.drop(ctx, id, conn)
Member

🟠 IMPORTANT — a transient error on the promotion's read quorum deletes the holder from the local live view, and the deliberate retry cannot undo it. mesh.drop (server.go:818) calls m.markDead(id) (server.go:834) whenever the failed request was on the cached control Conn. askHolderOnce drops the Conn precisely in the transient case the retry exists for ('one transient transport error must not fail a read quorum', askHolder doc) — the retry dials a fresh Conn and succeeds — but the success path returns the reply without any markAlive. The only sender-side markAlive in the codebase is probePeer (promote.go:266); mesh.forward's relay success also never marks alive. So the holder stays dead in this node's live view until the next heartbeat (≤500ms). In that window forwardTo skips the announced leader (s.mesh.alive(leader) false) and routes to the placement candidate — which is not serving — so the stream's requests are refused (codeFor maps ErrNotLeader to codeLeaderUnavailable), and if that candidate is this node it attempts a promotion whose rejoin check is skipped because top == termValue (equal term), promoting over a live announced leader. One transient error can thus migrate a stream's leadership spuriously. Evidence: grep -n 'markAlive|markDead' *.go shows sender-side markAlive only at promote.go:266; markDead at server.go:834 (inside drop) and promote.go:246/259 (probePeer); the askHolderOnce source at 829-866. Fix: markAlive(id) on the retry's success (and on mesh.forward success), or re-arm the peer when a fresh Conn round-trip succeeds.

🟠 IMPORTANT — a transient error on the promotion's read quorum deletes the holder from the local live view, and the deliberate retry cannot undo it. `mesh.drop` (server.go:818) calls `m.markDead(id)` (server.go:834) whenever the failed request was on the cached control Conn. `askHolderOnce` drops the Conn precisely in the transient case the retry exists for ('one transient transport error must not fail a read quorum', askHolder doc) — the retry dials a fresh Conn and succeeds — but the success path returns the reply without any `markAlive`. The only sender-side `markAlive` in the codebase is `probePeer` (promote.go:266); `mesh.forward`'s relay success also never marks alive. So the holder stays dead in this node's live view until the next heartbeat (≤500ms). In that window `forwardTo` skips the announced leader (`s.mesh.alive(leader)` false) and routes to the placement candidate — which is not serving — so the stream's requests are refused (codeFor maps ErrNotLeader to codeLeaderUnavailable), and if that candidate is this node it attempts a promotion whose rejoin check is skipped because `top == termValue` (equal term), promoting over a live announced leader. One transient error can thus migrate a stream's leadership spuriously. Evidence: `grep -n 'markAlive|markDead' *.go` shows sender-side markAlive only at promote.go:266; markDead at server.go:834 (inside drop) and promote.go:246/259 (probePeer); the askHolderOnce source at 829-866. Fix: `markAlive(id)` on the retry's success (and on `mesh.forward` success), or re-arm the peer when a fresh Conn round-trip succeeds.
promote.go Outdated
@ -0,0 +1101,4 @@
break
}
size := len(encodeRecordFrame(rec)) + wireUint32Size
Member

🟡 SUGGESTION — each record is encoded twice on the copy path. size := len(encodeRecordFrame(rec)) + wireUint32Size builds a full encoded frame per record purely for the byte budget, and writeRecordPage then encodes the same record again on the wire. For a full 256-record page of ~1KB records that is ~512KB of throwaway encode/allocations per page while copying a backlog. A field-based size estimate (seq 8 + timestamp 8 + dedupID 4+len + data 4+len + frame prefixes) avoids the discarded encode.

🟡 SUGGESTION — each record is encoded twice on the copy path. `size := len(encodeRecordFrame(rec)) + wireUint32Size` builds a full encoded frame per record purely for the byte budget, and `writeRecordPage` then encodes the same record again on the wire. For a full 256-record page of ~1KB records that is ~512KB of throwaway encode/allocations per page while copying a backlog. A field-based size estimate (seq 8 + timestamp 8 + dedupID 4+len + data 4+len + frame prefixes) avoids the discarded encode.
Resolves the four findings of the PR #18 review.

The peer lease is three probe intervals, and `heartbeatOnce` probed the peers
one at a time, each bounded by one interval: a cycle grew with the mesh (or with
one stalled peer), so at four or more nodes — or behind anything slow — a
healthy peer probed early fell out of the live view before its next probe, every
cycle. The probes now run concurrently, so a cycle costs one interval however
many peers there are. TestHeartbeatCycleCostsOneInterval measures the serial
cost first (two peers whose handler never answers, so a sequential probe really
does spend its deadline) and then asserts one cycle fits in one interval.

A successful request never recorded its peer as alive: only `probePeer` did. A
transient transport error therefore outlived the request that recovered from it
— the promotion's read-quorum retry re-dialed, got its answer, and left the
holder dead in the live view until the next probe; in that window the relay
skipped the announced leader and routed at a node that does not serve the
stream, and a candidate could promote over a live leader at an equal term. Every
answered round trip now records it: `mesh.forward`, `forwardOwnConn`,
`askHolderOnce`, `sendTermAnnouncement` and the record copy, refusals included
(`refusedByPeer` distinguishes a peer's own answer from a transport failure, and
a refusal no longer drops the Conn it arrived on). A *dial* is deliberately not
evidence: a stopped server's host still negotiates the protocol and only the
request fails. TestAnsweredRequestRestoresALivePeer covers a refused relay, an
answered relay and a loop-guard refusal.

`takeOver` rebuilt the command log after restarting the age loop, and the
Reset's base and the records it restates were separate locked reads, so a trim
could land between them and make every replaying replica resync. It rebuilds
first now, and `windowSnapshot` returns the base and the records under one hold.

`serveFetchRecords` built a whole encoded frame per record just to measure it;
`recordPageSize` computes the same number from the fields, and
TestRecordPageSizeMatchesTheFrame pins it to `encodeRecordFrame`.

Also fixes a flake the review's tests exposed: TestConcurrentPromotionOf-
TheCandidateIsIdempotent promoted before a survivor held the stream, so its read
quorum counted a holder that answers "stream not found" as a zero reply.
pi-agent left a comment

Findings resolved in eb7e4f4

All four addressed; each fix has a test that fails without it.

#1 (IMPORTANT, serial probe loop vs the lease) — fixed. heartbeatOnce probes every peer concurrently via sync.WaitGroup.Go, so one cycle costs one heartbeatInterval regardless of mesh size or one stalled peer, which is what the 3× lease is derived from. TestHeartbeatCycleCostsOneInterval (synctest) installs a handler on two peers that answers the protocol negotiation and then never replies, measures the sequential cost first (asserts it exceeds one interval, so the test cannot pass vacuously), then asserts one cycle fits inside one interval.

#2 (IMPORTANT, no markAlive on a successful request) — fixed, and it was wider than the retry. Every answered round trip now records liveness: mesh.forward, forwardOwnConn, askHolderOnce, sendTermAnnouncement and copyFrom. A refusal counts as an answer (refusedByPeer distinguishes a peer's own reply from a transport failure), and a refusal no longer drops the Conn it arrived on. A dial is deliberately not evidence: in this review round I first added markAlive to the successful-dial paths too, and TestRepeatedPromotionsAdvanceTheTerm caught it — a stopped server's host still negotiates the protocol, so only the request proves the node serves. TestAnsweredRequestRestoresALivePeer covers three cases: a relayed refusal (resp.err() == ErrConsumerNotFound; note Conn.forward returns an inner refusal as a successful relay, which the test pins), an answered relay, and a loop-guard refusal (ErrForwardLoop).

#3 (SUGGESTION, base and window in separate critical sections) — fixed both ways. takeOver now rebuilds the command log before state.makeActive restarts the age loop, and windowSnapshot() returns the Reset's base and the retained records under one hold, so no trim can land between them. The Reset and the window it describes therefore always agree, and a replaying replica no longer pays a resync for it.

#4 (SUGGESTION, double encode on the copy path) — fixed. recordPageSize/recordFrameSize (store.go, next to encodeRecordFrame) compute the wire size from the fields; serveFetchRecords no longer builds a throwaway frame per record. TestRecordPageSizeMatchesTheFrame pins the arithmetic to encodeRecordFrame across empty, typical and 1 KiB records.

Extra: the review's own test surfaced a flake in mine — TestConcurrentPromotionOfTheCandidateIsIdempotent promoted before a survivor held the stream, so that holder's "stream not found" counted as a zero reply to the read quorum. It now waits for both survivors to hold the record first.

Verification at eb7e4f4: go vet (+-tags rapid), golangci-lint run (+--build-tags rapid) 0 issues; go test -race ./... green 14 consecutive runs (8 plain, 6 with -coverprofile) and -tags rapid green; fuzz smoke runs unchanged. Coverage 89.0 → 89.1 %, gremlins 1350/96/93.36 % → 1369/97/93.38 %; the only uncovered mutants in the new code remain the const arithmetic of the three intervals and the seam-needing probeTerm→adoptTerm failure branch (coverage.log records it).

## Findings resolved in eb7e4f4 All four addressed; each fix has a test that fails without it. **#1 (IMPORTANT, serial probe loop vs the lease) — fixed.** `heartbeatOnce` probes every peer concurrently via `sync.WaitGroup.Go`, so one cycle costs one `heartbeatInterval` regardless of mesh size or one stalled peer, which is what the 3× lease is derived from. `TestHeartbeatCycleCostsOneInterval` (synctest) installs a handler on two peers that answers the protocol negotiation and then never replies, *measures* the sequential cost first (asserts it exceeds one interval, so the test cannot pass vacuously), then asserts one cycle fits inside one interval. **#2 (IMPORTANT, no `markAlive` on a successful request) — fixed, and it was wider than the retry.** Every answered round trip now records liveness: `mesh.forward`, `forwardOwnConn`, `askHolderOnce`, `sendTermAnnouncement` and `copyFrom`. A refusal counts as an answer (`refusedByPeer` distinguishes a peer's own reply from a transport failure), and a refusal no longer drops the Conn it arrived on. A *dial* is deliberately **not** evidence: in this review round I first added `markAlive` to the successful-dial paths too, and `TestRepeatedPromotionsAdvanceTheTerm` caught it — a stopped server's host still negotiates the protocol, so only the request proves the node serves. `TestAnsweredRequestRestoresALivePeer` covers three cases: a relayed refusal (`resp.err() == ErrConsumerNotFound`; note `Conn.forward` returns an inner refusal as a *successful* relay, which the test pins), an answered relay, and a loop-guard refusal (`ErrForwardLoop`). **#3 (SUGGESTION, base and window in separate critical sections) — fixed both ways.** `takeOver` now rebuilds the command log *before* `state.makeActive` restarts the age loop, and `windowSnapshot()` returns the Reset's base and the retained records under one hold, so no trim can land between them. The Reset and the window it describes therefore always agree, and a replaying replica no longer pays a resync for it. **#4 (SUGGESTION, double encode on the copy path) — fixed.** `recordPageSize`/`recordFrameSize` (`store.go`, next to `encodeRecordFrame`) compute the wire size from the fields; `serveFetchRecords` no longer builds a throwaway frame per record. `TestRecordPageSizeMatchesTheFrame` pins the arithmetic to `encodeRecordFrame` across empty, typical and 1 KiB records. **Extra:** the review's own test surfaced a flake in mine — `TestConcurrentPromotionOfTheCandidateIsIdempotent` promoted before a survivor held the stream, so that holder's "stream not found" counted as a zero reply to the read quorum. It now waits for both survivors to hold the record first. Verification at eb7e4f4: `go vet` (+`-tags rapid`), `golangci-lint run` (+`--build-tags rapid`) 0 issues; `go test -race ./...` green 14 consecutive runs (8 plain, 6 with `-coverprofile`) and `-tags rapid` green; fuzz smoke runs unchanged. Coverage 89.0 → 89.1 %, gremlins 1350/96/93.36 % → 1369/97/93.38 %; the only uncovered mutants in the new code remain the const arithmetic of the three intervals and the seam-needing `probeTerm`→`adoptTerm` failure branch (`coverage.log` records it).
Comment-only pass over this branch, in the style PR #17 set: a comment stays
only if it carries a locking or ordering constraint, a failure mode, an on-disk
or wire invariant, or why an alternative was rejected. Exported identifiers keep
a short godoc line opening with their name; a comment that restates the code is
gone; `*_test.go` carries no comments at all.

promote.go loses its design essay — the rules live in doc.go — and each
remaining comment is one to three lines: the invariants are the peer lease's
relation to the probe cycle, the write quorum's intersection with a promotion's
read quorum, the recorded holder set versus the live view, the Reset's place in
a rebuilt log, the one-shot `markAlive` on every answered round trip, and the
floor that keeps a rebuilt window's Reset alive. The same trim is applied to the
comments this branch added in server.go, replicate.go, stream_state.go,
protocol.go and store.go; doc.go keeps the architecture it is the only home for
(failover, the command log's command set, quorum acks) in 141 lines instead of
the 148 it had.

Net: 395 comment lines removed (642 removed, 247 added across 14 files), and
0 comment lines in `*_test.go` — the same standard as PR #17.

No code changed: the diff touches only comments and blank lines.
pi-agent left a comment

Comment pass (3941dea)

Done, to the standard PR #17 sets: an exported identifier keeps one short godoc line opening with its name; every other comment either summarises a constraint the code cannot show (locking/ordering, failure mode, on-disk/wire invariant) or is gone. *_test.go now carries no comments at all.

before this pass after
comment lines added by this branch 420 312
promote.go doc comments 196 130
promotion_test.go comments 87 0
comment lines in *_test.go (whole repo) 2016 + mine 2 (//go:build rapid, //go:noinline)

The removals were mostly narration that restated the following statement ("Every acked sequence is on the new leader: …", "It serves, and the survivors converge…", "Highest position first: it can fill the most", the step-by-step lists inside test bodies). What stayed, in one to three lines each, is the set of facts a reader cannot recover locally: the lease's relation to the probe cycle, the write quorum intersecting a promotion's read quorum, the recorded holder set versus the live view, why a dial is not liveness evidence but a refusal is, the Reset's place in a rebuilt log and the floor that keeps it alive, the writeLock that makes appendCommandPosition's read-back position exact, and the frameForward-only path that makes an internal name unreachable to a client.

doc.go went 148 → 141 lines, keeping the failover architecture it is the only home for.

No code changed: go/scanner token streams (comments excluded) of all 14 changed files against the parent commit are identical. gofmt -l clean, go vet (± rapid) clean, golangci-lint (± --build-tags rapid) 0 issues, go test -race ./... and -race -tags rapid ./... green.

## Comment pass (3941dea) Done, to the standard PR #17 sets: an exported identifier keeps one short godoc line opening with its name; every other comment either summarises a constraint the code cannot show (locking/ordering, failure mode, on-disk/wire invariant) or is gone. `*_test.go` now carries no comments at all. | | before this pass | after | |---|---|---| | comment lines added by this branch | 420 | 312 | | `promote.go` doc comments | 196 | 130 | | `promotion_test.go` comments | 87 | 0 | | comment lines in `*_test.go` (whole repo) | 2016 + mine | 2 (`//go:build rapid`, `//go:noinline`) | The removals were mostly narration that restated the following statement ("Every acked sequence is on the new leader: …", "It serves, and the survivors converge…", "Highest position first: it can fill the most", the step-by-step lists inside test bodies). What stayed, in one to three lines each, is the set of facts a reader cannot recover locally: the lease's relation to the probe cycle, the write quorum intersecting a promotion's read quorum, the recorded holder set versus the live view, why a dial is not liveness evidence but a refusal is, the Reset's place in a rebuilt log and the floor that keeps it alive, the writeLock that makes `appendCommandPosition`'s read-back position exact, and the `frameForward`-only path that makes an internal name unreachable to a client. `doc.go` went 148 → 141 lines, keeping the failover architecture it is the only home for. No code changed: `go/scanner` token streams (comments excluded) of all 14 changed files against the parent commit are identical. `gofmt -l` clean, `go vet` (± `rapid`) clean, `golangci-lint` (± `--build-tags rapid`) 0 issues, `go test -race ./...` and `-race -tags rapid ./...` green.
sam merged commit a436f15e7a into main 2026-10-20 15:32:24 +00:00
sam deleted branch feat/phase-3-failure-detection-and-promotion 2026-10-20 15:32:24 +00:00
Sign in to join this conversation.
No reviewers
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
norobo/webstream!18
No description provided.