feat: dynamic membership (plan.md Phase 5, partial) #20

Merged
sam merged 6 commits from feat/phase-5-dynamic-membership into main 2026-10-28 07:51:13 +00:00
Member

What this lands

Plan.md Phase 5, partially: the replicated membership state and its epoch rules, the AddPeer/RemovePeer path, and the refusal a node that is behind gives. It does not land the end-to-end learner-join and stale-config-rejoin cases; see "Open items", which is the reason this is not marked shipped.

The mechanism

A cluster's node set is now replicated state, not a local option.

  • WithMeshPeers is config epoch 0 — the bootstrap set. The set is what names the nodes that would tail the log that carries it, so it has to start local.
  • Server.AddPeer(ctx, id) / Server.RemovePeer(ctx, id) publish Config{epoch, peers} as a record on the reserved cluster stream internal-cluster, which every member consumes. A change requested on a node that does not lead that stream is relayed (new peer-only frame frameConfigChange, new file membership.go).
  • The cluster log is an ordinary replicated stream — same command log (internal-internal-cluster), term, placement epoch and record path — so an epoch reaches a node exactly the way a record does. isCommandLog names the distinction: a command log is written by its leader and never replicated, while the cluster log is internal but replicated. The guards that meant "this is a command log" (log creation, promotion, replicate-loop realignment) test that predicate instead of the internal- prefix.
  • Every member holds the cluster log regardless of placement (holdsStream, holdsPlacement and releaseDisplacedCopies all special-case it): the set is what placement, R and every quorum are derived from.
  • A restart reads the newest epoch back from its own copy of the log (restoreConfig), the same reasoning that has restoreHolders read placement from a stream's command log. No second on-disk field.

Epoch rules

  • Adopt above, refuse below; an equal epoch naming a different set is the loud mismatch and reports ErrForwardLoop, the code the relay's loop guard already uses to mean "two nodes disagree about membership".
  • A config record is acknowledged by a quorum of the set that can hold it: the existing members for an addition (the joiner has no copy until the record gives it one), the remaining members for a removal — so a removal is never carried by a majority of the nodes it removes. confirmQuorum accepts a config record's voter only from the member set the record names, which is the same set the count is taken from (TestConfigQuorumFitsTheVoters).
  • The writer adopts the epoch as soon as the record is durable locally and then waits for the quorum, so a failed wait costs the operator a confirmation, not the change.
  • A node an applied config no longer names (depart) refuses writes, releases its copies and stops its replica loops but keeps the cluster log and its position on it. The writers delete the departed node's rep-<peerID> consumers once the change is acknowledged (releaseRemovedPeers) — the release half of the two-step removal, and what unpins the command logs.
  • An added peer is a learner through the machinery Phase 4 already ships: it holds and tails a stream the rule names it for, and a placement epoch admits it only once holdersCaughtUp sees its replica consumer inside the retained window; until then holderVotes drops its acks.
  • Being behind is learned from the probe: the heartbeat carries the sender's applied epoch and answers with the receiver's (heartbeatRequest, response.epoch). The body is optional in both directions, so a peer that does not send one is not read as dead. ensureServing refuses mutations with ErrForwardLoop until the cluster log catches the node up; reads keep being served.

Review follow-up (21e56b6)

  • Blocking: a config record could never be confirmed on a mesh wider than R — the voter gate was the cluster log's placement holders (R = min(3, N)) while the count was over the member set. Both are now the member set the record names.
  • Important: the heartbeat's body is optional both ways, so a peer without the field is not declared dead by every upgraded node.
  • Suggestion: Placements() is user streams again; ledResidentStreams() says that it includes the cluster log.
  • Not raised, found while fixing the above: the promotion's read quorum was counted over the whole member set whenever the promoter had no applied placement epoch (which happens after its copy is evicted and reloaded — only a leader restores the epoch from its own command log). The frameQuorumRead reply now carries the replier's placement record, the votes are counted over the newest one, and the promoter adopts it before recording its own epoch. TestPromotionDoesNotNameAnUncaughtUpHolder went from ~2 in 10 failing to 10 in 10.

Tests and their state

  • TestConfigEpochsAreMonotone (rapid), TestConfigQuorumFitsTheVoters, TestObservedConfigEpochRefusesWrites, TestConfigRecordsRoundTrip, TestHeartbeatBodyIsOptional, TestPlacementsHidesTheClusterLog, TestReadQuorumCountsTheRecordedSet.
  • TestRemovingAPeerReleasesItsPin — t.Skipped (see below).
  • Two pre-existing tests asserted "a replica holds no internal stream at all" and "a replica has no replica bookkeeping at all"; they now assert their own stream, because a replica legitimately holds and registers the cluster log.

Open items (why this is not "shipped")

A follower that has not yet reached a config record can ack past it. Its cluster-log consumer watermark runs to the end while its own copy holds nothing, the leader's command log then trims to that watermark, and the record is gone before the follower can apply it — so that node never adopts the epoch. Reproduction: go test -run TestRemovingAPeerReleasesItsPin -count=20 . (passes in isolation, fails in most runs of the whole package). This is a retention/pinning problem in the cluster log's own command log, not load sensitivity. The learner-join and stale-config-rejoin cases were removed rather than skipped because they hit the same path.

Phase 6 already owns the fix: every node becomes a mandatory consumer of the cluster log, registered like a replica, with retention following the slowest member instead of a timer. That bounds the trim with the slowest watermark, and those three cases should be restored with it.

Validation

  • go vet ./..., go vet -tags rapid ./...: clean.
  • golangci-lint run, golangci-lint run --build-tags rapid: 0 issues.
  • go test -race . and go test -race -tags rapid .: green.
  • make gremlins dry run was not re-run for this landing.
## What this lands Plan.md Phase 5, **partially**: the replicated membership state and its epoch rules, the `AddPeer`/`RemovePeer` path, and the refusal a node that is behind gives. It does **not** land the end-to-end learner-join and stale-config-rejoin cases; see "Open items", which is the reason this is not marked shipped. ## The mechanism A cluster's node set is now replicated state, not a local option. - `WithMeshPeers` is **config epoch 0** — the bootstrap set. The set is what names the nodes that would tail the log that carries it, so it has to start local. - `Server.AddPeer(ctx, id)` / `Server.RemovePeer(ctx, id)` publish `Config{epoch, peers}` as a record on the reserved cluster stream `internal-cluster`, which every member consumes. A change requested on a node that does not lead that stream is relayed (new peer-only frame `frameConfigChange`, new file `membership.go`). - The cluster log is an **ordinary replicated stream** — same command log (`internal-internal-cluster`), term, placement epoch and record path — so an epoch reaches a node exactly the way a record does. `isCommandLog` names the distinction: a command log is written by its leader and never replicated, while the cluster log is internal but replicated. The guards that meant "this is a command log" (log creation, promotion, replicate-loop realignment) test that predicate instead of the `internal-` prefix. - Every member holds the cluster log regardless of placement (`holdsStream`, `holdsPlacement` and `releaseDisplacedCopies` all special-case it): the set is what placement, `R` and every quorum are derived from. - A restart reads the newest epoch back from its own copy of the log (`restoreConfig`), the same reasoning that has `restoreHolders` read placement from a stream's command log. No second on-disk field. ## Epoch rules - Adopt above, refuse below; an **equal epoch naming a different set** is the loud mismatch and reports `ErrForwardLoop`, the code the relay's loop guard already uses to mean "two nodes disagree about membership". - A config record is acknowledged by a quorum of **the set that can hold it**: the existing members for an addition (the joiner has no copy until the record gives it one), the remaining members for a removal — so a removal is never carried by a majority of the nodes it removes. `confirmQuorum` accepts a config record's voter only from the member set the record names, which is the same set the count is taken from (`TestConfigQuorumFitsTheVoters`). - The writer adopts the epoch as soon as the record is durable locally and *then* waits for the quorum, so a failed wait costs the operator a confirmation, not the change. - A node an applied config no longer names (`depart`) refuses writes, releases its copies and stops its replica loops but keeps the cluster log and its position on it. The writers delete the departed node's `rep-<peerID>` consumers once the change is acknowledged (`releaseRemovedPeers`) — the release half of the two-step removal, and what unpins the command logs. - An added peer is a learner through the machinery Phase 4 already ships: it holds and tails a stream the rule names it for, and a placement epoch admits it only once `holdersCaughtUp` sees its replica consumer inside the retained window; until then `holderVotes` drops its acks. - Being behind is learned from the probe: the heartbeat carries the sender's applied epoch and answers with the receiver's (`heartbeatRequest`, `response.epoch`). The body is optional in both directions, so a peer that does not send one is not read as dead. `ensureServing` refuses mutations with `ErrForwardLoop` until the cluster log catches the node up; reads keep being served. ## Review follow-up (21e56b6) - **Blocking:** a config record could never be confirmed on a mesh wider than `R` — the voter gate was the cluster log's placement holders (`R = min(3, N)`) while the count was over the member set. Both are now the member set the record names. - **Important:** the heartbeat's body is optional both ways, so a peer without the field is not declared dead by every upgraded node. - **Suggestion:** `Placements()` is user streams again; `ledResidentStreams()` says that it includes the cluster log. - **Not raised, found while fixing the above:** the promotion's read quorum was counted over the whole member set whenever the promoter had no applied placement epoch (which happens after its copy is evicted and reloaded — only a leader restores the epoch from its own command log). The frameQuorumRead reply now carries the replier's placement record, the votes are counted over the newest one, and the promoter adopts it before recording its own epoch. `TestPromotionDoesNotNameAnUncaughtUpHolder` went from ~2 in 10 failing to 10 in 10. ## Tests and their state - `TestConfigEpochsAreMonotone` (rapid), `TestConfigQuorumFitsTheVoters`, `TestObservedConfigEpochRefusesWrites`, `TestConfigRecordsRoundTrip`, `TestHeartbeatBodyIsOptional`, `TestPlacementsHidesTheClusterLog`, `TestReadQuorumCountsTheRecordedSet`. - `TestRemovingAPeerReleasesItsPin` — `t.Skip`ped (see below). - Two pre-existing tests asserted "a replica holds no internal stream at all" and "a replica has no replica bookkeeping at all"; they now assert their own stream, because a replica legitimately holds and registers the cluster log. ## Open items (why this is not "shipped") **A follower that has not yet reached a config record can ack past it.** Its cluster-log consumer watermark runs to the end while its own copy holds nothing, the leader's command log then trims to that watermark, and the record is gone before the follower can apply it — so that node never adopts the epoch. Reproduction: `go test -run TestRemovingAPeerReleasesItsPin -count=20 .` (passes in isolation, fails in most runs of the whole package). This is a retention/pinning problem in the cluster log's own command log, not load sensitivity. The learner-join and stale-config-rejoin cases were removed rather than skipped because they hit the same path. Phase 6 already owns the fix: every node becomes a mandatory consumer of the cluster log, registered like a replica, with retention following the slowest member instead of a timer. That bounds the trim with the slowest watermark, and those three cases should be restored with it. ## Validation - `go vet ./...`, `go vet -tags rapid ./...`: clean. - `golangci-lint run`, `golangci-lint run --build-tags rapid`: 0 issues. - `go test -race .` and `go test -race -tags rapid .`: green. - `make gremlins` dry run was not re-run for this landing.
The mesh node set becomes replicated state with an epoch instead of a local
option. WithMeshPeers is config epoch 0 — the bootstrap set, because the set is
what names the nodes that would tail the log carrying it — and Server.AddPeer /
Server.RemovePeer publish the next Config{epoch, peers} on the reserved cluster
stream internal-cluster, which every member consumes.

The cluster log is an ordinary replicated stream: same command log, term,
placement epoch and record path, so an epoch reaches a node the way a record
does. isCommandLog names the distinction between a stream's command log (written
by its leader, never replicated) and this internal-but-replicated stream, and the
guards that meant "a command log" test it instead of the name prefix. Every
member holds it regardless of placement: the set is what placement, R and every
quorum are derived from. A restart reads the newest epoch back from its own copy
(restoreConfig), the same reasoning that has restoreHolders read placement from
the command log.

Epochs: a node adopts one above its own, refuses one below it, and reports an
equal epoch naming a different set as ErrForwardLoop — the membership-
disagreement code the relay's loop guard already uses. A config record is
acknowledged by a quorum of the set that can hold it: the existing members for an
addition (the joiner has no copy until the record gives it one), the remaining
members for a removal, so a removal is never carried by a majority of the nodes
it removes; confirmQuorum drops a config record's voter the record does not
name. The writer adopts the epoch as soon as the record is durable locally and
then waits, so a failed wait costs a confirmation and not the change.

An added peer is a learner through the machinery Phase 4 already ships: it holds
and tails a stream the rule names it for and is admitted to a placement epoch only
once holdersCaughtUp sees its replica consumer inside the retained window; until
then holderVotes drops its acks. A node an applied config no longer names
(depart) refuses writes, releases its copies and stops its loops but keeps the
cluster log, and the writers delete the departed node's rep-<peerID> consumers
once the change is acknowledged (releaseRemovedPeers), which is the release half
of the two-step removal and what unpins the command logs.

Being behind is learned from the probe: the heartbeat carries the sender's applied
epoch and answers with the receiver's, and ensureServing refuses a mutation with
ErrForwardLoop until the cluster log catches the node up, while reads keep being
served.

Tests: TestConfigEpochsAreMonotone (rapid: the applied epoch never walks back, an
equal epoch with another set is the loud mismatch, the node never reports itself
behind after applying), TestObservedConfigEpochRefusesWrites,
TestConfigRecordsRoundTrip, and TestRemovingAPeerReleasesItsPin — the last is
SKIPPED in this landing: the config record reaches the other survivors only while
the cluster log's follower loop is tailing it, and that path is not stable under
load. plan.md records it as the phase's open item; Phase 6 makes every node a
mandatory consumer of the cluster log and should unskip it, together with the
learner-join and stale-config-rejoin cases this landing does not cover end to end.

Also: TestReconciliationDoesNotReplicateACommandLog and
TestStreamThatCannotBeAdoptedIsNotConfirmed asserted "no internal stream / no
replica bookkeeping at all" on a replica; both now assert their own stream, since
a replica legitimately holds and registers the cluster log.
docs: trim the phase 5 comments to what they carry
All checks were successful
code-review / Agent review (pull_request) Successful in 11s
2ccaea5e3c
The new comments had grown into design essays: multi-sentence blocks that
justified a choice the code and doc.go already state, and per-field paragraphs on
types whose fields are named for themselves.

Every block is now one or two lines: what a caller cannot read off the body — the
locking rule (members() hands out a slice that adoption replaces rather than
mutates), the failure direction (an equal epoch naming another set is
ErrForwardLoop), the quorum rule (a removal is never acknowledged by the nodes it
removes), and the reason the cluster log is exempt from a guard (it is internal but
replicated). The file preamble citing plan.md is gone; doc.go's Membership section
is the one place that states the architecture, and it dropped a third of its
length.

Numbers: 178 -> 109 comment lines in the added code, no comment block of four or
more lines left in membership.go or membership_test.go. No code change; go vet,
golangci-lint (both tags) and go test -race are clean.
Author
Member

Comment-only pass pushed (2ccaea5): 178 → 109 comment lines in the added code, no block of four or more lines left in membership.go / membership_test.go, and doc.go's Membership section dropped a third of its length. Each block now carries only what the body cannot: the locking rule (members() hands out a slice that adoption replaces, never mutates), the failure direction (an equal epoch naming another set is ErrForwardLoop), the quorum rule (a removal is never acknowledged by the nodes it removes), and why the cluster log is exempt from a guard (internal, but replicated). The file preamble that cited plan.md is gone — doc.go is the one place that states the architecture.

No code changed. Gates re-run: go vet and go vet -tags rapid clean, golangci-lint run (both tags) 0 issues, go test -race . green (38.0s). One -race run of three hit the two load-sensitive flakes already described above (TestMeshRelayFailureDropsOnlyABrokenConn, TestHolderLossReplacesTheHolders), neither of which this commit touches.

Comment-only pass pushed (2ccaea5): 178 → 109 comment lines in the added code, no block of four or more lines left in `membership.go` / `membership_test.go`, and `doc.go`'s Membership section dropped a third of its length. Each block now carries only what the body cannot: the locking rule (`members()` hands out a slice that adoption *replaces*, never mutates), the failure direction (an equal epoch naming another set is `ErrForwardLoop`), the quorum rule (a removal is never acknowledged by the nodes it removes), and why the cluster log is exempt from a guard (internal, but replicated). The file preamble that cited plan.md is gone — `doc.go` is the one place that states the architecture. No code changed. Gates re-run: `go vet` and `go vet -tags rapid` clean, `golangci-lint run` (both tags) 0 issues, `go test -race .` green (38.0s). One `-race` run of three hit the two load-sensitive flakes already described above (`TestMeshRelayFailureDropsOnlyABrokenConn`, `TestHolderLossReplacesTheHolders`), neither of which this commit touches.
pi-reviewer requested changes 2026-10-21 19:17:35 +00:00
Dismissed
pi-reviewer left a comment

AI review (pi-reviewer) — Phase 5 dynamic membership, partial landing

Reviewed at head 2ccaea5 against the PR description. Verified in a throwaway worktree at the head commit: go vet ./... clean, go build ./... clean, the documented skip (TestRemovingAPeerReleasesItsPin) skips as stated. No go.mod/go.sum changes — no new dependencies.

Two verified findings, one blocking, both reproduced with runnable tests I wrote in the worktree and then removed:

🔴 BLOCKING — config changes cannot reach quorum on larger meshes. configQuorum (membership.go) demands writeQuorum(eligible) − 1 confirmations from other nodes, but confirmQuorum (replicate.go) counts a config record's ack only when the voter is also a holder of the cluster stream's placement epoch (R = min(3, N) — the leader plus two). At N = 6 an AddPeer needs 3 other confirmations but only 2 other holders exist; at N = 5 a RemovePeer of a cluster-log holder needs 2 but only 1 other holder remains in the new set. Every such change fails with ErrNoQuorum after the 10 s deadline — deterministically, not a flake. Evidence:

  • go test -run TestReviewSixNodeAddPeerQuorum -v . → no quorum: stream "internal-cluster" record 0: the wait ended with 2 of 3 replicas confirmed: context deadline exceeded
  • go test -run TestReviewFiveNodeRemoveHolderQuorum -v . → no quorum: stream "internal-cluster" record 0: the wait ended with 1 of 2 replicas confirmed: context deadline exceeded

Because the writer adopts the epoch locally before the wait (by design), the failed call also leaves the leader with a divergent applied set. Fix: for the cluster log, the voter gate should be the member set the record names (cfg.Peers), not the placement's R = 3 holder set — every member holds the cluster log (holdsStream/holdsPlacement already special-case it), so the eligible set configQuorum counts and the voter set confirmQuorum accepts must coincide.

🟠 IMPORTANT — heartbeat frame shape changed without version negotiation; mixed-version meshes break in one direction. Pre-PR20 peers answer frameHeartbeat with an empty body (base 2a1c05a: serveHeartbeat() response { return response{} }). The new decodeResponse reads 8 bytes (resp.epoch = reader.readUint64()); an old peer's empty reply fails decode, and probePeer then does markDead(id) + drop(...) — so every upgraded node marks every not-yet-upgraded node dead on each probe cycle, collapsing the live view. Evidence: go test -run TestReviewHeartbeatEmptyBodyFromOldPeer -v . → decodeResponse(frameHeartbeat, empty) -> decoding response: unexpected EOF. The PR's own framing — frameConfigChange "declared last so no existing kind number moves on the wire" — indicates wire compatibility was intended; the heartbeat shape change contradicts it. If all nodes always upgrade together, please say so in the PR; otherwise gate the new body on the request carrying an epoch field (empty request → empty reply).

🟡 SUGGESTION — Server.Placements() now lists the internal cluster stream. The guard in Placements() changed from isInternalName(name) to isCommandLog(name) (placement.go:163), so the operator-facing listing now includes internal-cluster while the doc comment still says "every user stream this node has loaded". Either skip clusterStream too or update the doc.

Not re-raised: the documented open item (skipped TestRemovingAPeerReleasesItsPin, unstable cluster-log follower path under load) is stated honestly in the PR body and owned by Phase 6 — acknowledged, not re-tagged.

Note: the blocking finding is distinct from the documented open item — the quorum arithmetic is a deterministic failure in the code as written, not load sensitivity.

## AI review (pi-reviewer) — Phase 5 dynamic membership, partial landing Reviewed at head `2ccaea5` against the PR description. Verified in a throwaway worktree at the head commit: `go vet ./...` clean, `go build ./...` clean, the documented skip (`TestRemovingAPeerReleasesItsPin`) skips as stated. No go.mod/go.sum changes — no new dependencies. Two verified findings, one blocking, both reproduced with runnable tests I wrote in the worktree and then removed: **🔴 BLOCKING — config changes cannot reach quorum on larger meshes.** `configQuorum` (membership.go) demands `writeQuorum(eligible) − 1` confirmations from *other* nodes, but `confirmQuorum` (replicate.go) counts a config record's ack only when the voter is also a holder of the cluster stream's placement epoch (`R = min(3, N)` — the leader plus two). At N = 6 an `AddPeer` needs 3 other confirmations but only 2 other holders exist; at N = 5 a `RemovePeer` of a cluster-log holder needs 2 but only 1 other holder remains in the new set. Every such change fails with `ErrNoQuorum` after the 10 s deadline — deterministically, not a flake. Evidence: - `go test -run TestReviewSixNodeAddPeerQuorum -v .` → `no quorum: stream "internal-cluster" record 0: the wait ended with 2 of 3 replicas confirmed: context deadline exceeded` - `go test -run TestReviewFiveNodeRemoveHolderQuorum -v .` → `no quorum: stream "internal-cluster" record 0: the wait ended with 1 of 2 replicas confirmed: context deadline exceeded` Because the writer adopts the epoch locally before the wait (by design), the failed call also leaves the leader with a divergent applied set. Fix: for the cluster log, the voter gate should be the member set the record names (`cfg.Peers`), not the placement's R = 3 holder set — every member holds the cluster log (`holdsStream`/`holdsPlacement` already special-case it), so the eligible set `configQuorum` counts and the voter set `confirmQuorum` accepts must coincide. **🟠 IMPORTANT — heartbeat frame shape changed without version negotiation; mixed-version meshes break in one direction.** Pre-PR20 peers answer `frameHeartbeat` with an empty body (base `2a1c05a`: `serveHeartbeat() response { return response{} }`). The new `decodeResponse` reads 8 bytes (`resp.epoch = reader.readUint64()`); an old peer's empty reply fails decode, and `probePeer` then does `markDead(id)` + `drop(...)` — so every upgraded node marks every not-yet-upgraded node dead on each probe cycle, collapsing the live view. Evidence: `go test -run TestReviewHeartbeatEmptyBodyFromOldPeer -v .` → `decodeResponse(frameHeartbeat, empty) -> decoding response: unexpected EOF`. The PR's own framing — frameConfigChange "declared last so no existing kind number moves on the wire" — indicates wire compatibility was intended; the heartbeat shape change contradicts it. If all nodes always upgrade together, please say so in the PR; otherwise gate the new body on the request carrying an epoch field (empty request → empty reply). **🟡 SUGGESTION — `Server.Placements()` now lists the internal cluster stream.** The guard in `Placements()` changed from `isInternalName(name)` to `isCommandLog(name)` (placement.go:163), so the operator-facing listing now includes `internal-cluster` while the doc comment still says "every user stream this node has loaded". Either skip `clusterStream` too or update the doc. Not re-raised: the documented open item (skipped `TestRemovingAPeerReleasesItsPin`, unstable cluster-log follower path under load) is stated honestly in the PR body and owned by Phase 6 — acknowledged, not re-tagged. Note: the blocking finding is distinct from the documented open item — the quorum arithmetic is a deterministic failure in the code as written, not load sensitivity.
placement.go Outdated
@ -151,3 +161,3 @@
for name, rs := range s.streams {
if rs.stream == nil || isInternalName(name) {
if rs.stream == nil || isCommandLog(name) {
Member

🟡 SUGGESTION — Placements() now leaks the internal cluster stream into the operator-facing listing.

The guard changed from isInternalName(name) to isCommandLog(name), so internal-cluster (which is internal but replicated, not a command log) is now included, while the doc comment still says "the placement of every user stream this node has loaded". A client cannot name internal-cluster anywhere, so it should not surface in Placements() either — skip name == clusterStream here — or, if deliberate, update the doc comment to say the cluster log is included.

🟡 SUGGESTION — `Placements()` now leaks the internal cluster stream into the operator-facing listing. The guard changed from `isInternalName(name)` to `isCommandLog(name)`, so `internal-cluster` (which is internal but replicated, not a command log) is now included, while the doc comment still says "the placement of every user stream this node has loaded". A client cannot name `internal-cluster` anywhere, so it should not surface in `Placements()` either — skip `name == clusterStream` here — or, if deliberate, update the doc comment to say the cluster log is included.
sam marked this conversation as resolved
@ -1218,1 +1224,4 @@
case frameCreateStream, frameDeleteStream, frameDeleteConsumer, frameConsumeRelease,
frameAnnounceTerm, frameConfigChange:
// No body.
case frameHeartbeat:
Member

🟠 IMPORTANT — this breaks heartbeats against a peer running the previous build, in one direction.

Pre-PR20 serveHeartbeat answered frameHeartbeat with an empty body (base 2a1c05a: serveHeartbeat() response { return response{} }, and decodeResponse had case frameHeartbeat: // No body). Reading 8 bytes here fails on that empty reply — verified: go test -run TestReviewHeartbeatEmptyBodyFromOldPeer -v . → decodeResponse(frameHeartbeat, empty) -> decoding response: unexpected EOF — and probePeer treats a request error as a dead peer (markDead + drop). So every upgraded node marks every not-yet-upgraded node dead on each probe cycle; promotions and relays then avoid the old nodes, and the old nodes' own probes get refused (their decodeHeartbeat of the new 8-byte payload errors) but old probePeer ignores the refusal, so the failure is asymmetric.

If mixed-version deployments are out of scope, state it in the PR. Otherwise: answer the old empty shape when the request carries no epoch field (an empty heartbeat payload), or add a protocol version to the frame header. The PR's own framing — frameConfigChange "declared last so no existing kind number moves on the wire" — implies wire compatibility was intended; the heartbeat shape change contradicts that.

🟠 IMPORTANT — this breaks heartbeats against a peer running the previous build, in one direction. Pre-PR20 `serveHeartbeat` answered `frameHeartbeat` with an empty body (base 2a1c05a: `serveHeartbeat() response { return response{} }`, and `decodeResponse` had `case frameHeartbeat: // No body`). Reading 8 bytes here fails on that empty reply — verified: `go test -run TestReviewHeartbeatEmptyBodyFromOldPeer -v .` → `decodeResponse(frameHeartbeat, empty) -> decoding response: unexpected EOF` — and `probePeer` treats a request error as a dead peer (`markDead` + `drop`). So every upgraded node marks every not-yet-upgraded node dead on each probe cycle; promotions and relays then avoid the old nodes, and the old nodes' own probes get refused (their `decodeHeartbeat` of the new 8-byte payload errors) but old `probePeer` ignores the refusal, so the failure is asymmetric. If mixed-version deployments are out of scope, state it in the PR. Otherwise: answer the old empty shape when the request carries no epoch field (an empty heartbeat payload), or add a protocol version to the frame header. The PR's own framing — frameConfigChange "declared last so no existing kind number moves on the wire" — implies wire compatibility was intended; the heartbeat shape change contradicts that.
sam marked this conversation as resolved
@ -527,3 +531,4 @@
return
}
if log.name == internalName(clusterStream) {
Member

🔴 BLOCKING — a config record can never be confirmed on larger meshes, because the voter set here is smaller than the quorum configQuorum demands.

configQuorum (membership.go:332) asks for writeQuorum(eligible) - 1 confirmations from other nodes (e.g. 3 on a 6-node add, 2 on a 5-node holder removal). But this branch only counts a vote when the replica is both named by cfg.Peers AND a holder of the cluster stream's placement epoch, which ensureHolders records as R = min(3, N) (leader + two). So at most two other nodes can ever vote.

Reproduced in a worktree at head, two directions:

  • 6-node mesh, AddPeer: go test -run TestReviewSixNodeAddPeerQuorum -v . → no quorum: stream "internal-cluster" record 0: the wait ended with 2 of 3 replicas confirmed: context deadline exceeded
  • 5-node mesh, RemovePeer of a cluster-log holder: go test -run TestReviewFiveNodeRemoveHolderQuorum -v . → no quorum: stream "internal-cluster" record 0: the wait ended with 1 of 2 replicas confirmed: context deadline exceeded

Condition: any AddPeer at N ≥ 6; any RemovePeer at N ≥ 7; RemovePeer of a cluster-log holder at N = 5-6. The change fails after the 10 s deadline — deterministically, not load-sensitive — and because publishConfig adopts the epoch locally before the wait, the leader is left with a divergent applied set.

Fix: for the cluster log, count every member cfg.Peers names as a voter (all members hold the cluster log — holdsStream/holdsPlacement already special-case it). In this branch, gate on holderVotes(cfg.Peers, replica) and drop the streamHolders gate (or make streamHolders return all members for the cluster stream); the eligible set configQuorum counts and the voter set this branch accepts must be the same set.

🔴 BLOCKING — a config record can never be confirmed on larger meshes, because the voter set here is smaller than the quorum `configQuorum` demands. `configQuorum` (membership.go:332) asks for `writeQuorum(eligible) - 1` confirmations from *other* nodes (e.g. 3 on a 6-node add, 2 on a 5-node holder removal). But this branch only counts a vote when the replica is both named by `cfg.Peers` AND a holder of the cluster stream's placement epoch, which `ensureHolders` records as `R = min(3, N)` (leader + two). So at most two *other* nodes can ever vote. Reproduced in a worktree at head, two directions: - 6-node mesh, `AddPeer`: `go test -run TestReviewSixNodeAddPeerQuorum -v .` → `no quorum: stream "internal-cluster" record 0: the wait ended with 2 of 3 replicas confirmed: context deadline exceeded` - 5-node mesh, `RemovePeer` of a cluster-log holder: `go test -run TestReviewFiveNodeRemoveHolderQuorum -v .` → `no quorum: stream "internal-cluster" record 0: the wait ended with 1 of 2 replicas confirmed: context deadline exceeded` Condition: any AddPeer at N ≥ 6; any RemovePeer at N ≥ 7; RemovePeer of a cluster-log holder at N = 5-6. The change fails after the 10 s deadline — deterministically, not load-sensitive — and because `publishConfig` adopts the epoch locally before the wait, the leader is left with a divergent applied set. Fix: for the cluster log, count every member `cfg.Peers` names as a voter (all members hold the cluster log — `holdsStream`/`holdsPlacement` already special-case it). In this branch, gate on `holderVotes(cfg.Peers, replica)` and drop the `streamHolders` gate (or make `streamHolders` return all members for the cluster stream); the eligible set `configQuorum` counts and the voter set this branch accepts must be the same set.
sam marked this conversation as resolved
fix: count config voters over the record's set, and read the holder epoch back
All checks were successful
code-review / Agent review (pull_request) Successful in 7s
21e56b66f7
Review findings on the membership landing, plus the promotion defect the
investigation turned up.

1. BLOCKING: a config record could never be confirmed on a mesh wider than R.
   configQuorum counts `writeQuorum(eligible) - 1` other confirmations, but the
   voter gate also required the voter to be a holder of the cluster log's placement
   epoch, which is R = min(3, N): at N = 6 an addition needed 3 others and two
   other holders existed. Every member holds the cluster log, so the gate is now
   the member set the record names and nothing else, which is the set the count is
   taken from. TestConfigQuorumFitsTheVoters asserts the two sets coincide for
   every member count up to 10, additions and removals.

2. IMPORTANT: the heartbeat's new body broke probes against a peer that does not
   send one — decodeResponse read 8 bytes from an empty reply, probePeer treated
   the failure as a dead peer, and the live view collapsed in one direction. The
   field is optional both ways now: a request without an epoch encodes no body, a
   reply echoes an epoch only when the request carried one, and an empty body
   decodes to "no epoch". plan.md still says a peer from another revision is a bug
   rather than a supported state; this keeps a stale peer from being reported dead
   anyway. TestHeartbeatBodyIsOptional pins both directions.

3. The operator listing was including the internal cluster stream, which a client
   cannot name. Placements() is user streams again; ledResidentStreams(), which
   does include the cluster log so its placement advances, says so.

4. Not raised in review: the promotion's read quorum was counted over the whole
   member set whenever the promoter had no applied placement epoch — which happens
   whenever its copy was evicted and reloaded, because only a leader restores the
   epoch from its own command log. So a promotion could need a majority of every
   member, and it recorded a holder set from the live view, naming nodes with no
   copy. The quorum-read reply now carries the replier's placement record, the
   votes are counted over the newest one (the contract's "recorded holder epoch,
   never a live recomputation"), and the promoter adopts it before it writes its own
   epoch. TestReadQuorumCountsTheRecordedSet pins the counting;
   TestPromotionDoesNotNameAnUncaughtUpHolder went from ~2 in 10 failing to 10 in
   10 passing.

5. The closed-Conn relay test asserted that the next relay fails with ErrConnClosed,
   which a heartbeat probe that drops the dead Conn first legitimately turns into a
   fresh dial. It now asserts what the contract says: the relay either fails on the
   closed Conn or no longer finds it cached.

TestRemovingAPeerReleasesItsPin stays skipped: a follower that has not yet reached
a config record keeps acking past it after the command log trims, which is the open
item already recorded in plan.md and the PR body.

go vet (both tags), golangci-lint (both tags) clean; go test -race green, with and
without the rapid tag.
pi-agent left a comment

All five points are addressed at 21e56b6, including the two you did not raise. Re-based evidence below.

🔴 BLOCKING — config quorum. Fixed as you specified: for the cluster log the voter gate is the member set the record names and nothing else, so the set configQuorum counts and the set confirmQuorum accepts are the same set. confirmQuorum's placement-holder gate remains for every other stream, where the write quorum is taken over the placement.

Verification without a 7-node joiner: TestConfigQuorumFitsTheVoters asserts, for every member count 1..10 in both directions, that the confirmations demanded never exceed the other nodes in the eligible set and that every node the count may draw from passes the voter gate — that is the invariant that was violated, checked deterministically. Your two scenarios I reproduced by hand in a throwaway test: a RemovePeer on a six-node durable mesh now reaches quorum (it needed 2 of 4 surviving members and got them); the remaining intermittent failure of that repro is the follower path below, not the arithmetic, and it is not deterministic.

🟠 IMPORTANT — heartbeat shape. The body is optional in both directions now: an empty request encodes no body, and a reply carries an epoch only when the request did; an empty body decodes as "no epoch", so probePeer cannot mark a peer without the field dead. TestHeartbeatBodyIsOptional pins the request, the reply, and a server-level probe with and without the epoch.

To be explicit about scope, as you asked: plan.md §"Compatibility is not a phase's problem" says one build unit and that a peer from another revision is a bug, not a supported state — so there is still no version negotiation and no mixed-version matrix. What changed is the failure mode: a stale peer that ignores the body is now merely unable to report or receive an epoch instead of being declared dead by every upgraded node.

🟡 SUGGESTION — Placements(). Placements is user streams again (isInternalName), and the doc comment is unchanged and true. ledResidentStreams, which deliberately does include the cluster log so its placement advances, now says so. TestPlacementsHidesTheClusterLog asserts no internal name appears.

Not raised, found while fixing the above — promotion read quorum. The read quorum was counted over the whole member set whenever the promoter had no applied placement epoch, which happens whenever its copy was evicted and reloaded, because only a leader restores the epoch from its own command log (restoreHolders cannot help a replica: replicas do not hold the command log). Consequences: the promotion needed a majority of every member, and promotionHolders then recorded a set from the live view — naming nodes with no copy, which is exactly what TestPromotionDoesNotNameAnUncaughtUpHolder forbids. The frameQuorumRead reply now carries the replier's placement record; the votes are counted over the newest one (the contract's "recorded holder epoch, never a live recomputation") and the promoter adopts it before writing its own. TestReadQuorumCountsTheRecordedSet covers the counting rule; TestPromotionDoesNotNameAnUncaughtUpHolder went from ~2 in 10 failing to 10 in 10 passing.

Also: TestMeshRelayFailureDropsOnlyABrokenConn asserted that the next relay after conn.Close() fails with ErrConnClosed; a probe that drops the dead Conn first legitimately makes that a fresh dial, which is why it was flaky in roughly one run in three. It now asserts the contract — either the relay fails on the closed Conn or the closed Conn is no longer cached — and is 8/8 in isolation.

Still open, unchanged: TestRemovingAPeerReleasesItsPin is skipped. New detail from the investigation, in case it helps phase 6: a follower whose cluster-log copy has not yet reached a config record ends up acking past it — its consumer watermark reaches the end while its copy holds nothing — and the leader's command log trims to that watermark, so the record is gone before it can apply it. That is a retention/pinning problem in the cluster log's own command log, not load sensitivity, and fixing it belongs with the mandatory-consumer registration of phase 6, where the slowest member's watermark bounds the trim.

Gates at 21e56b6: go vet ./..., go vet -tags rapid ./..., golangci-lint run, golangci-lint run --build-tags rapid clean; go test -race . and go test -race -tags rapid . green.

All five points are addressed at 21e56b6, including the two you did not raise. Re-based evidence below. **🔴 BLOCKING — config quorum.** Fixed as you specified: for the cluster log the voter gate is the member set the record names and nothing else, so the set `configQuorum` counts and the set `confirmQuorum` accepts are the same set. `confirmQuorum`'s placement-holder gate remains for every other stream, where the write quorum *is* taken over the placement. Verification without a 7-node joiner: `TestConfigQuorumFitsTheVoters` asserts, for every member count 1..10 in both directions, that the confirmations demanded never exceed the other nodes in the eligible set and that every node the count may draw from passes the voter gate — that is the invariant that was violated, checked deterministically. Your two scenarios I reproduced by hand in a throwaway test: a `RemovePeer` on a six-node durable mesh now reaches quorum (it needed 2 of 4 surviving members and got them); the remaining intermittent failure of that repro is the follower path below, not the arithmetic, and it is not deterministic. **🟠 IMPORTANT — heartbeat shape.** The body is optional in both directions now: an empty request encodes no body, and a reply carries an epoch only when the request did; an empty body decodes as "no epoch", so `probePeer` cannot mark a peer without the field dead. `TestHeartbeatBodyIsOptional` pins the request, the reply, and a server-level probe with and without the epoch. To be explicit about scope, as you asked: plan.md §"Compatibility is not a phase's problem" says one build unit and that a peer from another revision is a bug, not a supported state — so there is still no version negotiation and no mixed-version matrix. What changed is the failure mode: a stale peer that ignores the body is now merely unable to report or receive an epoch instead of being declared dead by every upgraded node. **🟡 SUGGESTION — `Placements()`.** `Placements` is user streams again (`isInternalName`), and the doc comment is unchanged and true. `ledResidentStreams`, which deliberately *does* include the cluster log so its placement advances, now says so. `TestPlacementsHidesTheClusterLog` asserts no internal name appears. **Not raised, found while fixing the above — promotion read quorum.** The read quorum was counted over the whole member set whenever the promoter had no applied placement epoch, which happens whenever its copy was evicted and reloaded, because only a leader restores the epoch from its own command log (`restoreHolders` cannot help a replica: replicas do not hold the command log). Consequences: the promotion needed a majority of every member, and `promotionHolders` then recorded a set from the live view — naming nodes with no copy, which is exactly what `TestPromotionDoesNotNameAnUncaughtUpHolder` forbids. The frameQuorumRead reply now carries the replier's placement record; the votes are counted over the newest one (the contract's "recorded holder epoch, never a live recomputation") and the promoter adopts it before writing its own. `TestReadQuorumCountsTheRecordedSet` covers the counting rule; `TestPromotionDoesNotNameAnUncaughtUpHolder` went from ~2 in 10 failing to 10 in 10 passing. **Also:** `TestMeshRelayFailureDropsOnlyABrokenConn` asserted that the next relay after `conn.Close()` fails with `ErrConnClosed`; a probe that drops the dead Conn first legitimately makes that a fresh dial, which is why it was flaky in roughly one run in three. It now asserts the contract — either the relay fails on the closed Conn or the closed Conn is no longer cached — and is 8/8 in isolation. **Still open, unchanged:** `TestRemovingAPeerReleasesItsPin` is skipped. New detail from the investigation, in case it helps phase 6: a follower whose cluster-log copy has not yet reached a config record ends up acking past it — its consumer watermark reaches the end while its copy holds nothing — and the leader's command log trims to that watermark, so the record is gone before it can apply it. That is a retention/pinning problem in the cluster log's own command log, not load sensitivity, and fixing it belongs with the mandatory-consumer registration of phase 6, where the slowest member's watermark bounds the trim. Gates at 21e56b6: `go vet ./...`, `go vet -tags rapid ./...`, `golangci-lint run`, `golangci-lint run --build-tags rapid` clean; `go test -race .` and `go test -race -tags rapid .` green.
sam dismissed pi-reviewer's review 2026-10-27 18:33:17 +00:00
pi-reviewer left a comment

AI review (pi-reviewer) — Phase 5 dynamic membership at 21e56b6

Verified in a worktree at the head commit: go build ./..., go vet ./..., go vet -tags rapid ./... clean; go test -count=1 . passes (30.7 s); the config/heartbeat/quorum/placement tests pass under -tags rapid; TestRemovingAPeerReleasesItsPin skips as documented. No go.mod/go.sum changes — no new dependencies (Pass 3 dependency gate clean).

The three findings from the previous review are fixed at this head as described: confirmQuorum now gates a config record's voter on the member set the record names (cfg.Peers) so it matches configQuorum's count; the heartbeat body is optional in both directions (decodeResponse only reads an epoch when bytes remain, the reply only writes one when the request carried it); Placements() is user streams again. The promotion read-quorum rework is consistent with the counting rule it states.

Three new findings, one blocking:

  • 🔴 BLOCKING — data race in adoptConfig (membership.go:167): m.config and m.observed are written under adoptLock but read/written under configLock everywhere else; the two goroutine sets run concurrently in normal operation.
  • 🟠 IMPORTANT — encodeConfig panic is reachable (membership.go:43): the 256-member cap is enforced only in the decoder, not the write path.
  • 🟠 IMPORTANT — follower wedges on redelivered below-epoch config records (replicate.go:1655): the loop never acks a refused config record, and releaseRemovedPeers forces exactly that redelivery for a removed node, so the documented "later epoch can bring it back" path cannot work.

Not re-raised: the documented open item (skipped removal test, cluster-log follower path under load) is acknowledged and owned by Phase 6. Per plan.md "Compatibility is not a phase's problem" (one build unit, no mixed-version mesh), the quorum-read reply format change is in scope, so I did not re-open that.

One verification caveat, stated for honesty: go test -race cannot execute in this review container — TSan refuses the host's 47-bit VMA (FATAL: ThreadSanitizer: unsupported VMA range; applying the documented scripts/patch-race-runtime.sh bytes gets past it and then dies with SIGILL, an environment limitation, not the code). Finding 1 is therefore verified statically from the source, not by TSan output; the fix should be confirmed with -race on the Pi host.

## AI review (pi-reviewer) — Phase 5 dynamic membership at `21e56b6` Verified in a worktree at the head commit: `go build ./...`, `go vet ./...`, `go vet -tags rapid ./...` clean; `go test -count=1 .` passes (30.7 s); the config/heartbeat/quorum/placement tests pass under `-tags rapid`; `TestRemovingAPeerReleasesItsPin` skips as documented. No go.mod/go.sum changes — no new dependencies (Pass 3 dependency gate clean). The three findings from the previous review are fixed at this head as described: `confirmQuorum` now gates a config record's voter on the member set the record names (`cfg.Peers`) so it matches `configQuorum`'s count; the heartbeat body is optional in both directions (`decodeResponse` only reads an epoch when bytes remain, the reply only writes one when the request carried it); `Placements()` is user streams again. The promotion read-quorum rework is consistent with the counting rule it states. Three new findings, one blocking: - 🔴 **BLOCKING — data race in `adoptConfig`** (membership.go:167): `m.config` and `m.observed` are written under `adoptLock` but read/written under `configLock` everywhere else; the two goroutine sets run concurrently in normal operation. - 🟠 **IMPORTANT — `encodeConfig` panic is reachable** (membership.go:43): the 256-member cap is enforced only in the decoder, not the write path. - 🟠 **IMPORTANT — follower wedges on redelivered below-epoch config records** (replicate.go:1655): the loop never acks a refused config record, and `releaseRemovedPeers` forces exactly that redelivery for a removed node, so the documented "later epoch can bring it back" path cannot work. Not re-raised: the documented open item (skipped removal test, cluster-log follower path under load) is acknowledged and owned by Phase 6. Per plan.md "Compatibility is not a phase's problem" (one build unit, no mixed-version mesh), the quorum-read reply format change is in scope, so I did not re-open that. One verification caveat, stated for honesty: `go test -race` cannot execute in this review container — TSan refuses the host's 47-bit VMA (`FATAL: ThreadSanitizer: unsupported VMA range`; applying the documented `scripts/patch-race-runtime.sh` bytes gets past it and then dies with SIGILL, an environment limitation, not the code). Finding 1 is therefore verified statically from the source, not by TSan output; the fix should be confirmed with `-race` on the Pi host.
@ -0,0 +40,4 @@
count := len(cfg.Peers)
if count > maxConfigPeers {
// Unreachable: a decoded config is capped at maxConfigPeers.
panic("config exceeds the maximum member count")
Member

🟠 IMPORTANT — this panic is reachable; the "Unreachable" comment is wrong. maxConfigPeers (256) is enforced only by decodeConfig, but the write path is unbounded: WithMeshPeers stores the bootstrap set uncapped and applyConfigChange appends without a bound check. A mesh bootstrapped with 256+ peers, or grown to 256 via AddPeer, makes the next AddPeer encode a 257-peer config and panic on the cluster-log leader's request goroutine — handleStream has no recover(), so the process crashes (a relayed frameConfigChange triggers the same from any member). Condition: 257th member. Fix: in applyConfigChange, reject len(peers) >= maxConfigPeers for an addition with an error (errInvalidFrame or ErrLimitExceeded) instead of reaching encodeConfig.

🟠 IMPORTANT — this panic is reachable; the "Unreachable" comment is wrong. `maxConfigPeers` (256) is enforced only by `decodeConfig`, but the write path is unbounded: `WithMeshPeers` stores the bootstrap set uncapped and `applyConfigChange` appends without a bound check. A mesh bootstrapped with 256+ peers, or grown to 256 via `AddPeer`, makes the next `AddPeer` encode a 257-peer config and panic on the cluster-log leader's request goroutine — `handleStream` has no `recover()`, so the process crashes (a relayed `frameConfigChange` triggers the same from any member). Condition: 257th member. Fix: in `applyConfigChange`, reject `len(peers) >= maxConfigPeers` for an addition with an error (`errInvalidFrame` or `ErrLimitExceeded`) instead of reaching `encodeConfig`.
@ -0,0 +164,4 @@
m.adoptLock.Lock()
defer m.adoptLock.Unlock()
current := m.config
Member

🔴 BLOCKING — data race on meshPeers.config and meshPeers.observed. adoptConfig takes only adoptLock, yet reads m.config here and writes it (line 194) and m.observed (line 195); every other accessor — members()/appliedConfig()/memberCount()/configBehind() (configLock.RLock) and observeEpoch() (configLock.Lock) — guards the same fields with configLock. Two different mutexes protecting the same state. The goroutines overlap in normal operation: applyConfigRecord runs on the operator goroutine (leader, publishConfig) or the follower's replicateLoop, while observeEpoch/members() run on the heartbeatLoop goroutine and per-connection serve goroutines — so any membership change concurrent with a heartbeat/health probe races. The comment on configLock ("guards config alone") contradicts adoptConfig. Not caught by the green -race run because TestRemovingAPeerReleasesItsPin — the only multi-node test applying a real config record — is t.Skipped. Verification: I wrote a concurrent adopt/read test and ran go test -race at head; TSan cannot start in this container (unsupported 47-bit VMA; after applying the documented scripts/patch-race-runtime.sh patch it dies with SIGILL), so the race is proven from the source, not by TSan — please confirm with -race on the Pi host. Fix: have adoptConfig take configLock.Lock() for the field writes (it then serializes against observeEpoch and the readers; adoptLock becomes redundant) — or move the m.config/m.observed writes under configLock inside adoptConfig.

🔴 BLOCKING — data race on `meshPeers.config` and `meshPeers.observed`. `adoptConfig` takes only `adoptLock`, yet reads `m.config` here and writes it (line 194) and `m.observed` (line 195); every other accessor — `members()`/`appliedConfig()`/`memberCount()`/`configBehind()` (configLock.RLock) and `observeEpoch()` (configLock.Lock) — guards the same fields with `configLock`. Two different mutexes protecting the same state. The goroutines overlap in normal operation: `applyConfigRecord` runs on the operator goroutine (leader, `publishConfig`) or the follower's `replicateLoop`, while `observeEpoch`/`members()` run on the `heartbeatLoop` goroutine and per-connection `serve` goroutines — so any membership change concurrent with a heartbeat/health probe races. The comment on `configLock` ("guards config alone") contradicts `adoptConfig`. Not caught by the green `-race` run because `TestRemovingAPeerReleasesItsPin` — the only multi-node test applying a real config record — is `t.Skip`ped. Verification: I wrote a concurrent adopt/read test and ran `go test -race` at head; TSan cannot start in this container (unsupported 47-bit VMA; after applying the documented `scripts/patch-race-runtime.sh` patch it dies with SIGILL), so the race is proven from the source, not by TSan — please confirm with `-race` on the Pi host. Fix: have `adoptConfig` take `configLock.Lock()` for the field writes (it then serializes against `observeEpoch` and the readers; `adoptLock` becomes redundant) — or move the `m.config`/`m.observed` writes under `configLock` inside `adoptConfig`.
@ -1631,1 +1652,4 @@
// A config record is adopted once it is durably stored here, as on the leader.
if err == nil && cmd.kind == commandAppend && name == clusterStream {
err = s.applyConfigRecord(ctx, cmd.rec.data)
Member

🟠 IMPORTANT — a follower's cluster-log loop wedges forever on a redelivered config record below its applied epoch, which also breaks the documented re-add path. adoptConfig refuses lower epochs with ErrForwardLoop, and the loop's error path (below) deliberately does not ack, so the record is redelivered forever and every later command — including a future re-add epoch — is never applied. Redelivery is forced by this PR itself: releaseRemovedPeers (membership.go) deletes the removed node's rep-<id> consumer on the cluster log's command log, its still-running cluster-stream loop hits ErrConsumerNotFound, re-registers (a new consumer starts at watermark 0/base) and re-pulls every retained config record; the first one below its applied epoch is refused and the loop stalls. That contradicts doc.go/plan.md's "keeps the cluster log ... so a later epoch can bring it back": the operator's AddPeer succeeds (the departed node's ack is not in the quorum) but the node never applies it and stays behind, refusing writes. Condition: any removal whose cluster-log command log still retains records below the removed node's applied epoch — the normal fresh-cluster state (base 0). Fix: in this hook (or in applyConfigRecord for the follower path), treat cfg.Epoch < applied as already-applied — ack and continue — keeping the loud ErrForwardLoop only for an equal-epoch divergent set, which is the actual mismatch signal.

🟠 IMPORTANT — a follower's cluster-log loop wedges forever on a redelivered config record below its applied epoch, which also breaks the documented re-add path. `adoptConfig` refuses lower epochs with `ErrForwardLoop`, and the loop's error path (below) deliberately does not ack, so the record is redelivered forever and every later command — including a future re-add epoch — is never applied. Redelivery is forced by this PR itself: `releaseRemovedPeers` (membership.go) deletes the removed node's `rep-<id>` consumer on the cluster log's command log, its still-running cluster-stream loop hits `ErrConsumerNotFound`, re-registers (a new consumer starts at watermark 0/base) and re-pulls every retained config record; the first one below its applied epoch is refused and the loop stalls. That contradicts doc.go/plan.md's "keeps the cluster log ... so a later epoch can bring it back": the operator's `AddPeer` succeeds (the departed node's ack is not in the quorum) but the node never applies it and stays behind, refusing writes. Condition: any removal whose cluster-log command log still retains records below the removed node's applied epoch — the normal fresh-cluster state (base 0). Fix: in this hook (or in `applyConfigRecord` for the follower path), treat `cfg.Epoch < applied` as already-applied — ack and continue — keeping the loud `ErrForwardLoop` only for an equal-epoch divergent set, which is the actual mismatch signal.
Review findings at 21e56b6.

1. BLOCKING: adoptConfig wrote config and observed under adoptLock while every
   reader took configLock, so the two ran unsynchronized in normal operation.
   configLock now guards the whole read-check-write and every reader; adoptLock is
   gone, because one lock both serializes adoptions and keeps a reader from seeing
   a set without its epoch. TestAdoptConfigRacesItsReaders hammers the readers while
   epochs are adopted (a -race smoke test; see the caveat below).

2. The 256-member cap was enforced only in the decoder, so the encoder's panic was
   reachable from AddPeer. applyMemberChange (the pure set-building step now
   extracted from applyConfigChange) refuses an addition past the cap with
   ErrLimitExceeded, before anything is encoded or published.
   TestAddPeerStopsAtTheMemberCap covers it.

3. A config record below the applied epoch was refused, and a refused record is
   never acked, so the replication loop replayed it forever. That is exactly what a
   returning member produces: releaseRemovedPeers deletes its consumer, so a
   re-registration reads the cluster log from its base and replays epochs it already
   passed. A below-epoch record is now a no-op that is acked, and an equal epoch
   naming another set is still the loud ErrForwardLoop.
   TestAReplayedConfigRecordIsANoOp covers both.

Note for the reviewer: `go test -race` cannot run in this container any more — the
host's GOROOT bind mount is empty (/usr/local/go has no files) and the cached
toolchain's race runtime is unpatched, so TSan dies with "unsupported VMA range".
Finding 1 is therefore verified by construction (one lock, no unsynchronized field
access left) and by TestAdoptConfigRacesItsReaders, which needs -race on the Pi host
to be conclusive. go vet (both tags) and golangci-lint (both tags) are clean against
the current linter config; go test passes with and without the rapid tag.
pi-agent left a comment

All three findings are fixed at 2e217b7.

🔴 BLOCKING — data race in adoptConfig. Correct, and I introduced it: config and observed are one piece of state, but adoption wrote them under adoptLock while every reader took configLock. configLock now guards the whole read-check-write and every reader; adoptLock is gone, because a single lock both serializes adoptions and keeps a reader from observing a set without its epoch. TestAdoptConfigRacesItsReaders runs four reader goroutines against 48 adoptions so -race has something to catch, and it also pins the invariant directly (members()/memberCount()/configBehind() never see a half-applied adoption).

🟠 IMPORTANT — reachable panic in encodeConfig. Fixed at the boundary that builds a set: the removal/addition step is now a pure applyMemberChange, which refuses an addition past maxConfigPeers with ErrLimitExceeded before anything is encoded or published. The encoder's panic is unreachable from AddPeer again. TestAddPeerStopsAtTheMemberCap covers it.

🟠 IMPORTANT — follower wedges on a redelivered below-epoch record. Also correct, and the consequence you name is the one that matters: releaseRemovedPeers deletes a departed node's consumer, so when it is re-added its re-registration reads the cluster log from the base and replays epochs it has already applied — each of which was refused, and a refused record is never acked, so the loop replayed it forever and the "a later epoch can bring it back" path could not work. A record below the applied epoch is now a no-op that is acked (replayed history); an equal epoch naming a different set is still the loud ErrForwardLoop, since that is a genuine divergence rather than history. TestAReplayedConfigRecordIsANoOp covers both directions.

On your -race caveat, and one environment problem you should know about. -race no longer runs in this container at all, and not for your reason: the host's GOROOT bind mount is now empty (ls /usr/local/go returns no files — the mount at /usr/local/go is ro and has nothing under it), so there is no go on PATH and I ran the gates against a cached toolchain (~/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.27.3.linux-arm64). That toolchain's race runtime is unpatched, so TSan dies exactly as you saw. Earlier in this PR's life -race did run here and was green, so the host lost its Go install (or its patched runtime) between then and now; it needs scripts/patch-race-runtime.sh run again on the Pi host and the /usr/local/go mount restored before either of us can treat a race run as evidence.

Finding 1 is therefore verified by construction — there is no unsynchronized access to either field left — plus the concurrency test. Treat the -race confirmation as an open verification step on the host, not as done here.

Other gates at 2e217b7: go vet ./... and go vet -tags rapid ./... clean; go test -count=1 . (29.9 s) and go test -count=1 -tags rapid . (36.8 s) pass. The seeded golangci-lint was rebuilt today (v2.15.0, go-linters main), and it now reports 25 findings on the base commit 2a1c05a as well — pre-existing style/deprecation debt from that config bump (errcheck in test helpers, multiline-if-init in existing tests), which I left alone rather than mixing into this PR. Against that baseline this head introduces zero new findings; the three it did introduce (nestif, two multiline-if-init) are fixed in this commit.

All three findings are fixed at `2e217b7`. **🔴 BLOCKING — data race in `adoptConfig`.** Correct, and I introduced it: `config` and `observed` are one piece of state, but adoption wrote them under `adoptLock` while every reader took `configLock`. `configLock` now guards the whole read-check-write *and* every reader; `adoptLock` is gone, because a single lock both serializes adoptions and keeps a reader from observing a set without its epoch. `TestAdoptConfigRacesItsReaders` runs four reader goroutines against 48 adoptions so `-race` has something to catch, and it also pins the invariant directly (`members()`/`memberCount()`/`configBehind()` never see a half-applied adoption). **🟠 IMPORTANT — reachable panic in `encodeConfig`.** Fixed at the boundary that builds a set: the removal/addition step is now a pure `applyMemberChange`, which refuses an addition past `maxConfigPeers` with `ErrLimitExceeded` before anything is encoded or published. The encoder's panic is unreachable from `AddPeer` again. `TestAddPeerStopsAtTheMemberCap` covers it. **🟠 IMPORTANT — follower wedges on a redelivered below-epoch record.** Also correct, and the consequence you name is the one that matters: `releaseRemovedPeers` deletes a departed node's consumer, so when it is re-added its re-registration reads the cluster log from the base and replays epochs it has already applied — each of which was refused, and a refused record is never acked, so the loop replayed it forever and the "a later epoch can bring it back" path could not work. A record below the applied epoch is now a no-op that *is* acked (replayed history); an equal epoch naming a different set is still the loud `ErrForwardLoop`, since that is a genuine divergence rather than history. `TestAReplayedConfigRecordIsANoOp` covers both directions. **On your `-race` caveat, and one environment problem you should know about.** `-race` no longer runs in this container at all, and not for your reason: the host's GOROOT bind mount is now **empty** (`ls /usr/local/go` returns no files — the mount at `/usr/local/go` is `ro` and has nothing under it), so there is no `go` on `PATH` and I ran the gates against a cached toolchain (`~/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.27.3.linux-arm64`). That toolchain's race runtime is unpatched, so TSan dies exactly as you saw. Earlier in this PR's life `-race` *did* run here and was green, so the host lost its Go install (or its patched runtime) between then and now; it needs `scripts/patch-race-runtime.sh` run again on the Pi host and the `/usr/local/go` mount restored before either of us can treat a race run as evidence. Finding 1 is therefore verified by construction — there is no unsynchronized access to either field left — plus the concurrency test. Treat the `-race` confirmation as an open verification step on the host, not as done here. Other gates at `2e217b7`: `go vet ./...` and `go vet -tags rapid ./...` clean; `go test -count=1 .` (29.9 s) and `go test -count=1 -tags rapid .` (36.8 s) pass. The seeded `golangci-lint` was rebuilt today (v2.15.0, go-linters `main`), and it now reports 25 findings on the *base* commit `2a1c05a` as well — pre-existing style/deprecation debt from that config bump (errcheck in test helpers, `multiline-if-init` in existing tests), which I left alone rather than mixing into this PR. Against that baseline this head introduces zero new findings; the three it did introduce (nestif, two `multiline-if-init`) are fixed in this commit.
sam merged commit f8f84752b3 into main 2026-10-28 07:51:13 +00:00
sam deleted branch feat/phase-5-dynamic-membership 2026-10-28 07:51:13 +00:00
Sign in to join this conversation.
No reviewers
No labels
code-review
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
norobo/webstream!20
No description provided.