feat: replicate stream metadata on the cluster log (plan.md Phase 6) #22

Merged
sam merged 11 commits from feat/cluster-metadata-replication into main 2026-11-05 07:56:49 +00:00
Member

Implements Phase 6 of plan.md: polled reconciliation stops being the source of truth for which streams exist.

What the phase does

  • The cluster log internal-cluster carries a tagged record union: Config{epoch, peers} (Phase 5), StreamMeta{name, maxAge, holders} and Tombstone{name}.
  • Create and delete are cluster-log commands. The node the client reaches records the command itself (createClusteredStream / deleteClusteredStream), relays the record to the log's writer when it is not the writer, and applies it locally once the relay returns, so a client that creates and then lists or writes on the same node sees the stream.
  • The record is acknowledged by a majority of the applied member set (memberQuorum), the same majority a config change uses.
  • Every member consumes the log as meta-<peer ID>; a user stream's replica stays rep-<peer ID>. Voter parsing accepts both prefixes, so the leader can count a metadata ack.
  • ListStreams, Server.Streams and getStream answer from the applied metadata plus local copies, so a node that holds no copy still lists and resolves the stream.
  • A restart replays its own copy of the log once (restoreClusterState), which reproduces the config epoch, the stream set, the holder sets and the tombstones.
  • Polled discovery is behind WithPolledReconciliation(), default off, and its tests are re-pointed: the paging case is deleted (replaced by "a stream created while a node was down is present when it returns, through the log"), the out-of-band case runs on a polling mesh as the safety net, and the cases that stop a replica now seed createStreamLocal, which is the safety-net premise.

Defects found and fixed on the way

Each is a mechanism, not a timing knob.

  1. forwardTo fell back to the live view. Two nodes whose peer leases disagreed each named themselves writer of the cluster log — the one stream whose records decide the member set — so a create on one side waited 30 s for a confirmation the other side could never send. The fallback is now the set placement derives from, never the live view alone.
  2. A replica registered its consumer over the shared mesh Conn, which can still be holding the create relay whose quorum that ack completes, so the registration waited on the ack it was blocking. The loop registers over its own connection (registerConsumerOn), on the first pull that finds no consumer.
  3. restoreClusterState only replayed a resident cluster log, so a restarted durable node replayed nothing and came back with an empty stream set and epoch 0. It now acquires the log, cold or resident.
  4. reconcileLoadedStream (and the consumer-emit guards and markEvicted) tested the internal- prefix where they meant "is a command log". internal-cluster is internal and a replicated stream, so pinFloor decoded its records as commands (visible as a command log holds a command this build cannot decode … trailing bytes 79) and the cluster log skipped the leader's catch-up path on every reload.
  5. A reloaded stream state started with no placement epoch, so the next pass computed a fresh holder set and published it — re-placing a stream that lost no holder, and dropping any plan the pass held for it (reconcileLoadedStream now reads the epoch back).
  6. A parked placement plan held the pass. advancePlacement returned true for a plan that had already parked, and every pass pushed its deadline forward, so a plan that could not converge consumed the pass for ever — the exact churn bound Phase 4 promises. A parked plan now yields.
  7. A replica refused replayed history. A copy resuming at the leader's retained base met commands written before it learned its term, treated them as a superseded writer's writes, and stopped for good, which left every publish on that stream waiting out its quorum. A below-term Append whose record the copy already holds is history: acked, replay continues; a record it does not hold stays the divergence it was.
  8. A command log re-created above term 1 opened at term 1. It now opens with a Term command at the stream's current term, so a replica that applied the higher term can read it from its base.
  9. deleteLocalStream published a tombstone. Releasing a displaced copy would have deleted the stream cluster-wide; the local delete no longer records a command.
  10. The cluster log's applier did peer calls inline, which queue behind the relay whose quorum it is about to complete.

Verification

  • go vet ./... clean; golangci-lint run 0 issues; go vet -tags rapid and golangci-lint run --build-tags rapid clean.
  • go test ./... green (82.8 s on this box), go test -short ./... green (~10 s).
  • FuzzDecodeClusterRecord with seeds (round trip, 27k execs in 5 s).
  • TestClusterMetaModelKeepsTheAppliedSet: rapid model over the applied metadata against a shadow map — last record wins, a tombstone beats the copy, a later create clears it, all() is complete and sorted.

Follow-up (not done here)

  • A re-create after a delete and a restart in between does not reach the restarted node. TestATombstoneDoesNotResurrectOnRestart asserts the phase's claims (no resurrection; copy, replication loop and command log gone) and says in a comment where this one is left.
  • make test-fast gained the -short skip in the multi-node and durable harnesses; the two full-suite runs above are the evidence that make test is still the gate.
Implements Phase 6 of `plan.md`: polled reconciliation stops being the source of truth for which streams exist. ## What the phase does - The cluster log `internal-cluster` carries a tagged record union: `Config{epoch, peers}` (Phase 5), `StreamMeta{name, maxAge, holders}` and `Tombstone{name}`. - Create and delete are cluster-log commands. The node the client reaches records the command itself (`createClusteredStream` / `deleteClusteredStream`), relays the *record* to the log's writer when it is not the writer, and applies it locally once the relay returns, so a client that creates and then lists or writes on the same node sees the stream. - The record is acknowledged by a majority of the applied member set (`memberQuorum`), the same majority a config change uses. - Every member consumes the log as `meta-<peer ID>`; a user stream's replica stays `rep-<peer ID>`. Voter parsing accepts both prefixes, so the leader can count a metadata ack. - `ListStreams`, `Server.Streams` and `getStream` answer from the applied metadata plus local copies, so a node that holds no copy still lists and resolves the stream. - A restart replays its own copy of the log once (`restoreClusterState`), which reproduces the config epoch, the stream set, the holder sets and the tombstones. - Polled discovery is behind `WithPolledReconciliation()`, default off, and its tests are re-pointed: the paging case is deleted (replaced by "a stream created while a node was down is present when it returns, through the log"), the out-of-band case runs on a polling mesh as the safety net, and the cases that stop a replica now seed `createStreamLocal`, which is the safety-net premise. ## Defects found and fixed on the way Each is a mechanism, not a timing knob. 1. **`forwardTo` fell back to the live view.** Two nodes whose peer leases disagreed each named themselves writer of the cluster log — the one stream whose records decide the member set — so a create on one side waited 30 s for a confirmation the other side could never send. The fallback is now the set placement derives from, never the live view alone. 2. **A replica registered its consumer over the shared mesh Conn**, which can still be holding the create relay whose quorum that ack completes, so the registration waited on the ack it was blocking. The loop registers over its own connection (`registerConsumerOn`), on the first pull that finds no consumer. 3. **`restoreClusterState` only replayed a resident cluster log**, so a restarted durable node replayed nothing and came back with an empty stream set and epoch 0. It now acquires the log, cold or resident. 4. **`reconcileLoadedStream` (and the consumer-emit guards and `markEvicted`) tested the `internal-` prefix where they meant "is a command log".** `internal-cluster` is internal *and* a replicated stream, so `pinFloor` decoded its records as commands (visible as `a command log holds a command this build cannot decode … trailing bytes 79`) and the cluster log skipped the leader's catch-up path on every reload. 5. **A reloaded stream state started with no placement epoch**, so the next pass computed a fresh holder set and published it — re-placing a stream that lost no holder, and dropping any plan the pass held for it (`reconcileLoadedStream` now reads the epoch back). 6. **A parked placement plan held the pass.** `advancePlacement` returned true for a plan that had already parked, and every pass pushed its deadline forward, so a plan that could not converge consumed the pass for ever — the exact churn bound Phase 4 promises. A parked plan now yields. 7. **A replica refused replayed history.** A copy resuming at the leader's retained base met commands written before it learned its term, treated them as a superseded writer's writes, and stopped for good, which left every publish on that stream waiting out its quorum. A below-term Append whose record the copy already holds is history: acked, replay continues; a record it does not hold stays the divergence it was. 8. **A command log re-created above term 1 opened at term 1.** It now opens with a `Term` command at the stream's current term, so a replica that applied the higher term can read it from its base. 9. **`deleteLocalStream` published a tombstone.** Releasing a displaced copy would have deleted the stream cluster-wide; the local delete no longer records a command. 10. **The cluster log's applier did peer calls inline**, which queue behind the relay whose quorum it is about to complete. ## Verification - `go vet ./...` clean; `golangci-lint run` 0 issues; `go vet -tags rapid` and `golangci-lint run --build-tags rapid` clean. - `go test ./...` green (82.8 s on this box), `go test -short ./...` green (~10 s). - `FuzzDecodeClusterRecord` with seeds (round trip, 27k execs in 5 s). - `TestClusterMetaModelKeepsTheAppliedSet`: rapid model over the applied metadata against a shadow map — last record wins, a tombstone beats the copy, a later create clears it, `all()` is complete and sorted. ## Follow-up (not done here) - A re-create after a delete *and* a restart in between does not reach the restarted node. `TestATombstoneDoesNotResurrectOnRestart` asserts the phase's claims (no resurrection; copy, replication loop and command log gone) and says in a comment where this one is left. - `make test-fast` gained the `-short` skip in the multi-node and durable harnesses; the two full-suite runs above are the evidence that `make test` is still the gate.
A stream's existence is now a cluster-log command rather than something a polled
pass reconciles: create writes one StreamMeta{name, maxAge, holders} and delete one
Tombstone{name}, each acknowledged by a majority of the members and applied in log
order by every one of them. Existence and listing are answered from that applied
state plus local copies, so no peer is paged, and the newest record naming a stream
wins over a tombstone below it.

Every member consumes the log as meta-<peer ID> where a user stream's replica is
rep-<peer ID>, so its retention follows the slowest member; a restarted node replays
its own copy once, which reproduces the config epoch, the stream set and the holder
sets. The polled stream-set discovery stays as a bounded safety net behind
WithPolledReconciliation().

Getting the phase green turned up six defects outside the metadata code itself:

- forwardTo fell back to the live view, so two nodes whose views disagreed each
  named themselves writer of the cluster log -- the one stream whose records decide
  that view. The fallback is now the set placement derives from.
- a replica registered its consumer over the shared mesh Conn, which can still be
  holding the create relay whose quorum that ack completes, so the registration
  waited on the ack it was blocking. The loop registers over its own connection.
- restoreClusterState only replayed a resident cluster log, so a restarted durable
  node replayed nothing and came back with an empty stream set.
- rebuildCommandLog consulted the routing rule, so a concurrent term announcement
  could turn its create into a no-op and leave the log missing.
- deleteLocalStream published a tombstone, so releasing a displaced copy would have
  deleted the stream cluster-wide.
- the cluster log's applier did peer calls inline, which queue behind the relay whose
  quorum it is about to complete.

The polled and durable harnesses now skip under -short, with make test-fast and
make one as the inner loop.

Known red in the full suite: TestAStuckPlacementYieldsThePass and
TestTermDisagreementResolvesToTheHigherTerm. Both are recorded in the pull request
with what is known about their mechanism.
A stream state that came back from disk started with no placement epoch, so the next
pass computed a fresh holder set and published it -- re-placing a stream that lost no
holder, and dropping any plan the pass was holding for it.

Also: the stream-name tables no longer expect a create or a delete to be routed to a
stream leader (they are cluster-log commands now), and the two polled cases that stop a
replica seed their stream with createStreamLocal, which is the safety-net premise.
pi-agent force-pushed feat/cluster-metadata-replication from 915350efdc to 12d526466e 2026-11-01 17:11:25 +00:00 Compare
advancePlacement returned true for a plan that had already parked, so every pass
pushed its deadline forward and a plan that could not converge held the pass for
ever -- the churn bound Phase 4 promises. A parked plan now yields; only a plan
still inside its first window holds the pass.

reconcileLoadedStream (and the two consumer-emit guards and markEvicted) tested the
internal- name prefix where they meant "is a command log". internal-cluster is
internal and a replicated stream, so pinFloor decoded its records as commands and the
cluster log skipped the leader's catch-up path on every reload.
A replica that resumed at the leader's retained base met commands written before it
learned its term and treated them as a superseded writer's writes, so its loop died
and every publish on that stream waited out its quorum. A below-term Append whose
record the copy already holds is history: it is acked, and the replay moves on. A
record the copy does not hold stays the divergence it was.

A command log re-created while the stream was already above the first term now opens
with a Term command stamped at that term, so a replica that applied the higher term
can read the log from its base instead of refusing every command in it.
ensureCommandLog runs on every maintenance pass, so announcing unconditionally
appended a Term command per pass -- an unbounded log and a Term to replay each time.
Author
Member

Status after the restart-path fixes (head 21246a7): go test ./... green (85 s), go test -short ./... green, go vet clean, golangci-lint 0 issues on both tag sets.

One intermittent failure is left, and its evidence is recorded here rather than papered over.

TestTermDisagreementResolvesToTheHigherTerm fails roughly 1 run in 6 in isolation (it passed in the last three full-suite runs). The trace:

  • the restarted replica's pull reports consumer not found: "rep-<id>", then its re-registration reports stream not found: stream "internal-s-1" has no directory;
  • the publish then waits out quorumTimeout (30 s) and fails leader unavailable — the leader appended the record but never got an ack;
  • after the timeout the leader's command log is present again, so something re-created it during the failed publish (appendToCommandLog's ensure fallback), which is what makes the second registration succeed on a later attempt.

So the open question is narrow: why does the stream's command-log directory report "no directory" on the node the replica asks, immediately after the peer it restarts comes back, and re-appear moments later. No deleteInternalStream/deleteStaleCommandLog call is logged in those runs, and budget eviction deletes segments, not stream directories, so the remaining suspects are store recovery at the restarted node's newStore and a request served by a node whose own copy of the log does not exist (the restarted replica has s-<n> but never a command log for it).

Also fixed since the PR was opened, both found while chasing that test:

  • a replica refused replayed history: a copy resuming at the leader's retained base met commands written before it learned its term, treated them as a superseded writer's writes and stopped for good. A below-term Append whose record the copy already holds is now acked and the replay continues; a record it does not hold stays the divergence it was.
  • a command log re-created while the stream was already above term 1 now opens with a Term command at that term, and only when the log is genuinely new (announcing on every pass appended one Term per pass).
Status after the restart-path fixes (head `21246a7`): `go test ./...` green (85 s), `go test -short ./...` green, `go vet` clean, `golangci-lint` 0 issues on both tag sets. One intermittent failure is left, and its evidence is recorded here rather than papered over. `TestTermDisagreementResolvesToTheHigherTerm` fails roughly 1 run in 6 **in isolation** (it passed in the last three full-suite runs). The trace: - the restarted replica's pull reports `consumer not found: "rep-<id>"`, then its re-registration reports `stream not found: stream "internal-s-1" has no directory`; - the publish then waits out `quorumTimeout` (30 s) and fails `leader unavailable` — the leader appended the record but never got an ack; - after the timeout the leader's command log is present again, so something re-created it during the failed publish (`appendToCommandLog`'s ensure fallback), which is what makes the second registration succeed on a later attempt. So the open question is narrow: **why does the stream's command-log directory report "no directory" on the node the replica asks, immediately after the peer it restarts comes back**, and re-appear moments later. No `deleteInternalStream`/`deleteStaleCommandLog` call is logged in those runs, and budget eviction deletes segments, not stream directories, so the remaining suspects are store recovery at the restarted node's `newStore` and a request served by a node whose own copy of the log does not exist (the restarted replica has `s-<n>` but never a command log for it). Also fixed since the PR was opened, both found while chasing that test: - a replica refused **replayed history**: a copy resuming at the leader's retained base met commands written before it learned its term, treated them as a superseded writer's writes and stopped for good. A below-term Append whose record the copy already holds is now acked and the replay continues; a record it does not hold stays the divergence it was. - a command log re-created while the stream was already above term 1 now opens with a `Term` command at that term, and only when the log is genuinely new (announcing on every pass appended one `Term` per pass).
The replica's term rule refused every command carrying a term below the copy's
term and stopped the loop, on the premise that a lower term means the log went
backwards. That premise is false: a copy that resumes at the leader's retained base
replays commands written before it learned its term, because the retained prefix
predates the raise by definition. The refusal killed the loop for good, so every
publish on that stream then waited out its quorum and failed.

Applying those commands is safe by construction: an Append the copy holds is a
repeat, a Trim is a floor, a Holders epoch and a consumer upsert move only forward,
and a Reset is the replay's own primitive. The term still only moves forward, and a
Term announcement below the copy's term is ignored rather than moving routing back to
the leader it names — which is the one command that is not replayable.

This is a contract correction, so the model and the test move with it:
TestReplicaAppliesHistoryBelowItsTerm (was ...RefusesACommandBelowItsTerm) pins that
history is applied, the term does not regress and a below-term announcement does not
move the leader, TestLeadershipStateProperties models the same rule, and doc.go's
failover section states it.

Also: a replica that re-registers its consumer now sleeps one retry interval before
pulling again. Without the pause a leader that keeps dropping the consumer turns the
loop into a spin, and in a synctest bubble a spinning goroutine freezes the clock, so
unrelated multi-node tests hung for minutes.

Full suite: three consecutive runs green (88s, 84s, 87s). coverage.log gains the
untagged number for this phase (86.0%).
Author
Member

Head cdb0912: the intermittent is fixed, and one more restart-path defect fell out of chasing it.

The intermittent — TestTermDisagreementResolvesToTheHigherTerm — was the term rule, not timing. The replica refused every command carrying a term below its own and stopped its loop, on the premise "a lower term means the log went backwards". That premise is false: a copy resuming at the leader's retained base replays commands written before it learned its term, because the retained prefix predates the raise by definition. Applying them is safe by construction — an Append the copy holds is a repeat, a Trim is a floor, a Holders epoch and a consumer upsert only move forward, a Reset is the replay's own primitive — and the term still only moves forward. Only a Term announcement below the copy's term is not replayable, so it is ignored rather than moving routing back to the leader it names.

That is a contract correction, so the model moved with it: TestReplicaAppliesHistoryBelowItsTerm (renamed from ...RefusesACommandBelowItsTerm), TestLeadershipStateProperties now models "history is applied, term never regresses, a below-term announcement does not move the leader", and doc.go's failover section states the rule. 10 consecutive runs of the test green; three consecutive full-suite runs green (88 s, 84 s, 87 s).

A second defect, found because the full suite started hanging in unrelated tests: a replica that re-registers its consumer pulled again immediately, with no pause. A leader that keeps dropping the consumer therefore turned the loop into a spin — and inside a synctest bubble a spinning goroutine freezes the fake clock, so other multi-node tests hung for minutes with every goroutine reported select (durable). The re-registration is now paced by the same retry interval as every other loop error.

-race cannot run in this container, and this is a host-side issue. Verified, not guessed:

  • go version go1.27.3, GOROOT=/usr/local/go;
  • /proc/self/mountinfo shows 259:2 /usr/local/go /usr/local/go ro — the container does read the host's GOROOT;
  • the check bytes at file offset 0xC128 of src/runtime/race/race_linux_arm64.syso are 1f c0 00 71 (the unpatched cmp w0, #0x30), mtime 2026-10-19 22:20;
  • go test -race -a … still dies FATAL: Found 47 - Supported 48, so it is not a build-cache artifact.

So the host patch did not reach that file — most likely the toolchain was installed/replaced on 2026-10-19 after the last patch run (the script's own docs say to re-run it after every Go upgrade). On the host: sudo scripts/patch-race-runtime.sh (it verifies the bytes and refuses if they are neither original nor patched), then make test in the container should start normally. Ping me and I will run the race gate and report.

Everything else on the gate is clean: go vet and golangci-lint on both tag sets, go test -short ./... (~10 s), the full untagged profile at 86.0% with its coverage.log line added.

Head `cdb0912`: the intermittent is fixed, and one more restart-path defect fell out of chasing it. **The intermittent — `TestTermDisagreementResolvesToTheHigherTerm`** — was the term rule, not timing. The replica refused every command carrying a term below its own and stopped its loop, on the premise "a lower term means the log went backwards". That premise is false: a copy resuming at the leader's retained base replays commands written *before* it learned its term, because the retained prefix predates the raise by definition. Applying them is safe by construction — an Append the copy holds is a repeat, a Trim is a floor, a Holders epoch and a consumer upsert only move forward, a Reset is the replay's own primitive — and the term still only moves forward. Only a `Term` announcement below the copy's term is not replayable, so it is ignored rather than moving routing back to the leader it names. That is a contract correction, so the model moved with it: `TestReplicaAppliesHistoryBelowItsTerm` (renamed from `...RefusesACommandBelowItsTerm`), `TestLeadershipStateProperties` now models "history is applied, term never regresses, a below-term announcement does not move the leader", and `doc.go`'s failover section states the rule. 10 consecutive runs of the test green; three consecutive full-suite runs green (88 s, 84 s, 87 s). **A second defect, found because the full suite started hanging in unrelated tests:** a replica that re-registers its consumer pulled again immediately, with no pause. A leader that keeps dropping the consumer therefore turned the loop into a spin — and inside a `synctest` bubble a spinning goroutine freezes the fake clock, so other multi-node tests hung for minutes with every goroutine reported `select (durable)`. The re-registration is now paced by the same retry interval as every other loop error. **`-race` cannot run in this container, and this is a host-side issue.** Verified, not guessed: - `go version go1.27.3`, `GOROOT=/usr/local/go`; - `/proc/self/mountinfo` shows `259:2 /usr/local/go /usr/local/go ro` — the container does read the host's GOROOT; - the check bytes at file offset `0xC128` of `src/runtime/race/race_linux_arm64.syso` are `1f c0 00 71` (the **unpatched** `cmp w0, #0x30`), mtime `2026-10-19 22:20`; - `go test -race -a …` still dies `FATAL: Found 47 - Supported 48`, so it is not a build-cache artifact. So the host patch did not reach that file — most likely the toolchain was installed/replaced on 2026-10-19 after the last patch run (the script's own docs say to re-run it after every Go upgrade). On the host: `sudo scripts/patch-race-runtime.sh` (it verifies the bytes and refuses if they are neither original nor patched), then `make test` in the container should start normally. Ping me and I will run the race gate and report. Everything else on the gate is clean: `go vet` and `golangci-lint` on both tag sets, `go test -short ./...` (~10 s), the full untagged profile at 86.0% with its `coverage.log` line added.
Getting -race to run (host runtime patched, cache cleared) turned up a data race and two
tests that only the race build slowed enough to fail.

The race: startReplicaLoop added to a shared sync.WaitGroup while stopLoops waited on it.
A WaitGroup may not be added to while a Wait is in flight on a zero counter, and the
window was real -- a request still being served could spawn a loop after Close began
waiting, which is exactly the loop that would write to the store after it closed. Each
loop now carries its own done channel, closed by the loop itself, so stopping waits per
run and no Add can overlap a Wait. A permanent "stopped" flag was rejected: stopLoops
also serves tests that quiesce a replica and then restart its loop.

TestAddPeerStopsAtTheMemberCap assigned mesh.config directly while the pass read it under
configLock; it adopts through adoptConfig now.

ensureReplica created an adopted copy with createStream, which on a mesh publishes a
cluster record and waits for a member quorum -- blocking the reconciliation pass for
seconds to adopt a stream that by definition has no record. The safety net creates the
copy locally.

TestCreationRecordsThePlacementEpoch asserted the epoch on the node the hash names;
under -race another node promoted first. It now asserts on the node that serves.

Gates: go test -race ./... green (83s), go test -race -tags rapid ./... green (102s),
go vet and golangci-lint clean on both tag sets.
Author
Member

-race runs now, and it was worth waiting for.

Why it failed before: the syso is patched (bytes@0xC128 = 1f bc 00 71, mtime 20:54 today) — the blocker was the build cache, which still held the race runtime compiled from the unpatched file. go test -race -a (force rebuild) started fine, and go clean -cache once makes plain go test -race ./... work. Worth noting for the next Go upgrade: patch, then clean the cache, or keep using -a.

What the race build found (head 192b893):

  1. A real data race, in production code. startReplicaLoop added to a shared sync.WaitGroup (meshPeers.loopsDone) while stopLoops was waiting on it. A WaitGroup may not be added to while a Wait is in flight on a zero counter — and the window was live: a request still being served could spawn a loop after Close began waiting, which is precisely the loop that would write to the store after it closed. Each loop now has its own done channel closed by the loop itself, so stopping waits per run and no Add can overlap a Wait. A "loops stopped" flag was rejected: stopLoops also serves tests that quiesce a replica and restart its loop afterwards.
  2. TestAddPeerStopsAtTheMemberCap assigned mesh.config directly while the pass read it under configLock; it adopts through adoptConfig now.
  3. ensureReplica created an adopted copy with createStream, which on a mesh publishes a cluster record and waits for a member quorum — blocking a reconciliation pass for seconds to adopt a stream that by definition has no record. The safety net creates the copy locally.
  4. TestCreationRecordsThePlacementEpoch asserted the epoch on the node the hash names; under -race a different node promoted first. It asserts on the node that actually serves the stream now.

Gates, all green on 192b893:

check result
go test -race ./... ok, 83.5 s, no DATA RACE
go test -race -tags rapid ./... (make test-props) ok, 102 s
go test ./... ok, 84–88 s (three consecutive runs)
go test -short ./... ok, ~10 s
go vet, golangci-lint clean, both tag sets
untagged coverage 86.0%, coverage.log line added

Still open from the phase's own list: a re-create after delete and a restart does not reach the restarted node (TestATombstoneDoesNotResurrectOnRestart asserts the phase's claims and says where that one is left).

`-race` runs now, and it was worth waiting for. **Why it failed before:** the syso is patched (`bytes@0xC128 = 1f bc 00 71`, mtime 20:54 today) — the blocker was the **build cache**, which still held the race runtime compiled from the unpatched file. `go test -race -a` (force rebuild) started fine, and `go clean -cache` once makes plain `go test -race ./...` work. Worth noting for the next Go upgrade: patch, then clean the cache, or keep using `-a`. **What the race build found (head `192b893`):** 1. **A real data race, in production code.** `startReplicaLoop` added to a shared `sync.WaitGroup` (`meshPeers.loopsDone`) while `stopLoops` was waiting on it. A `WaitGroup` may not be added to while a `Wait` is in flight on a zero counter — and the window was live: a request still being served could spawn a loop *after* `Close` began waiting, which is precisely the loop that would write to the store after it closed. Each loop now has its own `done` channel closed by the loop itself, so stopping waits per run and no `Add` can overlap a `Wait`. A "loops stopped" flag was rejected: `stopLoops` also serves tests that quiesce a replica and restart its loop afterwards. 2. `TestAddPeerStopsAtTheMemberCap` assigned `mesh.config` directly while the pass read it under `configLock`; it adopts through `adoptConfig` now. 3. `ensureReplica` created an adopted copy with `createStream`, which on a mesh publishes a cluster record and waits for a member quorum — blocking a reconciliation pass for seconds to adopt a stream that by definition has no record. The safety net creates the copy locally. 4. `TestCreationRecordsThePlacementEpoch` asserted the epoch on the node the hash names; under `-race` a different node promoted first. It asserts on the node that actually serves the stream now. **Gates, all green on `192b893`:** | check | result | | --- | --- | | `go test -race ./...` | ok, 83.5 s, no `DATA RACE` | | `go test -race -tags rapid ./...` (`make test-props`) | ok, 102 s | | `go test ./...` | ok, 84–88 s (three consecutive runs) | | `go test -short ./...` | ok, ~10 s | | `go vet`, `golangci-lint` | clean, both tag sets | | untagged coverage | 86.0%, `coverage.log` line added | Still open from the phase's own list: a re-create after delete *and* a restart does not reach the restarted node (`TestATombstoneDoesNotResurrectOnRestart` asserts the phase's claims and says where that one is left).
The last open item of the phase was that a re-create after a delete, with a restart in
between, never reached the restarted node. The writer did not publish the second record
at all: Conn.DeleteStream kept its cached handle when the delete's quorum failed, so the
client's next CreateStream was answered from that stale handle and never asked the
server. A delete that could not reach its quorum is still durable on the writer and
applied there — the record is written, the wait is what failed — so the handle belongs to
a stream the mesh has deleted. It is dropped for every answer, including a failed one; a
transport failure keeps it, because there the outcome is unknown.

TestATombstoneDoesNotResurrectOnRestart now asserts the re-create end to end (three
consecutive runs), which is what that item was.

Also: TestConsumeSurvivesEviction waited five seconds for a consume loop to release its
stream. The loop releases on its way out of a pull it is already inside, so under -race
that can take longer; the assertion is that it released at all, so it gets the
convergence budget.

Gates: go test ./... green (88s), go test -race ./... green twice (87s, 86s),
go test -short ./... green, go vet and golangci-lint clean on both tag sets.
Author
Member

Last open item closed on 3a860aa: a re-create after a delete, with a restart in between, now reaches the restarted node.

It was a client bug, not a server one. The writer published nothing for the re-create because the create never reached the server: Conn.DeleteStream kept its cached *Stream handle when the delete's quorum failed. In the reproduction the delete is issued while one member is down, so it answers ErrNoQuorum — the record is durable on the writer and applied there, only the wait failed — and the following Conn.CreateStream was then answered from that stale handle (get-or-create hit, no RPC), so no StreamMeta was ever written.

Conn.DeleteStream now drops the handle for every answer the server gave, including a failed one; a transport failure still keeps it, because there the outcome is unknown. TestATombstoneDoesNotResurrectOnRestart asserts the whole sequence end to end — create, publish, converge, stop a member, delete (tolerating ErrNoQuorum), restart, no resurrection, then re-create and the restarted node adopting it — three consecutive runs green, and it is no longer commented as a known gap.

One test-tightness change came with it: TestConsumeSurvivesEviction waited five seconds for a consume loop to release its stream. The loop releases on its way out of a pull it is already inside, so under -race that can exceed five seconds; the assertion is that it released at all, so it uses the convergence budget.

Gates on 3a860aa: go test ./... ok (88 s), go test -race ./... ok twice (87 s, 86 s), go test -short ./... ok, go vet and golangci-lint clean on both tag sets. Nothing in the phase is left open now.

Last open item closed on `3a860aa`: a re-create after a delete, with a restart in between, now reaches the restarted node. **It was a client bug, not a server one.** The writer published nothing for the re-create because the create never reached the server: `Conn.DeleteStream` kept its cached `*Stream` handle when the delete's quorum failed. In the reproduction the delete is issued while one member is down, so it answers `ErrNoQuorum` — the record *is* durable on the writer and applied there, only the wait failed — and the following `Conn.CreateStream` was then answered from that stale handle (get-or-create hit, no RPC), so no `StreamMeta` was ever written. `Conn.DeleteStream` now drops the handle for every answer the server gave, including a failed one; a transport failure still keeps it, because there the outcome is unknown. `TestATombstoneDoesNotResurrectOnRestart` asserts the whole sequence end to end — create, publish, converge, stop a member, delete (tolerating `ErrNoQuorum`), restart, no resurrection, then re-create and the restarted node adopting it — three consecutive runs green, and it is no longer commented as a known gap. One test-tightness change came with it: `TestConsumeSurvivesEviction` waited five seconds for a consume loop to release its stream. The loop releases on its way out of a pull it is already inside, so under `-race` that can exceed five seconds; the assertion is that it released at all, so it uses the convergence budget. Gates on `3a860aa`: `go test ./...` ok (88 s), `go test -race ./...` ok twice (87 s, 86 s), `go test -short ./...` ok, `go vet` and `golangci-lint` clean on both tag sets. Nothing in the phase is left open now.
docs: trim the comments this branch added
All checks were successful
code-review / Agent review (pull_request) Successful in 10s
d1583a0140
Comment-only. The exported names keep a godoc line opening with the name; the rest keeps
only what the code cannot say (an ordering or locking constraint, a failure mode, a wire
or membership invariant) and drops the narration, the history and the retelling of the
body. Test comments are gone unless the setup itself is the reason the case is shaped
that way — which node a command is issued at, why a copy is seeded locally.

  493 comment lines removed or shortened across 16 files, and no code token changed: the
  only non-comment line in the diff is gofmt realigning a const block after its comment
  went.

Gates: go test ./... green (83s), go test -race ./... green (85s), go test -short ./...
green, go vet and golangci-lint clean on both tag sets.
Author
Member

Comment-trim pass done on d1583a0 (rebased onto your merge of main, 93196ad; pushed as a normal fast-forward, nothing forced).

Same rule as PR #17: the exported names keep a godoc line opening with the name, everything else keeps only what the code cannot say — an ordering or locking constraint, a failure mode, a wire or membership invariant, a rejected alternative that would otherwise be re-litigated — and the narration, the history and the retelling of the body are gone. Test comments are gone unless the setup is the reason the case is shaped that way (which node a command is issued at, why a copy is seeded locally, why the epoch is asserted on the serving node).

  • 493 comment lines removed or shortened across 16 files.
  • Comment-only: the single non-comment line in the diff is gofmt realigning the replicaConsumerPrefix/metaConsumerPrefix const block after the comment above it went away. No code token changed.
  • Public API docs (doc.go's "The cluster log" and term sections) kept, tightened; the roadmap line ("scheduled for deletion one release after the log ships") moved out of godoc — that belongs in plan.md, not the package documentation.

Gates after the rebase: go test ./... ok (83 s), go test -race ./... ok (93 s, no DATA RACE), go test -short ./... ok, go vet and golangci-lint clean on both tag sets.

Comment-trim pass done on `d1583a0` (rebased onto your merge of `main`, `93196ad`; pushed as a normal fast-forward, nothing forced). Same rule as PR #17: the exported names keep a godoc line opening with the name, everything else keeps only what the code cannot say — an ordering or locking constraint, a failure mode, a wire or membership invariant, a rejected alternative that would otherwise be re-litigated — and the narration, the history and the retelling of the body are gone. Test comments are gone unless the setup *is* the reason the case is shaped that way (which node a command is issued at, why a copy is seeded locally, why the epoch is asserted on the serving node). - 493 comment lines removed or shortened across 16 files. - Comment-only: the single non-comment line in the diff is `gofmt` realigning the `replicaConsumerPrefix`/`metaConsumerPrefix` const block after the comment above it went away. No code token changed. - Public API docs (`doc.go`'s "The cluster log" and term sections) kept, tightened; the roadmap line ("scheduled for deletion one release after the log ships") moved out of godoc — that belongs in `plan.md`, not the package documentation. Gates after the rebase: `go test ./...` ok (83 s), `go test -race ./...` ok (93 s, no `DATA RACE`), `go test -short ./...` ok, `go vet` and `golangci-lint` clean on both tag sets.
pi-reviewer left a comment

Phase 6 review (head d1583a0, base 5f9464f). I read the full diff and the surrounding code, then verified the tooling claims and every API/behaviour claim I make below with a command in this session.

Ran here (clone of origin/feat/cluster-metadata-replication, go1.27.3):

  • go build ./... — clean
  • go vet ./... — clean
  • gofmt -l . — no output
  • golangci-lint run (~/.golangci.yml default) — 0 issues.
  • go test -short ./... — ok git.norobo.chat/norobo/webstream 5.545s
  • go test ./... — ok git.norobo.chat/norobo/webstream 83.112s

So the PR's gate claims reproduce. The findings below are the mechanisms the green suite does not cover.

Findings: 3 IMPORTANT, 3 SUGGESTION.

  1. 🟠 A local create failure inside applyStreamMeta wedges the cluster-log applier loop (and surfaces to the client as a failed create for a record that is already durable).
  2. 🟠 admitTerm no longer refuses a below-term Append, and applying one at a sequence the copy does not hold resets the copy's window — so a superseded leader that still serves pulls can overwrite a replica's history. This is the same loosening that fixed defect 7, but it is wider than the comment claims.
  3. 🟠 behindLatecomers never forgets a peer, so one member that dies or is removed with a stale epoch blocks the cluster-log handover for the life of the process.
  4. 🟡 Server.Streams() is permanently empty without a mesh, undocumented.
  5. 🟡 every ListStreams page now materializes the whole stream set.
  6. 🟡 t.out (a test-run artifact) is committed.

Not flagged: the replicaConsumerName(stream, id) rework (I traced meta-<peer> through ack.stream = internalName(name) and checkRelayedAckVote's userName round trip — consistent for both the cluster log and its command log), and the removed pickAnnounced self-filter (both former callers now filter leader == s.host.ID() themselves).

Phase 6 review (head d1583a0, base 5f9464f). I read the full diff and the surrounding code, then verified the tooling claims and every API/behaviour claim I make below with a command in this session. **Ran here** (clone of origin/feat/cluster-metadata-replication, go1.27.3): - `go build ./...` — clean - `go vet ./...` — clean - `gofmt -l .` — no output - `golangci-lint run` (`~/.golangci.yml` default) — `0 issues.` - `go test -short ./...` — `ok git.norobo.chat/norobo/webstream 5.545s` - `go test ./...` — `ok git.norobo.chat/norobo/webstream 83.112s` So the PR's gate claims reproduce. The findings below are the mechanisms the green suite does not cover. **Findings:** 3 IMPORTANT, 3 SUGGESTION. 1. 🟠 A local create failure inside `applyStreamMeta` wedges the cluster-log applier loop (and surfaces to the client as a failed create for a record that is already durable). 2. 🟠 `admitTerm` no longer refuses a below-term Append, and applying one at a sequence the copy does not hold resets the copy's window — so a superseded leader that still serves pulls can overwrite a replica's history. This is the same loosening that fixed defect 7, but it is wider than the comment claims. 3. 🟠 `behindLatecomers` never forgets a peer, so one member that dies or is removed with a stale epoch blocks the cluster-log handover for the life of the process. 4. 🟡 `Server.Streams()` is permanently empty without a mesh, undocumented. 5. 🟡 every `ListStreams` page now materializes the whole stream set. 6. 🟡 `t.out` (a test-run artifact) is committed. Not flagged: the `replicaConsumerName(stream, id)` rework (I traced `meta-<peer>` through `ack.stream` = `internalName(name)` and `checkRelayedAckVote`'s `userName` round trip — consistent for both the cluster log and its command log), and the removed `pickAnnounced` self-filter (both former callers now filter `leader == s.host.ID()` themselves).
@ -147,0 +157,4 @@
}
// behindLatecomers reports whether a member last reported an epoch below the applied one.
func (m *meshPeers) behindLatecomers() bool {
Member

🟠 IMPORTANT — behindLatecomers never forgets a peer, so it can block the cluster-log handover for ever.

m.reported is written only by observeEpoch (from probePeer, i.e. only for current members that answer) and is never pruned: adoptConfig, releaseRemovedPeers and markDead all leave it alone. A member whose last answer was epoch 4, and which then dies, is partitioned, or is removed while the set moves to epoch 5, keeps reported[that peer] = 4 for the process's life.

handOverClusterLog (membership.go: if s.mesh.behindLatecomers() { return }) then returns early on every reconcile pass, so the log never moves to the node hashLeader(clusterStream, members) names, for as long as this node runs.

Probe I ran against this tree: members = [2g 2h], reported = map[2i:3], isMember(2i) = false, behindLatecomers = true.

Fix: in the loop, skip ids that are not in m.members() (or delete their entries in adoptConfig, next to where the applied set changes).

🟠 **IMPORTANT — `behindLatecomers` never forgets a peer, so it can block the cluster-log handover for ever.** `m.reported` is written only by `observeEpoch` (from `probePeer`, i.e. only for current members that answer) and is never pruned: `adoptConfig`, `releaseRemovedPeers` and `markDead` all leave it alone. A member whose last answer was epoch 4, and which then dies, is partitioned, or is removed while the set moves to epoch 5, keeps `reported[that peer] = 4` for the process's life. `handOverClusterLog` (`membership.go`: `if s.mesh.behindLatecomers() { return }`) then returns early on every reconcile pass, so the log never moves to the node `hashLeader(clusterStream, members)` names, for as long as this node runs. Probe I ran against this tree: `members = [2g 2h], reported = map[2i:3], isMember(2i) = false, behindLatecomers = true`. **Fix:** in the loop, skip ids that are not in `m.members()` (or delete their entries in `adoptConfig`, next to where the applied set changes).
metadata.go Outdated
@ -0,0 +288,4 @@
// Streams reports the stream set the cluster log describes, with each stream's retention window
// and the holders its placement epoch names. It answers from applied state and asks no peer.
func (s *Server) Streams() []StreamMeta {
Member

🟡 SUGGESTION — Streams() is empty on a server without a mesh, and the doc does not say so.

clusterMeta is filled only by applyStreamMeta/applyStreamTombstone, which are reached only from the mesh paths, so a standalone server answers with nothing while Conn.ListStreams lists its streams.

Probe I ran against this tree, standalone server with orders created: hasStream(orders) = true, len(Streams()) = 0, len(listStreamNames) = 1.

Fix: either state "on a mesh" in the doc comment, or answer from localStreamNames when s.mesh == nil, as listStreamNames already does.

🟡 **SUGGESTION — `Streams()` is empty on a server without a mesh, and the doc does not say so.** `clusterMeta` is filled only by `applyStreamMeta`/`applyStreamTombstone`, which are reached only from the mesh paths, so a standalone server answers with nothing while `Conn.ListStreams` lists its streams. Probe I ran against this tree, standalone server with `orders` created: `hasStream(orders) = true, len(Streams()) = 0, len(listStreamNames) = 1`. **Fix:** either state "on a mesh" in the doc comment, or answer from `localStreamNames` when `s.mesh == nil`, as `listStreamNames` already does.
metadata.go Outdated
@ -0,0 +390,4 @@
}
if resp := s.createStreamLocal(ctx, createStreamRequest{name: meta.Name, maxAge: meta.MaxAge}); resp.err() != nil {
return fmt.Errorf("creating stream %q: %w", meta.Name, resp.err())
Member

🟠 IMPORTANT — a local create failure wedges the applier loop.

This returns the error from createStreamLocal to the caller. There are two callers, and both are worse off for it:

  1. The cluster-log applier (replicate.go, replicateLoop: err = s.applyClusterRecord(ctx, cmd.rec.data)) then deliberately does not ack and re-pulls the same command — "the command stays unapplied on the leader's side of the watermark and is redelivered". A record this node can never materialize is therefore retried for ever, so this member stops applying every later record: later StreamMetas, Tombstones and Config epoch changes, while its meta-<peer> consumer keeps pinning the log's trim floor.
  2. createClusteredStream calls applyStreamMeta on the writer, so the client gets ErrLimitExceeded for a record that is already durable cluster-wide.

Condition: the member is at maxStreamsPerServer (1024) or createStreamLocal hits a store error (%w: creating stream: %v, ErrInternal) when the create record arrives. Both are local, permanent conditions.

Fix: for a failure to materialize the local copy, log and return nil — the metadata is already applied by s.meta.applyMeta(meta) and noteUnheldFromMetadata re-queues the copy on a later pass. Keep returning an error only for a decode/apply fault. Same for the writer: the record's durability does not depend on the local copy.

🟠 **IMPORTANT — a local create failure wedges the applier loop.** This returns the error from `createStreamLocal` to the caller. There are two callers, and both are worse off for it: 1. The cluster-log applier (`replicate.go`, `replicateLoop`: `err = s.applyClusterRecord(ctx, cmd.rec.data)`) then deliberately does not ack and re-pulls the same command — "the command stays unapplied on the leader's side of the watermark and is redelivered". A record this node can never materialize is therefore retried for ever, so this member stops applying every later record: later `StreamMeta`s, `Tombstone`s *and* `Config` epoch changes, while its `meta-<peer>` consumer keeps pinning the log's trim floor. 2. `createClusteredStream` calls `applyStreamMeta` on the writer, so the client gets `ErrLimitExceeded` for a record that is already durable cluster-wide. **Condition:** the member is at `maxStreamsPerServer` (1024) or `createStreamLocal` hits a store error (`%w: creating stream: %v`, `ErrInternal`) when the create record arrives. Both are local, permanent conditions. **Fix:** for a failure to materialize the *local copy*, log and `return nil` — the metadata is already applied by `s.meta.applyMeta(meta)` and `noteUnheldFromMetadata` re-queues the copy on a later pass. Keep returning an error only for a decode/apply fault. Same for the writer: the record's durability does not depend on the local copy.
@ -1468,3 +1561,1 @@
return fmt.Errorf("%w: stream %q is at term %d, command carries %d",
errStaleTermCommand, name, current, term)
case term == current:
case cmd.term < current:
Member

🟠 IMPORTANT — a below-term Append from a superseded writer is applied, and can reset the copy's window.

case cmd.term < current: return nil admits every below-term command, Appends included. admitTerm's only production caller is replicateLoop (if err = s.admitTerm(ctx, name, cmd); err == nil { switch cmd.kind { case commandAppend: ... }), which then applies the record.

Probe I ran against this tree (added to the package, run, then removed):

admitTerm(below-term Append seq 7) error = <nil>
window base before = 0, after applying seq 7 with base 0 = 7, next = 8

The second line is applyReplicated's if rec.seq > s.log.next { resetToLocked(max(base, rec.seq)) } — the copy's memlog is replaced and restarted at the foreign record (resetToLocked sets base = next = seq). So a record the copy does not hold is not "the divergence it was"; it is adopted, and what the copy held below it is discarded.

Condition: the copy learned a higher term (a Term announcement from the new leader, which admitTerm applies at cmd.term == current) while its loop still targets the superseded leader — the loop's leader is only realigned by realignReplicaLoops, once per reconcile pass — and that node still serves framePull because it still believes it leads. The pull path checks the serving node's leadership, never the sender's term, so the comment's "a superseded leader cannot reach here" does not hold for this direction.

Fix: pass the loop's leader into admitTerm and admit a below-term command only while that node is the copy's announced leader (state.leaderValue()); otherwise drop the command and retireReplicaLoop so the pass re-attaches to the announced leader. The case defect 7 needed still passes, because there the sender is the announced leader and the records are its own history.

🟠 **IMPORTANT — a below-term Append from a superseded writer is applied, and can reset the copy's window.** `case cmd.term < current: return nil` admits *every* below-term command, Appends included. `admitTerm`'s only production caller is `replicateLoop` (`if err = s.admitTerm(ctx, name, cmd); err == nil { switch cmd.kind { case commandAppend: ... }`), which then applies the record. Probe I ran against this tree (added to the package, run, then removed): ``` admitTerm(below-term Append seq 7) error = <nil> window base before = 0, after applying seq 7 with base 0 = 7, next = 8 ``` The second line is `applyReplicated`'s `if rec.seq > s.log.next { resetToLocked(max(base, rec.seq)) }` — the copy's memlog is replaced and restarted at the foreign record (`resetToLocked` sets `base = next = seq`). So a record the copy does not hold is not "the divergence it was"; it is adopted, and what the copy held below it is discarded. **Condition:** the copy learned a higher term (a `Term` announcement from the new leader, which `admitTerm` applies at `cmd.term == current`) while its loop still targets the superseded leader — the loop's `leader` is only realigned by `realignReplicaLoops`, once per reconcile pass — and that node still serves `framePull` because it still believes it leads. The pull path checks the *serving* node's leadership, never the sender's term, so the comment's "a superseded leader cannot reach here" does not hold for this direction. **Fix:** pass the loop's `leader` into `admitTerm` and admit a below-term command only while that node is the copy's announced leader (`state.leaderValue()`); otherwise drop the command and `retireReplicaLoop` so the pass re-attaches to the announced leader. The case defect 7 needed still passes, because there the sender *is* the announced leader and the records are its own history.
server.go Outdated
@ -2276,0 +2321,4 @@
return s.localStreamNames(req)
}
names, _, err := s.localStreamNames(listStreamsRequest{})
Member

🟡 SUGGESTION — every ListStreams page now materializes the whole stream set.

localStreamNames(listStreamsRequest{}) passes an empty request, and listStorePage treats limit == 0 as "no cap", so a durable node enumerates the entire streams/ directory; s.meta.all() then clones every holder slice and the union is sorted before req.limit is applied. The pre-change path read limit+1 directory entries. Clients poll ListStreams, so this is O(all streams) + a sort per page request.

Fix: both sources are already sorted — walk them with the cursor and stop at req.limit instead of materializing the union (the metadata side is s.meta.all(); keep a sorted index or sort once per generation if that walk matters).

🟡 **SUGGESTION — every `ListStreams` page now materializes the whole stream set.** `localStreamNames(listStreamsRequest{})` passes an empty request, and `listStorePage` treats `limit == 0` as "no cap", so a durable node enumerates the entire `streams/` directory; `s.meta.all()` then clones every holder slice and the union is sorted before `req.limit` is applied. The pre-change path read `limit+1` directory entries. Clients poll `ListStreams`, so this is O(all streams) + a sort per page request. **Fix:** both sources are already sorted — walk them with the cursor and stop at `req.limit` instead of materializing the union (the metadata side is `s.meta.all()`; keep a sorted index or sort once per generation if that walk matters).
t.out Outdated
@ -0,0 +1 @@
ok git.norobo.chat/norobo/webstream 3.173s
Member

🟡 SUGGESTION — a test-run artifact is committed.

t.out contains one line of go test output (ok git.norobo.chat/norobo/webstream 3.173s). .gitignore lists *.test and coverage.out, not this, and AGENTS.md now recommends redirecting long runs to a file — which is how it got here.

Fix: git rm --cached t.out and add t.out (or *.out) to .gitignore in the same commit.

🟡 **SUGGESTION — a test-run artifact is committed.** `t.out` contains one line of `go test` output (`ok git.norobo.chat/norobo/webstream 3.173s`). `.gitignore` lists `*.test` and `coverage.out`, not this, and `AGENTS.md` now recommends redirecting long runs to a file — which is how it got here. **Fix:** `git rm --cached t.out` and add `t.out` (or `*.out`) to `.gitignore` in the same commit.
1. A local copy that cannot be made no longer fails the record. applyStreamMeta logged
   and returned the create error, so the applier withheld its ack and re-pulled that
   record for ever — blocking every later StreamMeta, Tombstone and Config epoch while
   its meta- consumer pinned the log — and the writer returned ErrLimitExceeded for a
   create that was already durable cluster-wide. The copy is placed work the pass
   re-queues; the record's application is the metadata, which is already applied.

2. A below-term command is admitted only from the node the copy knows as its leader.
   admitTerm accepted any below-term command, so a superseded writer that still serves
   pulls could feed a copy its older log, and a gap in it is adopted by resetting the
   window — which the reviewer demonstrated (a below-term Append at seq 7 replaced the
   memlog and restarted it there). Anywhere else the command is refused and the loop
   ends, so the pass attaches to the announced leader.

3. behindLatecomers skips ids that are not members and does not wait on a dead one, and
   adoptConfig prunes the reports of departed members. One peer that died or was removed
   with a stale epoch blocked the cluster-log handover for the life of the process.

4. Server.Streams answers on a standalone server too (the local set with its windows),
   and returns an error there because it reads the store.

5. ListStreams pages the merge of the two sorted name sets after the cursor instead of
   materializing the union and sorting it: the local side is a directory page and the
   metadata side is a slice of a maintained name index. A tombstone can shorten a page,
   so the window widens until it is full, a source is exhausted, or four doublings pass.

6. t.out is gone and *.out is ignored.

Tests: TestAFailedLocalCopyDoesNotFailTheRecord, TestBehindLatecomersIgnoresTheDeparted
AndTheDead, TestAStandaloneStreamsReportsTheLocalSet, TestListStreamsPagesTheMergedName
Sets, and TestReplicaAppliesHistoryBelowItsTerm plus the leadership model extended for
the announced-leader rule.

Gates: go test ./... ok (86s), go test -race ./... ok (87s), go test -race -tags rapid
./... ok (95s), go test -short ./... ok, go vet and golangci-lint clean on both tags.
Author
Member

Review findings resolved on cce9f18. Each fix has the check that fails without it.

1 🟠 A local create failure wedging the applier — fixed. applyStreamMeta now logs a failed copy and returns nil (createPlacedCopy): the record's application is the metadata, which is already applied, and the copy is placed work the pass re-queues (noteUnheldFromMetadata). No withheld ack, no re-pull loop, and the writer no longer fails a create that is durable cluster-wide.
Check: TestAFailedLocalCopyDoesNotFailTheRecord — fills maxStreamsPerServer slots, applies a StreamMeta, asserts nil error, metadata present, no copy.

2 🟠 A below-term Append from a superseded writer — fixed, narrower. admitTerm now takes the sender (from peer.ID) and admits a below-term command only when that node is the copy's announced leader; anywhere else it returns errStaleTermCommand, the loop logs "replication left a superseded writer" and ends, so the pass re-attaches at the announced leader. The case defect 7 needed still holds: there the sender is the announced leader and the records are its own history. doc.go's failover paragraph and admitTerm's comment state the rule now, including the superseded-writer direction.
Checks: TestReplicaAppliesHistoryBelowItsTerm (history from the announced leader is applied; the same command from old-leader is refused) and TestLeadershipStateProperties, whose model now draws the sender and requires term < model.term && from != model.leader to be refused.

3 🟠 behindLatecomers never forgetting a peer — fixed. It skips ids that are not in the applied member set and does not wait on a dead one (m.alive), and adoptConfig prunes the reports of departed members next to the set change.
Check: TestBehindLatecomersIgnoresTheDepartedAndTheDead — older epoch from a live member blocks; markDead unblocks; removal via adoptConfig unblocks and prunes.

4 🟡 Streams() on a server without a mesh — fixed. It answers with the local set (name and window, no holders) and is documented as such. The signature gained (ctx context.Context) ([]StreamMeta, error) because that path reads the store, so it can fail instead of silently answering empty.
Check: TestAStandaloneStreamsReportsTheLocalSet.

5 🟡 Every ListStreams page materializing everything — fixed. clusterMeta keeps a name index (slices.BinarySearch/Insert/Delete on apply), namesAfter(cursor, n) returns a slice of it, and a page is the linear merge of that with a limit+1 directory page after the cursor. Both sides are incremental now; the window widens (bounded, four doublings) only when a tombstone shortens a page while a source still has names.
Check: TestListStreamsPagesTheMergedNameSets — five streams, one name per page, from the non-writer; every name once, in order, hasMore only while names remain.

6 🟡 t.out — fixed. Removed and *.out added to .gitignore.

Gates on cce9f18:

check result
go test ./... ok, 86 s
go test -race ./... ok, 87 s, no DATA RACE
go test -race -tags rapid ./... ok, 95 s
go test -short ./... ok
go vet, golangci-lint (both tag sets) clean

For the record on your two "not flagged" notes: agreed, and the replicaConsumerName path is now also exercised end to end by the metadata tests, since every converge in them depends on a meta- ack counting as a vote.

Review findings resolved on `cce9f18`. Each fix has the check that fails without it. **1 🟠 A local create failure wedging the applier — fixed.** `applyStreamMeta` now logs a failed copy and returns nil (`createPlacedCopy`): the record's application is the metadata, which is already applied, and the copy is placed work the pass re-queues (`noteUnheldFromMetadata`). No withheld ack, no re-pull loop, and the writer no longer fails a create that is durable cluster-wide. Check: `TestAFailedLocalCopyDoesNotFailTheRecord` — fills `maxStreamsPerServer` slots, applies a `StreamMeta`, asserts nil error, metadata present, no copy. **2 🟠 A below-term Append from a superseded writer — fixed, narrower.** `admitTerm` now takes the sender (`from peer.ID`) and admits a below-term command only when that node is the copy's announced leader; anywhere else it returns `errStaleTermCommand`, the loop logs "replication left a superseded writer" and ends, so the pass re-attaches at the announced leader. The case defect 7 needed still holds: there the sender *is* the announced leader and the records are its own history. `doc.go`'s failover paragraph and `admitTerm`'s comment state the rule now, including the superseded-writer direction. Checks: `TestReplicaAppliesHistoryBelowItsTerm` (history from the announced leader is applied; the same command from `old-leader` is refused) and `TestLeadershipStateProperties`, whose model now draws the sender and requires `term < model.term && from != model.leader` to be refused. **3 🟠 `behindLatecomers` never forgetting a peer — fixed.** It skips ids that are not in the applied member set and does not wait on a dead one (`m.alive`), and `adoptConfig` prunes the reports of departed members next to the set change. Check: `TestBehindLatecomersIgnoresTheDepartedAndTheDead` — older epoch from a live member blocks; `markDead` unblocks; removal via `adoptConfig` unblocks and prunes. **4 🟡 `Streams()` on a server without a mesh — fixed.** It answers with the local set (name and window, no holders) and is documented as such. The signature gained `(ctx context.Context) ([]StreamMeta, error)` because that path reads the store, so it can fail instead of silently answering empty. Check: `TestAStandaloneStreamsReportsTheLocalSet`. **5 🟡 Every `ListStreams` page materializing everything — fixed.** `clusterMeta` keeps a name index (`slices.BinarySearch`/`Insert`/`Delete` on apply), `namesAfter(cursor, n)` returns a slice of it, and a page is the linear merge of that with a `limit+1` directory page after the cursor. Both sides are incremental now; the window widens (bounded, four doublings) only when a tombstone shortens a page while a source still has names. Check: `TestListStreamsPagesTheMergedNameSets` — five streams, one name per page, from the non-writer; every name once, in order, `hasMore` only while names remain. **6 🟡 `t.out` — fixed.** Removed and `*.out` added to `.gitignore`. **Gates on `cce9f18`:** | check | result | | --- | --- | | `go test ./...` | ok, 86 s | | `go test -race ./...` | ok, 87 s, no `DATA RACE` | | `go test -race -tags rapid ./...` | ok, 95 s | | `go test -short ./...` | ok | | `go vet`, `golangci-lint` (both tag sets) | clean | For the record on your two "not flagged" notes: agreed, and the `replicaConsumerName` path is now also exercised end to end by the metadata tests, since every converge in them depends on a `meta-` ack counting as a vote.
sam merged commit 31ef490f80 into main 2026-11-05 07:56:49 +00:00
sam deleted branch feat/cluster-metadata-replication 2026-11-05 07:56:49 +00:00
Sign in to join this conversation.
No reviewers
No labels
code-review
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
norobo/webstream!22
No description provided.