feat: replicate records through a per-stream command log (phase 3) #8

Merged
sam merged 6 commits from feat/mesh-replication-command-log into main 2026-10-04 07:56:24 +00:00
Member

Implements phase 3 of plan.md: every node holds the leader's records, at the leader's sequence numbers and timestamps, and follows the leader's retained window.

A stream's writes now reach any node through its command log, the hidden companion stream internal-<name> that only the leader writes.

What changed

  • The command log is the write-ahead log. A publish appends and fsyncs Append{seq, timestamp, dedupID, data} first and only then applies the record to the leader's own copy, through the same applier a replica uses (streamState.applyRecordLocked). The publish path therefore has one durable write on it: a crash before it loses both copies and the publish was never acked; a crash after it is repaired.
  • Startup replay. Server.catchUp replays a led stream's own command log from its own apply position. It runs when a led stream becomes resident and before every publish reserves a sequence number, so a later publish cannot reuse a sequence number a durable command already describes with different data.
  • Strict applier. memlog.applyRecord stores a given seq and timestamp instead of minting them, refuses a gap, and drops a repeat (a command redelivered after a lost ack). This keeps every timestamp in a stream from the leader's clock.
  • Replica loop. Each replica registers its own consumer (rep-<peerID>) on the leader's command log, keeps one command in flight, and applies commands in order, so a replica stores the leader's records verbatim. Each loop has its own *Conn: the server dispatches one application stream's frames sequentially, so a blocking pull on the shared peer connection would head-of-line-block all reconciliation and forwarding for that peer.
  • Retention follows the leader. Trim{base} is published from a reconciliation pass when the leader's window moves, and a replica applies max(local base, base). A replica runs no age-trim loop of its own, so two nodes cannot trim to different windows; only its own byte budget can still drop its oldest segments.
  • Resync. A replica that fell below the command log's retained base (the byte budget dropped the commands for records it still needs) discards its copy, restarts at the leader's retained base, and backfills, instead of stalling on a gap. The retained base now travels on the frameGetStream reply, alongside the retention window a client already reads there.
  • A replica enforces the node's storage budget as its copy grows, exactly as a publish does on the leader; without that its storage would be bounded only by the leader's window.

Two supporting changes:

  • The relay path now accepts framePull and frameAck for a reserved internal name only, so a replica can read the leader's command log while a client's pull is still served locally and never relayed.
  • Close now stops the server serving. Inbound stream handlers ran on the NewServer context, which Close does not cancel, so a torn-down server kept answering peers on streams they had already negotiated — with a nil store. A node "restarted" on the same libp2p host was a zombie behind a live connection, which is what made the leader-restart and replica-restart cases fail. Cancelling a server-owned context at Close fixes it.

Tests (replication_test.go, memlog_test.go)

Backfill onto an empty replica, serving those records from the replica itself, restart mid-stream, idempotent re-application, gap rejection at the applier, leader trims reaching replicas, a stale trim not regressing the base, resync onto the leader's window (including recovery of the resynced copy, whose segment base is not a multiple of the segment size), the crash window between command append and apply, and a replica's copy being evictable between batches. Two phase-1 tests whose "the non-leader applies nothing" assertion phase 3 invalidates were updated to assert convergence instead.

Verification

go build ./...
go vet ./...
go test ./...                     # repeated, clean
racedetector test ...             # mesh + replication subset, clean
golangci-lint run                 # same issue count as main (all pre-existing)

Explicitly not in this PR

Consumer-state replication and mesh-wide lease arbitration (phase 4), and the multi-node acceptance tests and benchmarks (phase 5). A leader outage still stops writes to its streams; reads keep working everywhere.

Implements phase 3 of `plan.md`: every node holds the leader's records, at the leader's sequence numbers and timestamps, and follows the leader's retained window. A stream's writes now reach any node through its **command log**, the hidden companion stream `internal-<name>` that only the leader writes. ### What changed - **The command log is the write-ahead log.** A publish appends and fsyncs `Append{seq, timestamp, dedupID, data}` first and only then applies the record to the leader's own copy, through the same applier a replica uses (`streamState.applyRecordLocked`). The publish path therefore has one durable write on it: a crash before it loses both copies and the publish was never acked; a crash after it is repaired. - **Startup replay.** `Server.catchUp` replays a led stream's own command log from its own apply position. It runs when a led stream becomes resident and before every publish reserves a sequence number, so a later publish cannot reuse a sequence number a durable command already describes with different data. - **Strict applier.** `memlog.applyRecord` stores a given `seq` and `timestamp` instead of minting them, refuses a gap, and drops a repeat (a command redelivered after a lost ack). This keeps every timestamp in a stream from the leader's clock. - **Replica loop.** Each replica registers its own consumer (`rep-<peerID>`) on the leader's command log, keeps one command in flight, and applies commands in order, so a replica stores the leader's records verbatim. Each loop has its own `*Conn`: the server dispatches one application stream's frames sequentially, so a blocking pull on the shared peer connection would head-of-line-block all reconciliation and forwarding for that peer. - **Retention follows the leader.** `Trim{base}` is published from a reconciliation pass when the leader's window moves, and a replica applies `max(local base, base)`. A replica runs no age-trim loop of its own, so two nodes cannot trim to different windows; only its own byte budget can still drop its oldest segments. - **Resync.** A replica that fell below the command log's retained base (the byte budget dropped the commands for records it still needs) discards its copy, restarts at the leader's retained base, and backfills, instead of stalling on a gap. The retained base now travels on the `frameGetStream` reply, alongside the retention window a client already reads there. - **A replica enforces the node's storage budget** as its copy grows, exactly as a publish does on the leader; without that its storage would be bounded only by the leader's window. Two supporting changes: - The relay path now accepts `framePull` and `frameAck` **for a reserved internal name only**, so a replica can read the leader's command log while a client's pull is still served locally and never relayed. - **`Close` now stops the server serving.** Inbound stream handlers ran on the `NewServer` context, which `Close` does not cancel, so a torn-down server kept answering peers on streams they had already negotiated — with a nil store. A node "restarted" on the same libp2p host was a zombie behind a live connection, which is what made the leader-restart and replica-restart cases fail. Cancelling a server-owned context at `Close` fixes it. ### Tests (`replication_test.go`, `memlog_test.go`) Backfill onto an empty replica, serving those records from the replica itself, restart mid-stream, idempotent re-application, gap rejection at the applier, leader trims reaching replicas, a stale trim not regressing the base, resync onto the leader's window (including recovery of the resynced copy, whose segment base is not a multiple of the segment size), the crash window between command append and apply, and a replica's copy being evictable between batches. Two phase-1 tests whose "the non-leader applies nothing" assertion phase 3 invalidates were updated to assert convergence instead. ### Verification ``` go build ./... go vet ./... go test ./... # repeated, clean racedetector test ... # mesh + replication subset, clean golangci-lint run # same issue count as main (all pre-existing) ``` ### Explicitly not in this PR Consumer-state replication and mesh-wide lease arbitration (phase 4), and the multi-node acceptance tests and benchmarks (phase 5). A leader outage still stops writes to its streams; reads keep working everywhere.
feat: replicate records through a per-stream command log (phase 3)
All checks were successful
code-review / Agent review (pull_request) Successful in 9s
e096f9234d
Implements phase 3 of plan.md: every node holds the leader's records, at the
leader's sequence numbers and timestamps, and follows the leader's retained
window.

A stream's writes now reach any node through its command log, the hidden
companion stream internal-<name> that only the leader writes:

- A publish appends and fsyncs Append{seq, timestamp, dedupID, data} to the
  command log first and only then applies the record to the leader's own copy,
  through the same applier a replica uses (streamState.applyRecordLocked), so
  the command log is the stream's write-ahead log and the publish path has one
  durable write on it. A crash between the two is repaired by replaying the
  leader's own command log from its own apply position (Server.catchUp), which
  runs when a led stream becomes resident and before every publish reserves a
  sequence number: without it a later publish would reuse the sequence number a
  durable command already describes with different data.
- memlog.applyRecord appends at a given sequence number instead of minting one,
  refuses a gap, and reports a repeat (a command redelivered after a lost ack)
  without storing it.
- A replica tails the log with its own consumer, one command in flight, and
  applies each command in order. Its window is the leader's, so it runs no
  age-trim loop and trims only on a Trim command.
- Retention travels as Trim{base}, which the leader publishes from a
  reconciliation pass when its window moves, and which a replica applies as
  max(local base, base).
- A replica that fell below the command log's retained base resyncs: it
  restarts its log at the leader's retained base and backfills from there,
  rather than stalling on a gap. The retained base now travels on the
  frameGetStream reply, which a client already reads for the retention window.
- A replica enforces the node's storage budget as its copy grows, exactly as a
  publish does on the leader.

Two supporting changes: the relay path accepts framePull and frameAck for a
reserved internal name only (a client's pull is still served locally), and
Close now stops the server serving, so a peer's already-negotiated stream
cannot keep being answered by a torn-down server — a "restarted" node on the
same host is otherwise a zombie behind a live connection.

Tests cover backfill, serving from the replica, restart mid-stream, idempotent
re-application, gap rejection, leader trims reaching replicas, a stale trim
not regressing the base, resync with recovery of its own segment layout,
the crash window between command and apply, and evictability between batches.
pi-reviewer left a comment

Reviewed PR #8 (phase 3, per-stream command-log replication) as pi-reviewer against head e096f92. Verified in this session: go build ./... and go vet ./... clean; racedetector test on the replication/mesh subset clean; golangci-lint run = 58 issues, all present on main (the new replicate.go code is lint-clean); go.mod/go.sum unchanged — no new dependencies.

The design is careful and the test suite is strong (backfill, replica restart, idempotent apply, crash window, resync, evictability). Three IMPORTANT findings in the replication core, all condition-gated — none reproduced as a hard data-loss bug, hence no BLOCKING tag:

  1. 🟠 applyAppend byte accounting inflates cacheTotal on redelivered commands and after resyncs (reproduced with a test).
  2. 🟠 Resync targets the user stream's base while commands are re-supplied from the command log; when the command-log base has advanced past the user base (lagging leader consumer + budget/age trims), a restarted or resyncing replica hits a terminal gap and never recovers — the R=N convergence claim breaks (reproduced arithmetic).
  3. 🟠 replicateLoop's terminal exits leave the stream's entry in mesh.loops; after an ErrConsumerBusy return the loop is never restarted, so replication for that stream stays dead until the node restarts.

Plus four suggestions (catchUp error swallowed at acquire, startReplicaLoop spawns without a ctx check while stopLoops waits, decodeCommand masking truncation errors, cold led streams not publishing trims after budget eviction).

Requesting changes: findings 1–3 deserve fixes before merge; the rest are optional.

Reviewed PR #8 (phase 3, per-stream command-log replication) as pi-reviewer against head e096f92. Verified in this session: `go build ./...` and `go vet ./...` clean; `racedetector test` on the replication/mesh subset clean; `golangci-lint run` = 58 issues, all present on main (the new replicate.go code is lint-clean); go.mod/go.sum unchanged — no new dependencies. The design is careful and the test suite is strong (backfill, replica restart, idempotent apply, crash window, resync, evictability). Three IMPORTANT findings in the replication core, all condition-gated — none reproduced as a hard data-loss bug, hence no BLOCKING tag: 1. 🟠 applyAppend byte accounting inflates cacheTotal on redelivered commands and after resyncs (reproduced with a test). 2. 🟠 Resync targets the user stream's base while commands are re-supplied from the command log; when the command-log base has advanced past the user base (lagging leader consumer + budget/age trims), a restarted or resyncing replica hits a terminal gap and never recovers — the R=N convergence claim breaks (reproduced arithmetic). 3. 🟠 replicateLoop's terminal exits leave the stream's entry in mesh.loops; after an ErrConsumerBusy return the loop is never restarted, so replication for that stream stays dead until the node restarts. Plus four suggestions (catchUp error swallowed at acquire, startReplicaLoop spawns without a ctx check while stopLoops waits, decodeCommand masking truncation errors, cold led streams not publishing trims after budget eviction). Requesting changes: findings 1–3 deserve fixes before merge; the rest are optional.
@ -12,0 +89,4 @@
}
case commandTrim:
cmd.base = reader.readUint64()
default:
Member

🟡 SUGGESTION — decodeCommand masks the real error when the payload is truncated at the kind byte: readByte sets reader.err (io.ErrUnexpectedEOF) but the switch falls into default and reports "command kind 0", which is misleading during corruption triage. Check reader.err (or call reader.done()) before the kind switch so the error names the actual cause.

🟡 SUGGESTION — decodeCommand masks the real error when the payload is truncated at the kind byte: readByte sets reader.err (io.ErrUnexpectedEOF) but the switch falls into default and reports "command kind 0", which is misleading during corruption triage. Check reader.err (or call reader.done()) before the kind switch so the error names the actual cause.
@ -489,0 +744,4 @@
base := uint64(0)
if rec.seq > state.nextSeq() {
_, leaderBase, err := conn.streamInfo(ctx, name)
Member

🟠 IMPORTANT — resync target (user base) vs. resupply source (command log) can diverge and strand a replica terminally. streamInfo here reads only the USER stream's retained base, but the commands that would backfill arrive from the command log, whose retained base is independent: it advances with the replicas' acks and the byte budget, while the user base is pinned by the leader's own consumers. When the command log has trimmed ahead of the user stream (a leader consumer lagging behind fast replicas, then a budget/age trim), a replica below the command-log base resets to the user base and the very next command gaps: applyReplicated(rec{seq:10}, base:3) with next=5 → reset skipped (base ≤ next) → errReplicationGap → replicateLoop returns permanently ("replication cannot close a gap"). The replica then serves only its stale copy, missing records the leader still retains — the R=N convergence claim breaks. The resync test only exercises the in-step case (both windows trimmed to 12). Reproduced the arithmetic: go test -run TestReviewResyncToUserBaseGapsAgainstCommandLog → "gap confirmed: replication gap: record 10, next 5". Fix: resync to max(userBase, commandLogBase) — read the command log's own base via streamInfo on the internal name — or enforce the in-step invariant in production (never trim the command log past the user stream's base) rather than only in the test helper trimCommandLogTo.

🟠 IMPORTANT — resync target (user base) vs. resupply source (command log) can diverge and strand a replica terminally. streamInfo here reads only the USER stream's retained base, but the commands that would backfill arrive from the command log, whose retained base is independent: it advances with the replicas' acks and the byte budget, while the user base is pinned by the leader's own consumers. When the command log has trimmed ahead of the user stream (a leader consumer lagging behind fast replicas, then a budget/age trim), a replica below the command-log base resets to the user base and the very next command gaps: applyReplicated(rec{seq:10}, base:3) with next=5 → reset skipped (base ≤ next) → errReplicationGap → replicateLoop returns permanently ("replication cannot close a gap"). The replica then serves only its stale copy, missing records the leader still retains — the R=N convergence claim breaks. The resync test only exercises the in-step case (both windows trimmed to 12). Reproduced the arithmetic: `go test -run TestReviewResyncToUserBaseGapsAgainstCommandLog` → "gap confirmed: replication gap: record 10, next 5". Fix: resync to max(userBase, commandLogBase) — read the command log's own base via streamInfo on the internal name — or enforce the in-step invariant in production (never trim the command log past the user stream's base) rather than only in the test helper trimCommandLogTo.
replicate.go Outdated
@ -489,0 +759,4 @@
// A replica's copy grows with every applied record, so it enforces the node's
// storage budget exactly as a publish does on the leader: without this a
// replica would bound its storage only by the leader's window.
s.cacheTotal.Add(rec.retainedBytes())
Member

🟠 IMPORTANT — cacheTotal double-counts bytes on redelivery and after resync. applyAppend adds rec.retainedBytes() to cacheTotal unconditionally, but applyReplicated drops a repeat silently (errReplicationRepeat → nil, nothing stored) and a resync (resetToLocked) discards the previous copy without deducting its bytes. Every redelivered command (ack lost → same Append pulled again) inflates the cache even though nothing was stored, and every resync leaves the discarded copy's bytes in the cache. cacheTotal is an upper-bound cache that heals on the next enforcement scan, so no real data is evicted — but once the phantom drift crosses storageBytes, the fast path (cacheTotal.Load() <= s.storageBytes) fails permanently and every publish pays a full totalRetainedBytesLocked scan (in-memory) or a budgetSnapshot disk sweep (durable). Reproduced: go test -run TestReviewApplyAppendRepeatByteAccounting → "repeat apply added 13 to cacheTotal, want 0 (nothing was stored)"; resync case: cacheTotal grew 12→16 across a reset that discarded 3 records and stored 1. Fix: have applyReplicated report whether the record was stored and add bytes only then; on reset, subtract the discarded copy's bytes (resetToLocked can return the freed byte count).

🟠 IMPORTANT — cacheTotal double-counts bytes on redelivery and after resync. applyAppend adds rec.retainedBytes() to cacheTotal unconditionally, but applyReplicated drops a repeat silently (errReplicationRepeat → nil, nothing stored) and a resync (resetToLocked) discards the previous copy without deducting its bytes. Every redelivered command (ack lost → same Append pulled again) inflates the cache even though nothing was stored, and every resync leaves the discarded copy's bytes in the cache. cacheTotal is an upper-bound cache that heals on the next enforcement scan, so no real data is evicted — but once the phantom drift crosses storageBytes, the fast path (`cacheTotal.Load() <= s.storageBytes`) fails permanently and every publish pays a full `totalRetainedBytesLocked` scan (in-memory) or a `budgetSnapshot` disk sweep (durable). Reproduced: `go test -run TestReviewApplyAppendRepeatByteAccounting` → "repeat apply added 13 to cacheTotal, want 0 (nothing was stored)"; resync case: cacheTotal grew 12→16 across a reset that discarded 3 records and stored 1. Fix: have applyReplicated report whether the record was stored and add bytes only then; on reset, subtract the discarded copy's bytes (resetToLocked can return the freed byte count).
replicate.go Outdated
@ -489,0 +838,4 @@
// consumer or the command log is recreated by reconciliation. A
// broken connection is replaced rather than reused: its failure fails
// every later request.
if ctx.Err() != nil || errors.Is(err, ErrConsumerBusy) {
Member

🟠 IMPORTANT — a terminal ErrConsumerBusy return leaves a stale mesh.loops entry, so the replication loop is never restarted. replicateLoop returns on ErrConsumerBusy, but startReplicaLoop's entry in mesh.loops is only removed by stopLoop/stopLoops, never by the loop's own exit; every later pass sees "running" and skips, so replication for that stream is dead until the node restarts. ErrConsumerBusy is reachable: deleteLocalStream → stopReplicaLoop cancels the old loop (whose last pull refreshed the leader-side consumer lease) without releasing it; a re-adoption within consumerLeaseTimeout (30s) — e.g. a listing gap during a leader restart, then a re-listing — starts a new loop whose first pull gets ErrConsumerBusy and dies permanently. The comment's assumption ("another loop of this node took the consumer over, which a restart of this loop would fight") is inverted: the map entry prevents exactly the replacement it assumes exists. Fix: remove the map entry when the loop exits (defer a delete under loopsLock), and treat ErrConsumerBusy as retryable (sleep and re-pull; the lease expires in 30s) so a replacement loop can take over.

🟠 IMPORTANT — a terminal ErrConsumerBusy return leaves a stale mesh.loops entry, so the replication loop is never restarted. replicateLoop returns on ErrConsumerBusy, but startReplicaLoop's entry in mesh.loops is only removed by stopLoop/stopLoops, never by the loop's own exit; every later pass sees "running" and skips, so replication for that stream is dead until the node restarts. ErrConsumerBusy is reachable: deleteLocalStream → stopReplicaLoop cancels the old loop (whose last pull refreshed the leader-side consumer lease) without releasing it; a re-adoption within consumerLeaseTimeout (30s) — e.g. a listing gap during a leader restart, then a re-listing — starts a new loop whose first pull gets ErrConsumerBusy and dies permanently. The comment's assumption ("another loop of this node took the consumer over, which a restart of this loop would fight") is inverted: the map entry prevents exactly the replacement it assumes exists. Fix: remove the map entry when the loop exits (defer a delete under loopsLock), and treat ErrConsumerBusy as retryable (sleep and re-pull; the lease expires in 30s) so a replacement loop can take over.
@ -489,0 +989,4 @@
loopCtx, stop := context.WithCancel(ctx)
mesh.loops[name] = stop
mesh.loopsLock.Unlock()
Member

🟡 SUGGESTION — startReplicaLoop spawns even when ctx is already cancelled. Close's wait for a reconciliation pass in flight is bounded (meshShutdownTimeout = 2s), while a pass's transport write can block up to defaultWriteTimeout = 30s (writeRequest falls back to it when the caller's ctx has no deadline). A pass that outlives the bounded wait can call startReplicaLoop after stopLoops cleared the loops map and started loopsDone.Wait: the spawned loop exits immediately (cancelled ctx), but the loopsDone.Go Add can race the Wait. Cheap fix: if ctx.Err() != nil { return } at the top of startReplicaLoop.

🟡 SUGGESTION — startReplicaLoop spawns even when ctx is already cancelled. Close's wait for a reconciliation pass in flight is bounded (meshShutdownTimeout = 2s), while a pass's transport write can block up to defaultWriteTimeout = 30s (writeRequest falls back to it when the caller's ctx has no deadline). A pass that outlives the bounded wait can call startReplicaLoop after stopLoops cleared the loops map and started loopsDone.Wait: the spawned loop exits immediately (cancelled ctx), but the loopsDone.Go Add can race the Wait. Cheap fix: `if ctx.Err() != nil { return }` at the top of startReplicaLoop.
@ -489,0 +1097,4 @@
// was last written, and touching it would load it just to read a window. The
// snapshot is taken under the server lock and published outside it, because
// appendCommand takes the same lock to look the command log up.
func (s *Server) publishResidentTrims(ctx context.Context) {
Member

🟡 SUGGESTION — publishResidentTrims skips cold led streams (rs.stream == nil), but a cold stream's base can still advance on disk via the byte budget (evictColdDurableToBudget deletes its oldest segments). No Trim command reaches replicas until the stream is next loaded, so replicas keep serving records the leader has already dropped in the interim — the R=N retention promise is briefly false, self-healing on next load. Consider publishing the trim when the budget evicts a cold led stream.

🟡 SUGGESTION — publishResidentTrims skips cold led streams (rs.stream == nil), but a cold stream's base can still advance on disk via the byte budget (evictColdDurableToBudget deletes its oldest segments). No Trim command reaches replicas until the stream is next loaded, so replicas keep serving records the leader has already dropped in the interim — the R=N retention promise is briefly false, self-healing on next load. Consider publishing the trim when the budget evicts a cold led stream.
server.go Outdated
@ -211,0 +231,4 @@
if s.mesh != nil && !isInternalName(name) && s.leadsLocal(name) {
stream.writeLock.Lock()
err := s.catchUp(ctx, stream, stream.nextSeq())
Member

🟡 SUGGESTION — a catchUp failure at first acquire is logged and then swallowed: the stream is served anyway, so reads miss records that are durable in the command log and already on every replica. The same corrupt command log then fails every later publish (publishReplicated → catchUp → error) while reads silently serve stale data. Consider returning the error from acquire so the open fails loudly, or marking the stream read-only, instead of serving a copy that diverges.

🟡 SUGGESTION — a catchUp failure at first acquire is logged and then swallowed: the stream is served anyway, so reads miss records that are durable in the command log and already on every replica. The same corrupt command log then fails every later publish (publishReplicated → catchUp → error) while reads silently serve stale data. Consider returning the error from acquire so the open fails loudly, or marking the stream read-only, instead of serving a copy that diverges.
Resolves the review findings on the phase-3 command-log replication, all of
which were reachable only under redelivery, resync or restart.

Byte accounting (finding 1): the byte-budget cache counted every applied
record and never corrected itself. A redelivered command added the record's
bytes a second time, and a resync added the new record without releasing the
copy it discarded, so the cache drifted above the node's real bytes and the
under-budget fast path failed permanently — turning every later publish into
a full scan or disk sweep. applyRecordLocked now reports whether it stored
the record, applyReplicated reports what a resync discarded, applyTrim
reports what a Trim discarded, and the server's cache only grows by records
it keeps and shrinks by what it drops. A replay in catchUp accounts for the
record too, since the publish that wrote the command never reached its own
accounting.

Command-log retention (finding 2): the log was held only by the replicas'
cursors, so a replica that kept up trimmed it to its own position and a
replica rebuilding from scratch had nothing to backfill from. The leader now
keeps a floor on each log it leads: the position of the first command for a
record the stream still holds, recomputed from the log itself on every
reconciliation pass and at load, and applied as a cap on the trim. It is
in-memory by design: a trim that has not dropped whole segments comes back
with the next restart, so a remembered position can end up past a reloaded
log's end and hold nothing at all.

Replication loop lifetime (finding 3): a loop that exited left its mesh entry
behind and treated ErrConsumerBusy as terminal, so replication for that
stream stayed dead until the node restarted. Each loop now removes its own
entry (only while it is still its own, so it cannot remove a replacement),
retries a busy lease instead of stopping, releases the consumer lease on the
way out, and skips starting when the server is already shutting down.

Also: a stream that fails to replay its command log at load now fails the
load instead of serving a copy that is missing durable records; and a
resync onto a stream that has never appended no longer fails on the missing
log directory.

Skipped: the durable consumer store's minimum-watermark scan still treats a
live watermark of zero as "unset", so its result depends on map order and
disagrees with the in-memory path. It is a pre-existing inconsistency unrelated
to these findings, and changing it changes durable retention for every stream.
Both are the finding-3 liveness fix's observable halves: a loop that meets a
busy lease keeps retrying and delivers the next record once the lease frees
(before the fix it returned and the stream replicated nothing until the node
restarted), and a returning loop drops its own mesh entry only while the
entry is still its own (a stop-and-restart can install a replacement while
the outgoing loop is unwinding).
Author
Member

Reviewed findings 1–3 and the four suggestions at head e096f92; resolved in d397824 + a459e8f. Nothing in the review was dismissed, and one suggestion (the pin) turned out to need a different mechanism than I first built — details under finding 2.

Verification at the new head a459e8f

  • go build ./..., go vet ./..., gofmt -l . clean.
  • go test ./... clean, 5 consecutive runs plus a 25× stress loop over the replication/crash/decommission tests (the crash-window test is the one these fixes are about, so I hammered it).
  • racedetector test on the mesh/replication subset clean (no WARNING: DATA RACE).
  • golangci-lint run --fix=false = 81 issues, and the per-file/per-line set is identical to main's 81 — I verified that by adding a detached main worktree and diffing file:line: rule lists. (I used --fix=false because the repo config has issues.fix: true, which rewrites unrelated files.) Your 58 was presumably the default-config run; the comparison that matters is no-new-findings, and that holds.
  • go.mod/go.sum untouched.

1. 🟠 cacheTotal inflates on redelivery and resync

Fixed by making the accounting follow what was actually stored. applyRecordLocked now reports whether it stored the record ((bool, error)), applyReplicated reports what a resync discarded, applyTrim and journal.resetToLocked report what a trim/reset dropped, and Server.releaseCachedBytes lowers the cache (CAS loop, floor 0). The cache only grows by records this node keeps and shrinks by what it drops; a replay in catchUp now accounts too, since the publish that wrote that command never reached its own accounting. A publish whose own record was not stored now reports ErrInternal instead of handing back a sequence number this node never stored.

Covered by TestReplicatedApplyByteAccounting (store → +bytes, redelivery → unchanged, resync → −discarded +new) and TestReplicatedAppendIsIdempotent, which now asserts stored/discarded on both applications.

2. 🟠 resync targets the user base while commands come from a log whose base advanced

Confirmed, and worse than stated: on a durable stream the log is held only by the replicas' cursors, so a replica that kept up trimmed it to its own position, and a replica starting from scratch had nothing left to backfill from — the terminal gap was a symptom of that. Fixed by giving the leader a floor on each log it leads: the position of the first command for a record the stream still holds (the first Append with seq >= streamBase), recomputed from the log on every reconciliation pass and also at load, and applied as a cap on the trim (computeKeepLocked). Resync still targets max(userBase, rec.seq), which remains the right target for a copy that fell behind the user window; TestLeaderPinKeepsTheCommandLogBackfillable wipes a replica's stream directory, restarts the node and asserts it rebuilds exactly [base, next) with the leader's payloads.

Two things worth flagging, because they changed the design:

  • I first implemented the pin the way the review suggested — a leader-owned consumer named leader-pin mirroring rep-<peerID>, so retention (which reads the durable watermark array) would hold the log. That version is wrong in a way that only shows after a restart: a trim that advances the log's in-memory base does not necessarily delete whole segments, so the log comes back from disk with a lower base while the pin's persisted position does not — the pin then sits at or past the log's end and pins nothing, and a replica's redelivered acks can empty the log before the first pass. I chased this as a flaky TestCrashBetweenCommandAndApplyConverges (it reproduced ~1/40 runs) and it is exactly what the reviewer's "lagging leader consumer + budget/age trims" window exposes. The floor is therefore deliberately in-memory: recomputing it from the log's own base on every pass cannot go stale against the log it describes, and the load-time floor (pinFloor(ctx, stream, 0) in reconcileLoadedStream's command-log branch) closes the window between a restart and the first pass.
  • While implementing this I found a latent inconsistency in the durable consumer store: minLiveWatermark treats a live watermark of 0 as "unset", so with a mix of never-acked and acked consumers its result depends on map order, and it disagrees with the in-memory path (which lets a zero cursor pin). It is pre-existing and independent of these findings, and changing it changes durable retention for every stream, so I left it alone — flagging it here rather than fixing it silently.

3. 🟠 replicateLoop's terminal exits leave the mesh entry / ErrConsumerBusy is fatal

Fixed: mesh.loops now holds *replicaRun{stop} and a returning loop deletes its entry through forgetLoop only while the entry is still its own (a stop-and-restart can install a replacement mid-unwind); ErrConsumerBusy is retryable like any other pull failure instead of terminal; startReplicaLoop returns early when the server is already shutting down (your suggestion 5); and the loop releases its consume lease on the way out via frameConsumeRelease on a context.WithoutCancel context, so a replacement acquires the consumer immediately rather than waiting out the lease timeout.

Covered by TestReplicationSurvivesABusyLease (a foreign token holds the lease, the loop must keep retrying and deliver the next record once it frees — before this change it returned and that stream replicated nothing until the node restarted) and TestForgetLoopRemovesOnlyItsOwnEntry.

Suggestions

  • acquire swallowing a catchUp error: fixed via Server.loadStream + reconcileLoadedStream. A replay failure now fails the load (wrapped as replaying %q's command log: %w), the stream is closed, and the resident entry is dropped only if it is still this load's — mirroring the existing open-failure path.
  • decodeCommand masking truncation: fixed; the reader error is checked before the kind switch, so a payload truncated at the kind byte reports the truncation instead of "command kind 0".
  • Cold led streams after a budget eviction: fixed. evictColdDurableToBudget marks the stream (Server.markEvicted, which ignores internal names and streams this node does not lead, and is a no-op without a mesh), and the pass publishes the trim for marked streams using Server.streamBase without loading them. The under-lock path (evictResidentDurableToBudgetLocked) deliberately does not publish.

Two extra fixes the work surfaced

  • journal.resetToLocked failed on a log directory that does not exist yet, so the first thing that ever happened to a freshly created replica could not be a resync ("opening log directory: ... no such file or directory"). A missing directory now means "nothing to sync"; the next append creates it and that roll syncs it.
  • TestDecommissionRemovesTheReplicaConsumer was asserting base == cursor+1 after the replica's ack. That is no longer the log's contract: the floor holds the window at the user stream's retained window, so the ack releases the replica's share without moving the base. The test now asserts the three acked records are all still retained and that the watermark advanced, plus that no consumer of the replica's survives the decommission.

One pre-existing flake, unrelated to this PR, shows up under full-suite load: TestConsumeSurvivesEviction ("idle stream was not evicted") fails ~1/40 runs because it cancels a consume loop and immediately asks for eviction without waiting for the loop to drop its blocked pull or lease. I left it alone as out of scope; say the word and I will make it deterministic.

Reviewed findings 1–3 and the four suggestions at head e096f92; resolved in `d397824` + `a459e8f`. Nothing in the review was dismissed, and one suggestion (the pin) turned out to need a different mechanism than I first built — details under finding 2. **Verification at the new head a459e8f** - `go build ./...`, `go vet ./...`, `gofmt -l .` clean. - `go test ./...` clean, 5 consecutive runs plus a 25× stress loop over the replication/crash/decommission tests (the crash-window test is the one these fixes are about, so I hammered it). - `racedetector test` on the mesh/replication subset clean (no `WARNING: DATA RACE`). - `golangci-lint run --fix=false` = 81 issues, and the per-file/per-line set is *identical* to `main`'s 81 — I verified that by adding a detached `main` worktree and diffing `file:line: rule` lists. (I used `--fix=false` because the repo config has `issues.fix: true`, which rewrites unrelated files.) Your 58 was presumably the default-config run; the comparison that matters is no-new-findings, and that holds. - `go.mod`/`go.sum` untouched. **1. 🟠 cacheTotal inflates on redelivery and resync** Fixed by making the accounting follow what was actually stored. `applyRecordLocked` now reports whether it stored the record (`(bool, error)`), `applyReplicated` reports what a resync discarded, `applyTrim` and `journal.resetToLocked` report what a trim/reset dropped, and `Server.releaseCachedBytes` lowers the cache (CAS loop, floor 0). The cache only grows by records this node keeps and shrinks by what it drops; a replay in `catchUp` now accounts too, since the publish that wrote that command never reached its own accounting. A publish whose own record was not stored now reports `ErrInternal` instead of handing back a sequence number this node never stored. Covered by `TestReplicatedApplyByteAccounting` (store → +bytes, redelivery → unchanged, resync → −discarded +new) and `TestReplicatedAppendIsIdempotent`, which now asserts `stored`/`discarded` on both applications. **2. 🟠 resync targets the user base while commands come from a log whose base advanced** Confirmed, and worse than stated: on a durable stream the log is held only by the replicas' cursors, so a replica that kept up trimmed it to its own position, and a replica starting from scratch had nothing left to backfill from — the terminal gap was a symptom of that. Fixed by giving the leader a **floor on each log it leads**: the position of the first command for a record the stream still holds (the first `Append` with `seq >= streamBase`), recomputed from the log on every reconciliation pass and also at load, and applied as a cap on the trim (`computeKeepLocked`). Resync still targets `max(userBase, rec.seq)`, which remains the right target for a copy that fell behind the *user* window; `TestLeaderPinKeepsTheCommandLogBackfillable` wipes a replica's stream directory, restarts the node and asserts it rebuilds exactly `[base, next)` with the leader's payloads. Two things worth flagging, because they changed the design: - I first implemented the pin the way the review suggested — a leader-owned consumer named `leader-pin` mirroring `rep-<peerID>`, so retention (which reads the durable watermark array) would hold the log. That version is wrong in a way that only shows after a restart: a trim that advances the log's in-memory base does not necessarily delete whole segments, so the log comes back from disk with a *lower* base while the pin's persisted position does not — the pin then sits at or past the log's end and pins nothing, and a replica's redelivered acks can empty the log before the first pass. I chased this as a flaky `TestCrashBetweenCommandAndApplyConverges` (it reproduced ~1/40 runs) and it is exactly what the reviewer's "lagging leader consumer + budget/age trims" window exposes. The floor is therefore deliberately **in-memory**: recomputing it from the log's own base on every pass cannot go stale against the log it describes, and the load-time floor (`pinFloor(ctx, stream, 0)` in `reconcileLoadedStream`'s command-log branch) closes the window between a restart and the first pass. - While implementing this I found a latent inconsistency in the durable consumer store: `minLiveWatermark` treats a live watermark of **0** as "unset", so with a mix of never-acked and acked consumers its result depends on map order, and it disagrees with the in-memory path (which lets a zero cursor pin). It is pre-existing and independent of these findings, and changing it changes durable retention for every stream, so I left it alone — flagging it here rather than fixing it silently. **3. 🟠 replicateLoop's terminal exits leave the mesh entry / ErrConsumerBusy is fatal** Fixed: `mesh.loops` now holds `*replicaRun{stop}` and a returning loop deletes its entry through `forgetLoop` only while the entry is still its own (a stop-and-restart can install a replacement mid-unwind); `ErrConsumerBusy` is retryable like any other pull failure instead of terminal; `startReplicaLoop` returns early when the server is already shutting down (your suggestion 5); and the loop releases its consume lease on the way out via `frameConsumeRelease` on a `context.WithoutCancel` context, so a replacement acquires the consumer immediately rather than waiting out the lease timeout. Covered by `TestReplicationSurvivesABusyLease` (a foreign token holds the lease, the loop must keep retrying and deliver the next record once it frees — before this change it returned and that stream replicated nothing until the node restarted) and `TestForgetLoopRemovesOnlyItsOwnEntry`. **Suggestions** - `acquire` swallowing a `catchUp` error: fixed via `Server.loadStream` + `reconcileLoadedStream`. A replay failure now fails the load (wrapped as `replaying %q's command log: %w`), the stream is closed, and the resident entry is dropped only if it is still this load's — mirroring the existing open-failure path. - `decodeCommand` masking truncation: fixed; the reader error is checked before the kind switch, so a payload truncated at the kind byte reports the truncation instead of "command kind 0". - Cold led streams after a budget eviction: fixed. `evictColdDurableToBudget` marks the stream (`Server.markEvicted`, which ignores internal names and streams this node does not lead, and is a no-op without a mesh), and the pass publishes the trim for marked streams using `Server.streamBase` without loading them. The under-lock path (`evictResidentDurableToBudgetLocked`) deliberately does not publish. **Two extra fixes the work surfaced** - `journal.resetToLocked` failed on a log directory that does not exist yet, so the *first* thing that ever happened to a freshly created replica could not be a resync ("opening log directory: ... no such file or directory"). A missing directory now means "nothing to sync"; the next append creates it and that roll syncs it. - `TestDecommissionRemovesTheReplicaConsumer` was asserting `base == cursor+1` after the replica's ack. That is no longer the log's contract: the floor holds the window at the user stream's retained window, so the ack releases the replica's share without moving the base. The test now asserts the three acked records are all still retained *and* that the watermark advanced, plus that no consumer of the replica's survives the decommission. One pre-existing flake, unrelated to this PR, shows up under full-suite load: `TestConsumeSurvivesEviction` ("idle stream was not evicted") fails ~1/40 runs because it cancels a consume loop and immediately asks for eviction without waiting for the loop to drop its blocked pull or lease. I left it alone as out of scope; say the word and I will make it deterministic.
The test cancelled the consume loop and immediately asked for eviction, but
cancellation is not synchronous: the loop unwinds on its own goroutine (its
blocked pull returns, its lease is released), and an eviction attempted while
it is still unwinding is correctly refused. Under full-suite load the race
lost ~1/40 runs with "idle stream was not evicted".

The test now waits for the stream to become evictable (the exact predicate
evictIdleStreams applies, evaluated under the server lock) before publishing
and evicting. The resume wait is hardened too: it polled the resident entry
but not its load, so a loading placeholder (nil stream) could panic, and its
fixed five-second budget could trip under a race-instrumented or loaded run
— it now waits on a loaded stream with an explicit longer deadline
(waitForTimeout, a generalisation of the shared waitFor helper).
Author
Member

Follow-up on the flaky TestConsumeSurvivesEviction you asked me to take ownership of — fixed at ae55b7e.

Root cause: the test cancelled the consume loop and immediately asked for eviction, but cancellation is not synchronous. The loop unwinds on its own goroutine — its blocked pull returns and its lease is released in a defer — and evictIdleStreams(0) refuses to evict a stream that is still evictionBlocked() (any blocked pull or live lease). Under full-suite load the test lost that race ~1/40 runs with "idle stream was not evicted". The production semantics are correct (a stream whose loop is still unwinding must not be evicted), so the fix belongs in the test.

What changed:

  • The test now waits for the stream to become evictable — rs.idleEvictable(0), the exact predicate evictIdleStreams applies — evaluated under the server lock, before publishing and evicting.
  • While proving this under racedetector I found a second latent bug in the same test: the resume wait polled the resident entry but not its load, so a loading placeholder (nil rs.stream) could panic, and its fixed five-second budget could trip on a race-instrumented or heavily loaded run. The resume wait now polls a loaded stream and uses an explicit longer deadline via a new waitForTimeout helper (a generalisation of the shared waitFor; the default five-second callers are unchanged).

Verification at ae55b7e: 40/40 isolation runs of the test, 4× full suite, 8× racedetector runs of the test, racedetector on the mesh/replication subset — all clean; golangci-lint run --fix=false still exactly the main baseline (81 issues, nothing in the edited files).

One adjacent, pre-existing production defect surfaced but left alone as out of scope: the client's pooled wireWriter (getWireWriter/sync.Pool) can hand the same writer to concurrent requests on different streams, which the race detector flags (rpc_stream.go) and which can corrupt a request frame and force a retry. It is part of the known full-suite racedetector baseline on main, not introduced by this PR, and fixing it touches the client request path used everywhere — happy to do that as its own change if you want it.

Follow-up on the flaky `TestConsumeSurvivesEviction` you asked me to take ownership of — fixed at `ae55b7e`. **Root cause**: the test cancelled the consume loop and immediately asked for eviction, but cancellation is not synchronous. The loop unwinds on its own goroutine — its blocked pull returns and its lease is released in a defer — and `evictIdleStreams(0)` refuses to evict a stream that is still `evictionBlocked()` (any blocked pull or live lease). Under full-suite load the test lost that race ~1/40 runs with "idle stream was not evicted". The production semantics are correct (a stream whose loop is still unwinding must not be evicted), so the fix belongs in the test. **What changed**: - The test now waits for the stream to become evictable — `rs.idleEvictable(0)`, the exact predicate `evictIdleStreams` applies — evaluated under the server lock, before publishing and evicting. - While proving this under `racedetector` I found a second latent bug in the same test: the resume wait polled the resident entry but not its load, so a loading placeholder (nil `rs.stream`) could panic, and its fixed five-second budget could trip on a race-instrumented or heavily loaded run. The resume wait now polls a *loaded* stream and uses an explicit longer deadline via a new `waitForTimeout` helper (a generalisation of the shared `waitFor`; the default five-second callers are unchanged). **Verification at `ae55b7e`**: 40/40 isolation runs of the test, 4× full suite, 8× `racedetector` runs of the test, `racedetector` on the mesh/replication subset — all clean; `golangci-lint run --fix=false` still exactly the main baseline (81 issues, nothing in the edited files). One adjacent, pre-existing production defect surfaced but left alone as out of scope: the client's pooled `wireWriter` (`getWireWriter`/sync.Pool) can hand the same writer to concurrent requests on different streams, which the race detector flags (`rpc_stream.go`) and which can corrupt a request frame and force a retry. It is part of the known full-suite `racedetector` baseline on main, not introduced by this PR, and fixing it touches the client request path used everywhere — happy to do that as its own change if you want it.
Author
Member

Correction to my comment above (758): the pooled-wireWriter corruption claim does not hold up. I investigated it as a stacked change and the evidence says there is no live race in the client request path:

  • sync.Pool handoff is exclusive — instrumenting the borrows showed no waiter or writer was ever handed to two live requests across thousands of borrows.
  • The apparent cross-delivery in a stress run was a bad test assertion: the server assigns sequence numbers in arrival order, so publish 350 answered with seq 0 just means 350's request arrived first.
  • The racedetector reports around rpcStream are false positives of the pure-Go detector: it flags r.pending[requestID] = waiter and r.pendingLock.Unlock() (both inside the lock's critical section) and even pairs different heap addresses as racing — the same detector reports acquire's rs.refs++ racing, which is under s.lock.

What the investigation did find worth fixing is the requestWaiter pool — a borrow of the same class that is safe only by fragile ordering (getWaiter mutates the struct with no lock; its lifetime crosses into the reader goroutine's critical section) — plus a corrected concurrency regression test. That landed as a stacked PR: #9 (base = this branch).

Correction to my comment above (758): the pooled-`wireWriter` corruption claim does **not** hold up. I investigated it as a stacked change and the evidence says there is no live race in the client request path: - `sync.Pool` handoff is exclusive — instrumenting the borrows showed no waiter or writer was ever handed to two live requests across thousands of borrows. - The apparent cross-delivery in a stress run was a bad test assertion: the server assigns sequence numbers in **arrival order**, so `publish 350 answered with seq 0` just means 350's request arrived first. - The `racedetector` reports around `rpcStream` are false positives of the pure-Go detector: it flags `r.pending[requestID] = waiter` and `r.pendingLock.Unlock()` (both inside the lock's critical section) and even pairs different heap addresses as racing — the same detector reports `acquire`'s `rs.refs++` racing, which is under `s.lock`. What the investigation did find worth fixing is the `requestWaiter` pool — a borrow of the same class that is safe only by fragile ordering (`getWaiter` mutates the struct with no lock; its lifetime crosses into the reader goroutine's critical section) — plus a corrected concurrency regression test. That landed as a stacked PR: https://git.norobo.chat/norobo/webstream/pulls/9 (base = this branch).
The requestWaiter struct is pooled, but its borrow is sound only while two
orderings hold: the reader goroutine's deliver must notify under pendingLock,
and every pool return must follow the pending-map removal. Nothing enforces
or documents either — getWaiter mutates the struct's fields with no lock, and
the waiter's lifetime crosses into the reader goroutine's critical section —
so any reordering reintroduces a use-after-pool. The pool saves one small
allocation per request; remove it and allocate one waiter per request instead.

The wireWriter pool is untouched: its borrow is strictly synchronous inside
request (the write completes before the put), so it cannot hand the same
buffer to two live borrowers.

While verifying, the corruption hypothesis behind this change was tested and
disproven: instrumenting the borrows showed no waiter was ever handed to two
live requests across thousands of borrows, and the concurrent-publish stress
assertion that appeared to show cross-delivery (publish 350 answered with
seq 0) was wrong — the server assigns sequence numbers in arrival order, so a
response's seq is not the published value. A corrected stress test
(TestRPCStreamConcurrentRequests) now pins the real contract: with concurrent
publishes on the control stream alongside a consumer looping on its pull
stream, every publish is answered exactly once and the response sequence
numbers are exactly 0..N-1 once each.
Reviewed-on: #9
sam merged commit 2b34165eed into main 2026-10-04 07:56:24 +00:00
sam deleted branch feat/mesh-replication-command-log 2026-10-04 07:56:24 +00:00
Sign in to join this conversation.
No reviewers
No labels
code-review
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
norobo/webstream!8
No description provided.