feat: replicate records through a per-stream command log (phase 3) #8
No reviewers
Labels
No labels
code-review
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
norobo/webstream!8
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/mesh-replication-command-log"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Implements phase 3 of
plan.md: every node holds the leader's records, at the leader's sequence numbers and timestamps, and follows the leader's retained window.A stream's writes now reach any node through its command log, the hidden companion stream
internal-<name>that only the leader writes.What changed
Append{seq, timestamp, dedupID, data}first and only then applies the record to the leader's own copy, through the same applier a replica uses (streamState.applyRecordLocked). The publish path therefore has one durable write on it: a crash before it loses both copies and the publish was never acked; a crash after it is repaired.Server.catchUpreplays a led stream's own command log from its own apply position. It runs when a led stream becomes resident and before every publish reserves a sequence number, so a later publish cannot reuse a sequence number a durable command already describes with different data.memlog.applyRecordstores a givenseqandtimestampinstead of minting them, refuses a gap, and drops a repeat (a command redelivered after a lost ack). This keeps every timestamp in a stream from the leader's clock.rep-<peerID>) on the leader's command log, keeps one command in flight, and applies commands in order, so a replica stores the leader's records verbatim. Each loop has its own*Conn: the server dispatches one application stream's frames sequentially, so a blocking pull on the shared peer connection would head-of-line-block all reconciliation and forwarding for that peer.Trim{base}is published from a reconciliation pass when the leader's window moves, and a replica appliesmax(local base, base). A replica runs no age-trim loop of its own, so two nodes cannot trim to different windows; only its own byte budget can still drop its oldest segments.frameGetStreamreply, alongside the retention window a client already reads there.Two supporting changes:
framePullandframeAckfor a reserved internal name only, so a replica can read the leader's command log while a client's pull is still served locally and never relayed.Closenow stops the server serving. Inbound stream handlers ran on theNewServercontext, whichClosedoes not cancel, so a torn-down server kept answering peers on streams they had already negotiated — with a nil store. A node "restarted" on the same libp2p host was a zombie behind a live connection, which is what made the leader-restart and replica-restart cases fail. Cancelling a server-owned context atClosefixes it.Tests (
replication_test.go,memlog_test.go)Backfill onto an empty replica, serving those records from the replica itself, restart mid-stream, idempotent re-application, gap rejection at the applier, leader trims reaching replicas, a stale trim not regressing the base, resync onto the leader's window (including recovery of the resynced copy, whose segment base is not a multiple of the segment size), the crash window between command append and apply, and a replica's copy being evictable between batches. Two phase-1 tests whose "the non-leader applies nothing" assertion phase 3 invalidates were updated to assert convergence instead.
Verification
Explicitly not in this PR
Consumer-state replication and mesh-wide lease arbitration (phase 4), and the multi-node acceptance tests and benchmarks (phase 5). A leader outage still stops writes to its streams; reads keep working everywhere.
Implements phase 3 of plan.md: every node holds the leader's records, at the leader's sequence numbers and timestamps, and follows the leader's retained window. A stream's writes now reach any node through its command log, the hidden companion stream internal-<name> that only the leader writes: - A publish appends and fsyncs Append{seq, timestamp, dedupID, data} to the command log first and only then applies the record to the leader's own copy, through the same applier a replica uses (streamState.applyRecordLocked), so the command log is the stream's write-ahead log and the publish path has one durable write on it. A crash between the two is repaired by replaying the leader's own command log from its own apply position (Server.catchUp), which runs when a led stream becomes resident and before every publish reserves a sequence number: without it a later publish would reuse the sequence number a durable command already describes with different data. - memlog.applyRecord appends at a given sequence number instead of minting one, refuses a gap, and reports a repeat (a command redelivered after a lost ack) without storing it. - A replica tails the log with its own consumer, one command in flight, and applies each command in order. Its window is the leader's, so it runs no age-trim loop and trims only on a Trim command. - Retention travels as Trim{base}, which the leader publishes from a reconciliation pass when its window moves, and which a replica applies as max(local base, base). - A replica that fell below the command log's retained base resyncs: it restarts its log at the leader's retained base and backfills from there, rather than stalling on a gap. The retained base now travels on the frameGetStream reply, which a client already reads for the retention window. - A replica enforces the node's storage budget as its copy grows, exactly as a publish does on the leader. Two supporting changes: the relay path accepts framePull and frameAck for a reserved internal name only (a client's pull is still served locally), and Close now stops the server serving, so a peer's already-negotiated stream cannot keep being answered by a torn-down server — a "restarted" node on the same host is otherwise a zombie behind a live connection. Tests cover backfill, serving from the replica, restart mid-stream, idempotent re-application, gap rejection, leader trims reaching replicas, a stale trim not regressing the base, resync with recovery of its own segment layout, the crash window between command and apply, and evictability between batches.Reviewed PR #8 (phase 3, per-stream command-log replication) as pi-reviewer against head
e096f92. Verified in this session:go build ./...andgo vet ./...clean;racedetector teston the replication/mesh subset clean;golangci-lint run= 58 issues, all present on main (the new replicate.go code is lint-clean); go.mod/go.sum unchanged — no new dependencies.The design is careful and the test suite is strong (backfill, replica restart, idempotent apply, crash window, resync, evictability). Three IMPORTANT findings in the replication core, all condition-gated — none reproduced as a hard data-loss bug, hence no BLOCKING tag:
Plus four suggestions (catchUp error swallowed at acquire, startReplicaLoop spawns without a ctx check while stopLoops waits, decodeCommand masking truncation errors, cold led streams not publishing trims after budget eviction).
Requesting changes: findings 1–3 deserve fixes before merge; the rest are optional.
@ -12,0 +89,4 @@}case commandTrim:cmd.base = reader.readUint64()default:🟡 SUGGESTION — decodeCommand masks the real error when the payload is truncated at the kind byte: readByte sets reader.err (io.ErrUnexpectedEOF) but the switch falls into default and reports "command kind 0", which is misleading during corruption triage. Check reader.err (or call reader.done()) before the kind switch so the error names the actual cause.
@ -489,0 +744,4 @@base := uint64(0)if rec.seq > state.nextSeq() {_, leaderBase, err := conn.streamInfo(ctx, name)🟠 IMPORTANT — resync target (user base) vs. resupply source (command log) can diverge and strand a replica terminally. streamInfo here reads only the USER stream's retained base, but the commands that would backfill arrive from the command log, whose retained base is independent: it advances with the replicas' acks and the byte budget, while the user base is pinned by the leader's own consumers. When the command log has trimmed ahead of the user stream (a leader consumer lagging behind fast replicas, then a budget/age trim), a replica below the command-log base resets to the user base and the very next command gaps: applyReplicated(rec{seq:10}, base:3) with next=5 → reset skipped (base ≤ next) → errReplicationGap → replicateLoop returns permanently ("replication cannot close a gap"). The replica then serves only its stale copy, missing records the leader still retains — the R=N convergence claim breaks. The resync test only exercises the in-step case (both windows trimmed to 12). Reproduced the arithmetic:
go test -run TestReviewResyncToUserBaseGapsAgainstCommandLog→ "gap confirmed: replication gap: record 10, next 5". Fix: resync to max(userBase, commandLogBase) — read the command log's own base via streamInfo on the internal name — or enforce the in-step invariant in production (never trim the command log past the user stream's base) rather than only in the test helper trimCommandLogTo.@ -489,0 +759,4 @@// A replica's copy grows with every applied record, so it enforces the node's// storage budget exactly as a publish does on the leader: without this a// replica would bound its storage only by the leader's window.s.cacheTotal.Add(rec.retainedBytes())🟠 IMPORTANT — cacheTotal double-counts bytes on redelivery and after resync. applyAppend adds rec.retainedBytes() to cacheTotal unconditionally, but applyReplicated drops a repeat silently (errReplicationRepeat → nil, nothing stored) and a resync (resetToLocked) discards the previous copy without deducting its bytes. Every redelivered command (ack lost → same Append pulled again) inflates the cache even though nothing was stored, and every resync leaves the discarded copy's bytes in the cache. cacheTotal is an upper-bound cache that heals on the next enforcement scan, so no real data is evicted — but once the phantom drift crosses storageBytes, the fast path (
cacheTotal.Load() <= s.storageBytes) fails permanently and every publish pays a fulltotalRetainedBytesLockedscan (in-memory) or abudgetSnapshotdisk sweep (durable). Reproduced:go test -run TestReviewApplyAppendRepeatByteAccounting→ "repeat apply added 13 to cacheTotal, want 0 (nothing was stored)"; resync case: cacheTotal grew 12→16 across a reset that discarded 3 records and stored 1. Fix: have applyReplicated report whether the record was stored and add bytes only then; on reset, subtract the discarded copy's bytes (resetToLocked can return the freed byte count).@ -489,0 +838,4 @@// consumer or the command log is recreated by reconciliation. A// broken connection is replaced rather than reused: its failure fails// every later request.if ctx.Err() != nil || errors.Is(err, ErrConsumerBusy) {🟠 IMPORTANT — a terminal ErrConsumerBusy return leaves a stale mesh.loops entry, so the replication loop is never restarted. replicateLoop returns on ErrConsumerBusy, but startReplicaLoop's entry in mesh.loops is only removed by stopLoop/stopLoops, never by the loop's own exit; every later pass sees "running" and skips, so replication for that stream is dead until the node restarts. ErrConsumerBusy is reachable: deleteLocalStream → stopReplicaLoop cancels the old loop (whose last pull refreshed the leader-side consumer lease) without releasing it; a re-adoption within consumerLeaseTimeout (30s) — e.g. a listing gap during a leader restart, then a re-listing — starts a new loop whose first pull gets ErrConsumerBusy and dies permanently. The comment's assumption ("another loop of this node took the consumer over, which a restart of this loop would fight") is inverted: the map entry prevents exactly the replacement it assumes exists. Fix: remove the map entry when the loop exits (defer a delete under loopsLock), and treat ErrConsumerBusy as retryable (sleep and re-pull; the lease expires in 30s) so a replacement loop can take over.
@ -489,0 +989,4 @@loopCtx, stop := context.WithCancel(ctx)mesh.loops[name] = stopmesh.loopsLock.Unlock()🟡 SUGGESTION — startReplicaLoop spawns even when ctx is already cancelled. Close's wait for a reconciliation pass in flight is bounded (meshShutdownTimeout = 2s), while a pass's transport write can block up to defaultWriteTimeout = 30s (writeRequest falls back to it when the caller's ctx has no deadline). A pass that outlives the bounded wait can call startReplicaLoop after stopLoops cleared the loops map and started loopsDone.Wait: the spawned loop exits immediately (cancelled ctx), but the loopsDone.Go Add can race the Wait. Cheap fix:
if ctx.Err() != nil { return }at the top of startReplicaLoop.@ -489,0 +1097,4 @@// was last written, and touching it would load it just to read a window. The// snapshot is taken under the server lock and published outside it, because// appendCommand takes the same lock to look the command log up.func (s *Server) publishResidentTrims(ctx context.Context) {🟡 SUGGESTION — publishResidentTrims skips cold led streams (rs.stream == nil), but a cold stream's base can still advance on disk via the byte budget (evictColdDurableToBudget deletes its oldest segments). No Trim command reaches replicas until the stream is next loaded, so replicas keep serving records the leader has already dropped in the interim — the R=N retention promise is briefly false, self-healing on next load. Consider publishing the trim when the budget evicts a cold led stream.
@ -211,0 +231,4 @@if s.mesh != nil && !isInternalName(name) && s.leadsLocal(name) {stream.writeLock.Lock()err := s.catchUp(ctx, stream, stream.nextSeq())🟡 SUGGESTION — a catchUp failure at first acquire is logged and then swallowed: the stream is served anyway, so reads miss records that are durable in the command log and already on every replica. The same corrupt command log then fails every later publish (publishReplicated → catchUp → error) while reads silently serve stale data. Consider returning the error from acquire so the open fails loudly, or marking the stream read-only, instead of serving a copy that diverges.
Reviewed findings 1–3 and the four suggestions at head
e096f92; resolved ind397824+a459e8f. Nothing in the review was dismissed, and one suggestion (the pin) turned out to need a different mechanism than I first built — details under finding 2.Verification at the new head
a459e8fgo build ./...,go vet ./...,gofmt -l .clean.go test ./...clean, 5 consecutive runs plus a 25× stress loop over the replication/crash/decommission tests (the crash-window test is the one these fixes are about, so I hammered it).racedetector teston the mesh/replication subset clean (noWARNING: DATA RACE).golangci-lint run --fix=false= 81 issues, and the per-file/per-line set is identical tomain's 81 — I verified that by adding a detachedmainworktree and diffingfile:line: rulelists. (I used--fix=falsebecause the repo config hasissues.fix: true, which rewrites unrelated files.) Your 58 was presumably the default-config run; the comparison that matters is no-new-findings, and that holds.go.mod/go.sumuntouched.1. 🟠 cacheTotal inflates on redelivery and resync
Fixed by making the accounting follow what was actually stored.
applyRecordLockednow reports whether it stored the record ((bool, error)),applyReplicatedreports what a resync discarded,applyTrimandjournal.resetToLockedreport what a trim/reset dropped, andServer.releaseCachedByteslowers the cache (CAS loop, floor 0). The cache only grows by records this node keeps and shrinks by what it drops; a replay incatchUpnow accounts too, since the publish that wrote that command never reached its own accounting. A publish whose own record was not stored now reportsErrInternalinstead of handing back a sequence number this node never stored.Covered by
TestReplicatedApplyByteAccounting(store → +bytes, redelivery → unchanged, resync → −discarded +new) andTestReplicatedAppendIsIdempotent, which now assertsstored/discardedon both applications.2. 🟠 resync targets the user base while commands come from a log whose base advanced
Confirmed, and worse than stated: on a durable stream the log is held only by the replicas' cursors, so a replica that kept up trimmed it to its own position, and a replica starting from scratch had nothing left to backfill from — the terminal gap was a symptom of that. Fixed by giving the leader a floor on each log it leads: the position of the first command for a record the stream still holds (the first
Appendwithseq >= streamBase), recomputed from the log on every reconciliation pass and also at load, and applied as a cap on the trim (computeKeepLocked). Resync still targetsmax(userBase, rec.seq), which remains the right target for a copy that fell behind the user window;TestLeaderPinKeepsTheCommandLogBackfillablewipes a replica's stream directory, restarts the node and asserts it rebuilds exactly[base, next)with the leader's payloads.Two things worth flagging, because they changed the design:
leader-pinmirroringrep-<peerID>, so retention (which reads the durable watermark array) would hold the log. That version is wrong in a way that only shows after a restart: a trim that advances the log's in-memory base does not necessarily delete whole segments, so the log comes back from disk with a lower base while the pin's persisted position does not — the pin then sits at or past the log's end and pins nothing, and a replica's redelivered acks can empty the log before the first pass. I chased this as a flakyTestCrashBetweenCommandAndApplyConverges(it reproduced ~1/40 runs) and it is exactly what the reviewer's "lagging leader consumer + budget/age trims" window exposes. The floor is therefore deliberately in-memory: recomputing it from the log's own base on every pass cannot go stale against the log it describes, and the load-time floor (pinFloor(ctx, stream, 0)inreconcileLoadedStream's command-log branch) closes the window between a restart and the first pass.minLiveWatermarktreats a live watermark of 0 as "unset", so with a mix of never-acked and acked consumers its result depends on map order, and it disagrees with the in-memory path (which lets a zero cursor pin). It is pre-existing and independent of these findings, and changing it changes durable retention for every stream, so I left it alone — flagging it here rather than fixing it silently.3. 🟠 replicateLoop's terminal exits leave the mesh entry / ErrConsumerBusy is fatal
Fixed:
mesh.loopsnow holds*replicaRun{stop}and a returning loop deletes its entry throughforgetLooponly while the entry is still its own (a stop-and-restart can install a replacement mid-unwind);ErrConsumerBusyis retryable like any other pull failure instead of terminal;startReplicaLoopreturns early when the server is already shutting down (your suggestion 5); and the loop releases its consume lease on the way out viaframeConsumeReleaseon acontext.WithoutCancelcontext, so a replacement acquires the consumer immediately rather than waiting out the lease timeout.Covered by
TestReplicationSurvivesABusyLease(a foreign token holds the lease, the loop must keep retrying and deliver the next record once it frees — before this change it returned and that stream replicated nothing until the node restarted) andTestForgetLoopRemovesOnlyItsOwnEntry.Suggestions
acquireswallowing acatchUperror: fixed viaServer.loadStream+reconcileLoadedStream. A replay failure now fails the load (wrapped asreplaying %q's command log: %w), the stream is closed, and the resident entry is dropped only if it is still this load's — mirroring the existing open-failure path.decodeCommandmasking truncation: fixed; the reader error is checked before the kind switch, so a payload truncated at the kind byte reports the truncation instead of "command kind 0".evictColdDurableToBudgetmarks the stream (Server.markEvicted, which ignores internal names and streams this node does not lead, and is a no-op without a mesh), and the pass publishes the trim for marked streams usingServer.streamBasewithout loading them. The under-lock path (evictResidentDurableToBudgetLocked) deliberately does not publish.Two extra fixes the work surfaced
journal.resetToLockedfailed on a log directory that does not exist yet, so the first thing that ever happened to a freshly created replica could not be a resync ("opening log directory: ... no such file or directory"). A missing directory now means "nothing to sync"; the next append creates it and that roll syncs it.TestDecommissionRemovesTheReplicaConsumerwas assertingbase == cursor+1after the replica's ack. That is no longer the log's contract: the floor holds the window at the user stream's retained window, so the ack releases the replica's share without moving the base. The test now asserts the three acked records are all still retained and that the watermark advanced, plus that no consumer of the replica's survives the decommission.One pre-existing flake, unrelated to this PR, shows up under full-suite load:
TestConsumeSurvivesEviction("idle stream was not evicted") fails ~1/40 runs because it cancels a consume loop and immediately asks for eviction without waiting for the loop to drop its blocked pull or lease. I left it alone as out of scope; say the word and I will make it deterministic.Follow-up on the flaky
TestConsumeSurvivesEvictionyou asked me to take ownership of — fixed atae55b7e.Root cause: the test cancelled the consume loop and immediately asked for eviction, but cancellation is not synchronous. The loop unwinds on its own goroutine — its blocked pull returns and its lease is released in a defer — and
evictIdleStreams(0)refuses to evict a stream that is stillevictionBlocked()(any blocked pull or live lease). Under full-suite load the test lost that race ~1/40 runs with "idle stream was not evicted". The production semantics are correct (a stream whose loop is still unwinding must not be evicted), so the fix belongs in the test.What changed:
rs.idleEvictable(0), the exact predicateevictIdleStreamsapplies — evaluated under the server lock, before publishing and evicting.racedetectorI found a second latent bug in the same test: the resume wait polled the resident entry but not its load, so a loading placeholder (nilrs.stream) could panic, and its fixed five-second budget could trip on a race-instrumented or heavily loaded run. The resume wait now polls a loaded stream and uses an explicit longer deadline via a newwaitForTimeouthelper (a generalisation of the sharedwaitFor; the default five-second callers are unchanged).Verification at
ae55b7e: 40/40 isolation runs of the test, 4× full suite, 8×racedetectorruns of the test,racedetectoron the mesh/replication subset — all clean;golangci-lint run --fix=falsestill exactly the main baseline (81 issues, nothing in the edited files).One adjacent, pre-existing production defect surfaced but left alone as out of scope: the client's pooled
wireWriter(getWireWriter/sync.Pool) can hand the same writer to concurrent requests on different streams, which the race detector flags (rpc_stream.go) and which can corrupt a request frame and force a retry. It is part of the known full-suiteracedetectorbaseline on main, not introduced by this PR, and fixing it touches the client request path used everywhere — happy to do that as its own change if you want it.Correction to my comment above (758): the pooled-
wireWritercorruption claim does not hold up. I investigated it as a stacked change and the evidence says there is no live race in the client request path:sync.Poolhandoff is exclusive — instrumenting the borrows showed no waiter or writer was ever handed to two live requests across thousands of borrows.publish 350 answered with seq 0just means 350's request arrived first.racedetectorreports aroundrpcStreamare false positives of the pure-Go detector: it flagsr.pending[requestID] = waiterandr.pendingLock.Unlock()(both inside the lock's critical section) and even pairs different heap addresses as racing — the same detector reportsacquire'srs.refs++racing, which is unders.lock.What the investigation did find worth fixing is the
requestWaiterpool — a borrow of the same class that is safe only by fragile ordering (getWaitermutates the struct with no lock; its lifetime crosses into the reader goroutine's critical section) — plus a corrected concurrency regression test. That landed as a stacked PR: #9 (base = this branch).