feat: replicate stream metadata on the cluster log (plan.md Phase 6) #22
No reviewers
Labels
No labels
code-review
No milestone
No project
No assignees
3 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
norobo/webstream!22
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/cluster-metadata-replication"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Implements Phase 6 of
plan.md: polled reconciliation stops being the source of truth for which streams exist.What the phase does
internal-clustercarries a tagged record union:Config{epoch, peers}(Phase 5),StreamMeta{name, maxAge, holders}andTombstone{name}.createClusteredStream/deleteClusteredStream), relays the record to the log's writer when it is not the writer, and applies it locally once the relay returns, so a client that creates and then lists or writes on the same node sees the stream.memberQuorum), the same majority a config change uses.meta-<peer ID>; a user stream's replica staysrep-<peer ID>. Voter parsing accepts both prefixes, so the leader can count a metadata ack.ListStreams,Server.StreamsandgetStreamanswer from the applied metadata plus local copies, so a node that holds no copy still lists and resolves the stream.restoreClusterState), which reproduces the config epoch, the stream set, the holder sets and the tombstones.WithPolledReconciliation(), default off, and its tests are re-pointed: the paging case is deleted (replaced by "a stream created while a node was down is present when it returns, through the log"), the out-of-band case runs on a polling mesh as the safety net, and the cases that stop a replica now seedcreateStreamLocal, which is the safety-net premise.Defects found and fixed on the way
Each is a mechanism, not a timing knob.
forwardTofell back to the live view. Two nodes whose peer leases disagreed each named themselves writer of the cluster log — the one stream whose records decide the member set — so a create on one side waited 30 s for a confirmation the other side could never send. The fallback is now the set placement derives from, never the live view alone.registerConsumerOn), on the first pull that finds no consumer.restoreClusterStateonly replayed a resident cluster log, so a restarted durable node replayed nothing and came back with an empty stream set and epoch 0. It now acquires the log, cold or resident.reconcileLoadedStream(and the consumer-emit guards andmarkEvicted) tested theinternal-prefix where they meant "is a command log".internal-clusteris internal and a replicated stream, sopinFloordecoded its records as commands (visible asa command log holds a command this build cannot decode … trailing bytes 79) and the cluster log skipped the leader's catch-up path on every reload.reconcileLoadedStreamnow reads the epoch back).advancePlacementreturned true for a plan that had already parked, and every pass pushed its deadline forward, so a plan that could not converge consumed the pass for ever — the exact churn bound Phase 4 promises. A parked plan now yields.Termcommand at the stream's current term, so a replica that applied the higher term can read it from its base.deleteLocalStreampublished a tombstone. Releasing a displaced copy would have deleted the stream cluster-wide; the local delete no longer records a command.Verification
go vet ./...clean;golangci-lint run0 issues;go vet -tags rapidandgolangci-lint run --build-tags rapidclean.go test ./...green (82.8 s on this box),go test -short ./...green (~10 s).FuzzDecodeClusterRecordwith seeds (round trip, 27k execs in 5 s).TestClusterMetaModelKeepsTheAppliedSet: rapid model over the applied metadata against a shadow map — last record wins, a tombstone beats the copy, a later create clears it,all()is complete and sorted.Follow-up (not done here)
TestATombstoneDoesNotResurrectOnRestartasserts the phase's claims (no resurrection; copy, replication loop and command log gone) and says in a comment where this one is left.make test-fastgained the-shortskip in the multi-node and durable harnesses; the two full-suite runs above are the evidence thatmake testis still the gate.A stream's existence is now a cluster-log command rather than something a polled pass reconciles: create writes one StreamMeta{name, maxAge, holders} and delete one Tombstone{name}, each acknowledged by a majority of the members and applied in log order by every one of them. Existence and listing are answered from that applied state plus local copies, so no peer is paged, and the newest record naming a stream wins over a tombstone below it. Every member consumes the log as meta-<peer ID> where a user stream's replica is rep-<peer ID>, so its retention follows the slowest member; a restarted node replays its own copy once, which reproduces the config epoch, the stream set and the holder sets. The polled stream-set discovery stays as a bounded safety net behind WithPolledReconciliation(). Getting the phase green turned up six defects outside the metadata code itself: - forwardTo fell back to the live view, so two nodes whose views disagreed each named themselves writer of the cluster log -- the one stream whose records decide that view. The fallback is now the set placement derives from. - a replica registered its consumer over the shared mesh Conn, which can still be holding the create relay whose quorum that ack completes, so the registration waited on the ack it was blocking. The loop registers over its own connection. - restoreClusterState only replayed a resident cluster log, so a restarted durable node replayed nothing and came back with an empty stream set. - rebuildCommandLog consulted the routing rule, so a concurrent term announcement could turn its create into a no-op and leave the log missing. - deleteLocalStream published a tombstone, so releasing a displaced copy would have deleted the stream cluster-wide. - the cluster log's applier did peer calls inline, which queue behind the relay whose quorum it is about to complete. The polled and durable harnesses now skip under -short, with make test-fast and make one as the inner loop. Known red in the full suite: TestAStuckPlacementYieldsThePass and TestTermDisagreementResolvesToTheHigherTerm. Both are recorded in the pull request with what is known about their mechanism.915350efdcto12d526466eStatus after the restart-path fixes (head
21246a7):go test ./...green (85 s),go test -short ./...green,go vetclean,golangci-lint0 issues on both tag sets.One intermittent failure is left, and its evidence is recorded here rather than papered over.
TestTermDisagreementResolvesToTheHigherTermfails roughly 1 run in 6 in isolation (it passed in the last three full-suite runs). The trace:consumer not found: "rep-<id>", then its re-registration reportsstream not found: stream "internal-s-1" has no directory;quorumTimeout(30 s) and failsleader unavailable— the leader appended the record but never got an ack;appendToCommandLog's ensure fallback), which is what makes the second registration succeed on a later attempt.So the open question is narrow: why does the stream's command-log directory report "no directory" on the node the replica asks, immediately after the peer it restarts comes back, and re-appear moments later. No
deleteInternalStream/deleteStaleCommandLogcall is logged in those runs, and budget eviction deletes segments, not stream directories, so the remaining suspects are store recovery at the restarted node'snewStoreand a request served by a node whose own copy of the log does not exist (the restarted replica hass-<n>but never a command log for it).Also fixed since the PR was opened, both found while chasing that test:
Termcommand at that term, and only when the log is genuinely new (announcing on every pass appended oneTermper pass).Head
cdb0912: the intermittent is fixed, and one more restart-path defect fell out of chasing it.The intermittent —
TestTermDisagreementResolvesToTheHigherTerm— was the term rule, not timing. The replica refused every command carrying a term below its own and stopped its loop, on the premise "a lower term means the log went backwards". That premise is false: a copy resuming at the leader's retained base replays commands written before it learned its term, because the retained prefix predates the raise by definition. Applying them is safe by construction — an Append the copy holds is a repeat, a Trim is a floor, a Holders epoch and a consumer upsert only move forward, a Reset is the replay's own primitive — and the term still only moves forward. Only aTermannouncement below the copy's term is not replayable, so it is ignored rather than moving routing back to the leader it names.That is a contract correction, so the model moved with it:
TestReplicaAppliesHistoryBelowItsTerm(renamed from...RefusesACommandBelowItsTerm),TestLeadershipStatePropertiesnow models "history is applied, term never regresses, a below-term announcement does not move the leader", anddoc.go's failover section states the rule. 10 consecutive runs of the test green; three consecutive full-suite runs green (88 s, 84 s, 87 s).A second defect, found because the full suite started hanging in unrelated tests: a replica that re-registers its consumer pulled again immediately, with no pause. A leader that keeps dropping the consumer therefore turned the loop into a spin — and inside a
synctestbubble a spinning goroutine freezes the fake clock, so other multi-node tests hung for minutes with every goroutine reportedselect (durable). The re-registration is now paced by the same retry interval as every other loop error.-racecannot run in this container, and this is a host-side issue. Verified, not guessed:go version go1.27.3,GOROOT=/usr/local/go;/proc/self/mountinfoshows259:2 /usr/local/go /usr/local/go ro— the container does read the host's GOROOT;0xC128ofsrc/runtime/race/race_linux_arm64.sysoare1f c0 00 71(the unpatchedcmp w0, #0x30), mtime2026-10-19 22:20;go test -race -a …still diesFATAL: Found 47 - Supported 48, so it is not a build-cache artifact.So the host patch did not reach that file — most likely the toolchain was installed/replaced on 2026-10-19 after the last patch run (the script's own docs say to re-run it after every Go upgrade). On the host:
sudo scripts/patch-race-runtime.sh(it verifies the bytes and refuses if they are neither original nor patched), thenmake testin the container should start normally. Ping me and I will run the race gate and report.Everything else on the gate is clean:
go vetandgolangci-linton both tag sets,go test -short ./...(~10 s), the full untagged profile at 86.0% with itscoverage.logline added.-raceruns now, and it was worth waiting for.Why it failed before: the syso is patched (
bytes@0xC128 = 1f bc 00 71, mtime 20:54 today) — the blocker was the build cache, which still held the race runtime compiled from the unpatched file.go test -race -a(force rebuild) started fine, andgo clean -cacheonce makes plaingo test -race ./...work. Worth noting for the next Go upgrade: patch, then clean the cache, or keep using-a.What the race build found (head
192b893):startReplicaLoopadded to a sharedsync.WaitGroup(meshPeers.loopsDone) whilestopLoopswas waiting on it. AWaitGroupmay not be added to while aWaitis in flight on a zero counter — and the window was live: a request still being served could spawn a loop afterClosebegan waiting, which is precisely the loop that would write to the store after it closed. Each loop now has its owndonechannel closed by the loop itself, so stopping waits per run and noAddcan overlap aWait. A "loops stopped" flag was rejected:stopLoopsalso serves tests that quiesce a replica and restart its loop afterwards.TestAddPeerStopsAtTheMemberCapassignedmesh.configdirectly while the pass read it underconfigLock; it adopts throughadoptConfignow.ensureReplicacreated an adopted copy withcreateStream, which on a mesh publishes a cluster record and waits for a member quorum — blocking a reconciliation pass for seconds to adopt a stream that by definition has no record. The safety net creates the copy locally.TestCreationRecordsThePlacementEpochasserted the epoch on the node the hash names; under-racea different node promoted first. It asserts on the node that actually serves the stream now.Gates, all green on
192b893:go test -race ./...DATA RACEgo test -race -tags rapid ./...(make test-props)go test ./...go test -short ./...go vet,golangci-lintcoverage.logline addedStill open from the phase's own list: a re-create after delete and a restart does not reach the restarted node (
TestATombstoneDoesNotResurrectOnRestartasserts the phase's claims and says where that one is left).Last open item closed on
3a860aa: a re-create after a delete, with a restart in between, now reaches the restarted node.It was a client bug, not a server one. The writer published nothing for the re-create because the create never reached the server:
Conn.DeleteStreamkept its cached*Streamhandle when the delete's quorum failed. In the reproduction the delete is issued while one member is down, so it answersErrNoQuorum— the record is durable on the writer and applied there, only the wait failed — and the followingConn.CreateStreamwas then answered from that stale handle (get-or-create hit, no RPC), so noStreamMetawas ever written.Conn.DeleteStreamnow drops the handle for every answer the server gave, including a failed one; a transport failure still keeps it, because there the outcome is unknown.TestATombstoneDoesNotResurrectOnRestartasserts the whole sequence end to end — create, publish, converge, stop a member, delete (toleratingErrNoQuorum), restart, no resurrection, then re-create and the restarted node adopting it — three consecutive runs green, and it is no longer commented as a known gap.One test-tightness change came with it:
TestConsumeSurvivesEvictionwaited five seconds for a consume loop to release its stream. The loop releases on its way out of a pull it is already inside, so under-racethat can exceed five seconds; the assertion is that it released at all, so it uses the convergence budget.Gates on
3a860aa:go test ./...ok (88 s),go test -race ./...ok twice (87 s, 86 s),go test -short ./...ok,go vetandgolangci-lintclean on both tag sets. Nothing in the phase is left open now.Comment-trim pass done on
d1583a0(rebased onto your merge ofmain,93196ad; pushed as a normal fast-forward, nothing forced).Same rule as PR #17: the exported names keep a godoc line opening with the name, everything else keeps only what the code cannot say — an ordering or locking constraint, a failure mode, a wire or membership invariant, a rejected alternative that would otherwise be re-litigated — and the narration, the history and the retelling of the body are gone. Test comments are gone unless the setup is the reason the case is shaped that way (which node a command is issued at, why a copy is seeded locally, why the epoch is asserted on the serving node).
gofmtrealigning thereplicaConsumerPrefix/metaConsumerPrefixconst block after the comment above it went away. No code token changed.doc.go's "The cluster log" and term sections) kept, tightened; the roadmap line ("scheduled for deletion one release after the log ships") moved out of godoc — that belongs inplan.md, not the package documentation.Gates after the rebase:
go test ./...ok (83 s),go test -race ./...ok (93 s, noDATA RACE),go test -short ./...ok,go vetandgolangci-lintclean on both tag sets.Phase 6 review (head
d1583a0, base5f9464f). I read the full diff and the surrounding code, then verified the tooling claims and every API/behaviour claim I make below with a command in this session.Ran here (clone of origin/feat/cluster-metadata-replication, go1.27.3):
go build ./...— cleango vet ./...— cleangofmt -l .— no outputgolangci-lint run(~/.golangci.ymldefault) —0 issues.go test -short ./...—ok git.norobo.chat/norobo/webstream 5.545sgo test ./...—ok git.norobo.chat/norobo/webstream 83.112sSo the PR's gate claims reproduce. The findings below are the mechanisms the green suite does not cover.
Findings: 3 IMPORTANT, 3 SUGGESTION.
applyStreamMetawedges the cluster-log applier loop (and surfaces to the client as a failed create for a record that is already durable).admitTermno longer refuses a below-term Append, and applying one at a sequence the copy does not hold resets the copy's window — so a superseded leader that still serves pulls can overwrite a replica's history. This is the same loosening that fixed defect 7, but it is wider than the comment claims.behindLatecomersnever forgets a peer, so one member that dies or is removed with a stale epoch blocks the cluster-log handover for the life of the process.Server.Streams()is permanently empty without a mesh, undocumented.ListStreamspage now materializes the whole stream set.t.out(a test-run artifact) is committed.Not flagged: the
replicaConsumerName(stream, id)rework (I tracedmeta-<peer>throughack.stream=internalName(name)andcheckRelayedAckVote'suserNameround trip — consistent for both the cluster log and its command log), and the removedpickAnnouncedself-filter (both former callers now filterleader == s.host.ID()themselves).@ -147,0 +157,4 @@}// behindLatecomers reports whether a member last reported an epoch below the applied one.func (m *meshPeers) behindLatecomers() bool {🟠 IMPORTANT —
behindLatecomersnever forgets a peer, so it can block the cluster-log handover for ever.m.reportedis written only byobserveEpoch(fromprobePeer, i.e. only for current members that answer) and is never pruned:adoptConfig,releaseRemovedPeersandmarkDeadall leave it alone. A member whose last answer was epoch 4, and which then dies, is partitioned, or is removed while the set moves to epoch 5, keepsreported[that peer] = 4for the process's life.handOverClusterLog(membership.go:if s.mesh.behindLatecomers() { return }) then returns early on every reconcile pass, so the log never moves to the nodehashLeader(clusterStream, members)names, for as long as this node runs.Probe I ran against this tree:
members = [2g 2h], reported = map[2i:3], isMember(2i) = false, behindLatecomers = true.Fix: in the loop, skip ids that are not in
m.members()(or delete their entries inadoptConfig, next to where the applied set changes).@ -0,0 +288,4 @@// Streams reports the stream set the cluster log describes, with each stream's retention window// and the holders its placement epoch names. It answers from applied state and asks no peer.func (s *Server) Streams() []StreamMeta {🟡 SUGGESTION —
Streams()is empty on a server without a mesh, and the doc does not say so.clusterMetais filled only byapplyStreamMeta/applyStreamTombstone, which are reached only from the mesh paths, so a standalone server answers with nothing whileConn.ListStreamslists its streams.Probe I ran against this tree, standalone server with
orderscreated:hasStream(orders) = true, len(Streams()) = 0, len(listStreamNames) = 1.Fix: either state "on a mesh" in the doc comment, or answer from
localStreamNameswhens.mesh == nil, aslistStreamNamesalready does.@ -0,0 +390,4 @@}if resp := s.createStreamLocal(ctx, createStreamRequest{name: meta.Name, maxAge: meta.MaxAge}); resp.err() != nil {return fmt.Errorf("creating stream %q: %w", meta.Name, resp.err())🟠 IMPORTANT — a local create failure wedges the applier loop.
This returns the error from
createStreamLocalto the caller. There are two callers, and both are worse off for it:replicate.go,replicateLoop:err = s.applyClusterRecord(ctx, cmd.rec.data)) then deliberately does not ack and re-pulls the same command — "the command stays unapplied on the leader's side of the watermark and is redelivered". A record this node can never materialize is therefore retried for ever, so this member stops applying every later record: laterStreamMetas,Tombstones andConfigepoch changes, while itsmeta-<peer>consumer keeps pinning the log's trim floor.createClusteredStreamcallsapplyStreamMetaon the writer, so the client getsErrLimitExceededfor a record that is already durable cluster-wide.Condition: the member is at
maxStreamsPerServer(1024) orcreateStreamLocalhits a store error (%w: creating stream: %v,ErrInternal) when the create record arrives. Both are local, permanent conditions.Fix: for a failure to materialize the local copy, log and
return nil— the metadata is already applied bys.meta.applyMeta(meta)andnoteUnheldFromMetadatare-queues the copy on a later pass. Keep returning an error only for a decode/apply fault. Same for the writer: the record's durability does not depend on the local copy.@ -1468,3 +1561,1 @@return fmt.Errorf("%w: stream %q is at term %d, command carries %d",errStaleTermCommand, name, current, term)case term == current:case cmd.term < current:🟠 IMPORTANT — a below-term Append from a superseded writer is applied, and can reset the copy's window.
case cmd.term < current: return niladmits every below-term command, Appends included.admitTerm's only production caller isreplicateLoop(if err = s.admitTerm(ctx, name, cmd); err == nil { switch cmd.kind { case commandAppend: ... }), which then applies the record.Probe I ran against this tree (added to the package, run, then removed):
The second line is
applyReplicated'sif rec.seq > s.log.next { resetToLocked(max(base, rec.seq)) }— the copy's memlog is replaced and restarted at the foreign record (resetToLockedsetsbase = next = seq). So a record the copy does not hold is not "the divergence it was"; it is adopted, and what the copy held below it is discarded.Condition: the copy learned a higher term (a
Termannouncement from the new leader, whichadmitTermapplies atcmd.term == current) while its loop still targets the superseded leader — the loop'sleaderis only realigned byrealignReplicaLoops, once per reconcile pass — and that node still servesframePullbecause it still believes it leads. The pull path checks the serving node's leadership, never the sender's term, so the comment's "a superseded leader cannot reach here" does not hold for this direction.Fix: pass the loop's
leaderintoadmitTermand admit a below-term command only while that node is the copy's announced leader (state.leaderValue()); otherwise drop the command andretireReplicaLoopso the pass re-attaches to the announced leader. The case defect 7 needed still passes, because there the sender is the announced leader and the records are its own history.@ -2276,0 +2321,4 @@return s.localStreamNames(req)}names, _, err := s.localStreamNames(listStreamsRequest{})🟡 SUGGESTION — every
ListStreamspage now materializes the whole stream set.localStreamNames(listStreamsRequest{})passes an empty request, andlistStorePagetreatslimit == 0as "no cap", so a durable node enumerates the entirestreams/directory;s.meta.all()then clones every holder slice and the union is sorted beforereq.limitis applied. The pre-change path readlimit+1directory entries. Clients pollListStreams, so this is O(all streams) + a sort per page request.Fix: both sources are already sorted — walk them with the cursor and stop at
req.limitinstead of materializing the union (the metadata side iss.meta.all(); keep a sorted index or sort once per generation if that walk matters).@ -0,0 +1 @@ok git.norobo.chat/norobo/webstream 3.173s🟡 SUGGESTION — a test-run artifact is committed.
t.outcontains one line ofgo testoutput (ok git.norobo.chat/norobo/webstream 3.173s)..gitignorelists*.testandcoverage.out, not this, andAGENTS.mdnow recommends redirecting long runs to a file — which is how it got here.Fix:
git rm --cached t.outand addt.out(or*.out) to.gitignorein the same commit.Review findings resolved on
cce9f18. Each fix has the check that fails without it.1 🟠 A local create failure wedging the applier — fixed.
applyStreamMetanow logs a failed copy and returns nil (createPlacedCopy): the record's application is the metadata, which is already applied, and the copy is placed work the pass re-queues (noteUnheldFromMetadata). No withheld ack, no re-pull loop, and the writer no longer fails a create that is durable cluster-wide.Check:
TestAFailedLocalCopyDoesNotFailTheRecord— fillsmaxStreamsPerServerslots, applies aStreamMeta, asserts nil error, metadata present, no copy.2 🟠 A below-term Append from a superseded writer — fixed, narrower.
admitTermnow takes the sender (from peer.ID) and admits a below-term command only when that node is the copy's announced leader; anywhere else it returnserrStaleTermCommand, the loop logs "replication left a superseded writer" and ends, so the pass re-attaches at the announced leader. The case defect 7 needed still holds: there the sender is the announced leader and the records are its own history.doc.go's failover paragraph andadmitTerm's comment state the rule now, including the superseded-writer direction.Checks:
TestReplicaAppliesHistoryBelowItsTerm(history from the announced leader is applied; the same command fromold-leaderis refused) andTestLeadershipStateProperties, whose model now draws the sender and requiresterm < model.term && from != model.leaderto be refused.3 🟠
behindLatecomersnever forgetting a peer — fixed. It skips ids that are not in the applied member set and does not wait on a dead one (m.alive), andadoptConfigprunes the reports of departed members next to the set change.Check:
TestBehindLatecomersIgnoresTheDepartedAndTheDead— older epoch from a live member blocks;markDeadunblocks; removal viaadoptConfigunblocks and prunes.4 🟡
Streams()on a server without a mesh — fixed. It answers with the local set (name and window, no holders) and is documented as such. The signature gained(ctx context.Context) ([]StreamMeta, error)because that path reads the store, so it can fail instead of silently answering empty.Check:
TestAStandaloneStreamsReportsTheLocalSet.5 🟡 Every
ListStreamspage materializing everything — fixed.clusterMetakeeps a name index (slices.BinarySearch/Insert/Deleteon apply),namesAfter(cursor, n)returns a slice of it, and a page is the linear merge of that with alimit+1directory page after the cursor. Both sides are incremental now; the window widens (bounded, four doublings) only when a tombstone shortens a page while a source still has names.Check:
TestListStreamsPagesTheMergedNameSets— five streams, one name per page, from the non-writer; every name once, in order,hasMoreonly while names remain.6 🟡
t.out— fixed. Removed and*.outadded to.gitignore.Gates on
cce9f18:go test ./...go test -race ./...DATA RACEgo test -race -tags rapid ./...go test -short ./...go vet,golangci-lint(both tag sets)For the record on your two "not flagged" notes: agreed, and the
replicaConsumerNamepath is now also exercised end to end by the metadata tests, since every converge in them depends on ameta-ack counting as a vote.