feat: replicate the stream set and reserve the internal namespace (phase 2) #7
No reviewers
Labels
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
norobo/webstream!7
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/mesh-internal-namespace"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Phase 2 of
plan.md: every mesh node knows the stream set and holds a copy of every stream, and the reservedinternal-namespace holds each stream's replication command log.Based on
main(183d2eawhen branched;mainmerged again afterwards, tipae6348fwas included in7ca5d4d). The branch adds three commits:1650d14(phase 2),7ca5d4d(merge ofmain),fe016a8(review fix).What it does
Server.dispatchrejects theinternal-prefix for every client-facing kind that names a stream, not onlyCreateStream: stream create/delete/get, consumer create/delete/get, publish, pull, ack and consume-release. Both listings hide reserved names, andstore.listStreamsdoes too, so discovery cannot copy a command log. Internal names are accepted where the server itself must use them (validStoreName), so a command log never gets a command log of its own.frameForwardfrom a peer outsideWithMeshPeersis refused. The relay path is the only one that may address a reserved name, and it is node-to-node traffic, so a client can no longer reach a command log by wrapping a request in a relay.forwardToplacesinternal-<name>by the user stream name: the leader of<name>is the command log's only writer, so both sides agree and a replica's relayed command is not refused by the loop guard.replicate.go, new). A polled pass pages each peer's listing (reconcilePageSize, smaller than the client's page) and creates locally any stream this node lacks, with the peer'smaxAgefrom theframeGetStreamreply, which now carries it. Two rules keep create and delete converging without a tombstone: a peer's listing is authoritative only for a stream that peer leads (so a leader never adopts its own stream from a replica's copy, which would resurrect a stream it deleted), and a non-leader deletes its copy only for a stream its leader listed before and no longer does. An unconfirmed local copy is left alone, documented in code and covered byTestUnledLocalStreamIsLeftAlone.rep-<peerID>on the leader's command log, andServer.Closedeletes it on the leader before closing the mesh Conns — that deletion is what unpins the log's retained window, since retention follows consumer watermarks and a lapsed lease neither removes a consumer nor advances its watermark.doc.go's Mesh section now describes discovery and the reserved namespace as implemented.Tests (
discovery_test.go, new)Per-kind rejection of the reserved namespace; hidden listings in memory and durable; retention-exact replication of a stream created through the non-leader (in memory and durable); paged peer listings; no command log of a command log; out-of-band adoption; an unled local copy left alone; delete convergence on replicas including across a replica restart; decommission removing the replica consumer and the log no longer being pinned. One phase-1 assertion was updated: a non-leader now holds an empty replica copy, so
TestMeshPublishRelayedToLeaderasserts it applied no records from a relayed publish instead of holding no state at all.Not in this change
The command log's commands (
Append,Trim,ConsumerUpsert,ConsumerDelete), the replica applier andresyncare phase 3. The relay path still accepts only the kindsframeStreamNameroutes, so a replica's pull/ack on a command log needs the inner-kind widening phase 3's applier will add.Verification
go build ./...,go vet ./...,go test ./...all pass (repeated runs). No newgolangci-lintfindings: the branch reports exactly the same issue set asmain(56 issues onmaintoo; the repo's.golangci.ymlhasformatters.fix: true, so a plaingolangci-lint runalso rewrites unrelated files, which I reverted).racedetector test ./...is already dirty onmain(about 1000WARNING: DATA RACEplus a panic inTestConsumeSurvivesEviction, all inrpc_stream.go's pooled-waiter pattern); the branch adds no new warning class.Phase 2 review of the mesh discovery + reserved internal namespace.
Verification run in this session:
go build ./...,go vet ./..., andgo test ./...all pass on the branch head (fe016a8). I could not run-race(no C compiler in this container) andracedetectoris already dirty onmain, so the concurrency findings below are tagged IMPORTANT with the exact code path, not BLOCKING.High-level: The design is coherent and unusually well documented: relay gating at
dispatchFrom, theinternal-<name>placement rule, the no-tombstone reconciliation rules, and hidden-listing coverage are all sound and well tested. The per-kind reserved-name rejection and hidden-listings tests give real confidence. Issues concentrate in the shutdown path (reconcile goroutine vsClose) and one unbounded in-memory map.@ -0,0 +159,4 @@}if leader == id {s.mesh.confirm(name, id)🟡 SUGGESTION —
confirmed(andreplicas) map entries accumulate without bound in a churning mesh.confirm(name, id)here is called for every stream the peer lists (when it leads it), but entries are only removed byforget(line 313), which runs only indeleteLocalStream— i.e. when this node holds a local copy that it later deletes. A stream the peer lists that this node never manages to adopt locally (e.g.ensureReplica'smaxAgeread orcreateStreamfailed transiently, or the stream was deleted on the peer first) leaves aconfirmedentry that is never cleaned. Over a long-lived mesh with ephemeral streams these maps grow without bound. Consider pruningconfirmed/replicasentries for streams not present in the local listing after a pass, or keying cleanup to the local stream set.@ -0,0 +379,4 @@return true}return s.store != nil && s.store.streamExists(name)🟠 IMPORTANT — data race on
s.storebetween the reconcile goroutine andClose().hasStreamreadss.storehere (andstreamMaxAgedoes at lines 407/411) WITHOUT holdings.lock, whileServer.Close()writess.store = nilunders.lock.Lock()(server.go:442). ThereconcileLoopgoroutine started at server.go:383 is not stopped byClose()— it only exits when the NewServer ctx is cancelled, but the documented use ofClose()is before the ctx ends — so a concurrentreconcileOnce→ensureReplica→hasStream/streamMaxAgecan reads.storeat the same moment Close nils it. This is exactly the race the existingacquiremethod documents and defends against ('Capture the store under the lock: Close nils s.store under the same lock'). Fix: captures.storeinto a local unders.lock(likeacquire) before the nil/store-usage check, or stop the reconcile goroutine (a done channel) before decommission/nil inClose(). Reproduced-by-reading only; could not run-racein this container (no C compiler).@ -0,0 +390,4 @@rs, resident := s.streams[name]s.lock.RUnlock()if resident && rs.stream == nil {🟡 SUGGESTION — behavior change in
getStream: a resident stream whose first open is still in flight (rs.stream == nil) now reportsErrStreamNotFoundtransiently (streamMaxAge returns(0,false)). Previously a durable server answered the getStream existence check from the directory (streamExists), which stayed true during a concurrent load. A client'sConn.Streamexistence check (conn.go:116) can now transiently fail during a concurrent first open. For the mesh's ownmaxAgeread this is benign (retried next pass), but for a client it is a regression. Consider answering existence from the directory/presence whenrs.stream == nilrather than treating it as not-found.@ -375,0 +380,4 @@// starting has no handler to negotiate yet. A peer that is slower still is// retried on the next pass rather than waited for.if server.mesh != nil {go server.reconcileLoop(ctx)🟠 IMPORTANT — the
reconcileLoopgoroutine is never joined or stopped byClose(). It runs until the NewServer ctx is cancelled, butClose()is documented as being for shutdown before that ctx ends (server.go:394-395). After an explicitClose()the goroutine keeps ticking every 2s; duringClose()(between decommission andmesh.close()) it can still dial and reachhasStream/streamMaxAge, which is the race flagged on replicate.go:382. It also keeps callingreconcileOnceon a server whose store is closed/nil'd. Recommend adone/stopchannel thatClose()closes andreconcileLoopselects on (in addition toctx.Done()), so an explicit early Close actually halts reconciliation.Review 28 resolved in
2ced954(pushed tofeat/mesh-internal-namespace). All four findings addressed; each fix carries a test that fails without it except the store-capture one, which is covered by an explicitly concurrent test rather than an assertion.🟠 server.go:383 — reconcile loop not stopped by
Close. Fixed:NewServergives the loop its own child context (loopCtx, stop := context.WithCancel(ctx), stored asmeshPeers.reconcileStop), andClosecancels it first, before decommission and before the mesh Conns or the store are touched.Closewaits for a pass in flight (meshPeers.reconcileDone, closed byreconcileLoop'sdefer) with a bound (meshShutdownTimeout = 2s) because a pass can be waiting on an unreachable peer; a pass that outlives the window only reaches a closed store, whose operations fail rather than corrupt. Tests:TestCloseStopsReconciliation(Close returns with the loop's done channel closed, and a later pass on a closed replica adopts nothing) andTestCloseDuringReconciliation.🟠 replicate.go:382 — data race on
s.store. Fixed both sides:reconcileStop/reconcileDoneabove stop the loop's own passes before the store closes, andhasStream/streamMaxAgenow captures.storeinto a local unders.lock— the same lockClosenils it under — before anynilcheck or use, matching the ruleacquiredocuments.TestCloseDuringReconciliationdrives 50 passes concurrently withClose, then one after it; it runs clean underracedetector test -run 'TestCloseDuringReconciliation|...' ./...(0 races in the phase-2 test set, no frames inreplicate.go/discovery_test.go).🟡 replicate.go:162 — unbounded
confirmed/replicasmaps. Fixed by inverting the order: the pass now callsensureReplicafirst and recordsconfirmonly whens.hasStream(name)holds afterwards. A stream this node never adopted has no copy to retract, so recording it served no purpose but growth; both maps are now bounded by the streams this node holds (itself bounded bymaxStreamsPerServer), and entries still leave viaforgetondeleteLocalStream. Test:TestStreamThatCannotBeAdoptedIsNotConfirmed— a durable peer holds a stream whosestream.metais removed after eviction, so its window is unreadable, adoption fails, and the test asserts neither map gained an entry (and the replica holds nothing).🟡 replicate.go:393 —
getStreamtransiently 404s during a first open. Fixed by reading the window from the directory while a resident stream has no in-memory state yet:streamMaxAgenow falls through tostore.streamMaxAge(name)for the loading case (a mount writesstream.metabefore the stream is materialized) and only reports false when the stream is absent or the mount has not written the window yet. A client'sConn.Streamno longer drops its cached handle during a concurrent open, and a mesh node reads the real window instead of a zero one. Test:TestGetStreamWhileFirstOpenIsInFlightplants the loading marker the waycreateStreamreserves a name and asserts the existence check succeeds with the created window.Verification on
2ced954:go build ./...,go vet ./...,go test ./...— 8 consecutive clean full-suite runs;golangci-lint runreports no findings in the changed hunks (the branch's remaining findings are the same setmainreports).