feat: replicate the stream set and reserve the internal namespace (phase 2) #7

Merged
sam merged 4 commits from feat/mesh-internal-namespace into main 2026-09-26 16:43:42 +00:00
Member

Phase 2 of plan.md: every mesh node knows the stream set and holds a copy of every stream, and the reserved internal- namespace holds each stream's replication command log.

Based on main (183d2ea when branched; main merged again afterwards, tip ae6348f was included in 7ca5d4d). The branch adds three commits: 1650d14 (phase 2), 7ca5d4d (merge of main), fe016a8 (review fix).

What it does

  • Reserved namespace. Server.dispatch rejects the internal- prefix for every client-facing kind that names a stream, not only CreateStream: stream create/delete/get, consumer create/delete/get, publish, pull, ack and consume-release. Both listings hide reserved names, and store.listStreams does too, so discovery cannot copy a command log. Internal names are accepted where the server itself must use them (validStoreName), so a command log never gets a command log of its own.
  • Relay gating. frameForward from a peer outside WithMeshPeers is refused. The relay path is the only one that may address a reserved name, and it is node-to-node traffic, so a client can no longer reach a command log by wrapping a request in a relay.
  • Internal placement. forwardTo places internal-<name> by the user stream name: the leader of <name> is the command log's only writer, so both sides agree and a replica's relayed command is not refused by the loop guard.
  • Discovery (replicate.go, new). A polled pass pages each peer's listing (reconcilePageSize, smaller than the client's page) and creates locally any stream this node lacks, with the peer's maxAge from the frameGetStream reply, which now carries it. Two rules keep create and delete converging without a tombstone: a peer's listing is authoritative only for a stream that peer leads (so a leader never adopts its own stream from a replica's copy, which would resurrect a stream it deleted), and a non-leader deletes its copy only for a stream its leader listed before and no longer does. An unconfirmed local copy is left alone, documented in code and covered by TestUnledLocalStreamIsLeftAlone.
  • Replica position and decommission. A node that does not lead a stream registers a consumer named rep-<peerID> on the leader's command log, and Server.Close deletes it on the leader before closing the mesh Conns — that deletion is what unpins the log's retained window, since retention follows consumer watermarks and a lapsed lease neither removes a consumer nor advances its watermark.
  • docs. doc.go's Mesh section now describes discovery and the reserved namespace as implemented.

Tests (discovery_test.go, new)

Per-kind rejection of the reserved namespace; hidden listings in memory and durable; retention-exact replication of a stream created through the non-leader (in memory and durable); paged peer listings; no command log of a command log; out-of-band adoption; an unled local copy left alone; delete convergence on replicas including across a replica restart; decommission removing the replica consumer and the log no longer being pinned. One phase-1 assertion was updated: a non-leader now holds an empty replica copy, so TestMeshPublishRelayedToLeader asserts it applied no records from a relayed publish instead of holding no state at all.

Not in this change

The command log's commands (Append, Trim, ConsumerUpsert, ConsumerDelete), the replica applier and resync are phase 3. The relay path still accepts only the kinds frameStreamName routes, so a replica's pull/ack on a command log needs the inner-kind widening phase 3's applier will add.

Verification

go build ./..., go vet ./..., go test ./... all pass (repeated runs). No new golangci-lint findings: the branch reports exactly the same issue set as main (56 issues on main too; the repo's .golangci.yml has formatters.fix: true, so a plain golangci-lint run also rewrites unrelated files, which I reverted). racedetector test ./... is already dirty on main (about 1000 WARNING: DATA RACE plus a panic in TestConsumeSurvivesEviction, all in rpc_stream.go's pooled-waiter pattern); the branch adds no new warning class.

Phase 2 of `plan.md`: every mesh node knows the stream set and holds a copy of every stream, and the reserved `internal-` namespace holds each stream's replication command log. Based on `main` (`183d2ea` when branched; `main` merged again afterwards, tip `ae6348f` was included in `7ca5d4d`). The branch adds three commits: `1650d14` (phase 2), `7ca5d4d` (merge of `main`), `fe016a8` (review fix). ### What it does - **Reserved namespace.** `Server.dispatch` rejects the `internal-` prefix for every client-facing kind that names a stream, not only `CreateStream`: stream create/delete/get, consumer create/delete/get, publish, pull, ack and consume-release. Both listings hide reserved names, and `store.listStreams` does too, so discovery cannot copy a command log. Internal names are accepted where the server itself must use them (`validStoreName`), so a command log never gets a command log of its own. - **Relay gating.** `frameForward` from a peer outside `WithMeshPeers` is refused. The relay path is the only one that may address a reserved name, and it is node-to-node traffic, so a client can no longer reach a command log by wrapping a request in a relay. - **Internal placement.** `forwardTo` places `internal-<name>` by the *user* stream name: the leader of `<name>` is the command log's only writer, so both sides agree and a replica's relayed command is not refused by the loop guard. - **Discovery** (`replicate.go`, new). A polled pass pages each peer's listing (`reconcilePageSize`, smaller than the client's page) and creates locally any stream this node lacks, with the peer's `maxAge` from the `frameGetStream` reply, which now carries it. Two rules keep create and delete converging without a tombstone: a peer's listing is authoritative only for a stream that peer *leads* (so a leader never adopts its own stream from a replica's copy, which would resurrect a stream it deleted), and a non-leader deletes its copy only for a stream its leader listed before and no longer does. An unconfirmed local copy is left alone, documented in code and covered by `TestUnledLocalStreamIsLeftAlone`. - **Replica position and decommission.** A node that does not lead a stream registers a consumer named `rep-<peerID>` on the leader's command log, and `Server.Close` deletes it on the leader before closing the mesh Conns — that deletion is what unpins the log's retained window, since retention follows consumer watermarks and a lapsed lease neither removes a consumer nor advances its watermark. - **docs.** `doc.go`'s Mesh section now describes discovery and the reserved namespace as implemented. ### Tests (`discovery_test.go`, new) Per-kind rejection of the reserved namespace; hidden listings in memory and durable; retention-exact replication of a stream created through the non-leader (in memory and durable); paged peer listings; no command log of a command log; out-of-band adoption; an unled local copy left alone; delete convergence on replicas including across a replica restart; decommission removing the replica consumer and the log no longer being pinned. One phase-1 assertion was updated: a non-leader now holds an empty replica copy, so `TestMeshPublishRelayedToLeader` asserts it applied no records from a relayed publish instead of holding no state at all. ### Not in this change The command log's commands (`Append`, `Trim`, `ConsumerUpsert`, `ConsumerDelete`), the replica applier and `resync` are phase 3. The relay path still accepts only the kinds `frameStreamName` routes, so a replica's pull/ack on a command log needs the inner-kind widening phase 3's applier will add. ### Verification `go build ./...`, `go vet ./...`, `go test ./...` all pass (repeated runs). No new `golangci-lint` findings: the branch reports exactly the same issue set as `main` (56 issues on `main` too; the repo's `.golangci.yml` has `formatters.fix: true`, so a plain `golangci-lint run` also rewrites unrelated files, which I reverted). `racedetector test ./...` is already dirty on `main` (about 1000 `WARNING: DATA RACE` plus a panic in `TestConsumeSurvivesEviction`, all in `rpc_stream.go`'s pooled-waiter pattern); the branch adds no new warning class.
Phase 2 of plan.md: every mesh node knows the stream set and holds a copy
of every stream, and the reserved internal- namespace holds each stream's
replication command log.

- server.dispatch rejects the reserved prefix for every client-facing kind
  that names a stream (not only CreateStream), and both listings hide
  reserved names. A relay from a peer outside the mesh is refused, which
  keeps the relay path — node-to-node traffic, and the only path that may
  address a command log — out of a client's reach. An internal name is
  placed by its user stream, because that stream's leader is the command
  log's only writer, and frameGetStream now reports the retention window.
- replicate.go (new): the reserved namespace, the polled reconciliation
  loop and the replica wiring. A pass pages each peer's listing (the
  listing is paginated, so one reply is not the whole set) and creates
  locally any stream this node lacks, with the peer's retention window. A
  peer's listing is authoritative only for a stream that peer leads: a
  stream its leader deleted is dropped on the replicas, while a copy a
  leader has never listed is left alone rather than adopted (that would
  resurrect a deleted stream from a derived copy) or deleted (that would
  lose the only copy). A node that does not lead a stream registers a
  consumer named rep-<peerID> on the leader's command log, and deletes it
  on a graceful shutdown, which is what unpins the log's retained window.
- doc.go: the mesh section now describes discovery as implemented.

Tests (discovery_test.go): per-kind rejection of the reserved namespace,
hidden listings, retention-exact replication in memory and durable, paged
peer listings, no command log of a command log, out-of-band adoption,
delete convergence across a replica restart, and decommission.
fix: resolve the phase-2 review — retention read name, conn.go churn
All checks were successful
code-review / Agent review (pull_request) Successful in 8s
fe016a8c9a
Review of the phase-2 branch raised three findings. One is a real naming
problem in the new code; the other two read main's phase-1 relay code and
Conn.Close as additions, which is what a diff against the stale base
feature/server-mesh-replication shows: that branch predates the phase-1
merge (df33ac5), while this branch is based on main and has main merged.

- conn.go: rename getStreamInfo to maxAge, per the review. The doc comment
  now states the unit (nanoseconds) that the return value's type alone did
  not.
- conn.go: restore the original layout. Extracting the paging loop and
  inserting the two new methods after Close made the diff represent
  Conn.Close (and listStreamsPageSize) as deleted and re-added, which is
  what made pre-existing lines show up as new. The new methods are appended
  after the last Conn method instead, so the file's only changes are the
  extracted loop and the two additions.
pi-reviewer left a comment

Phase 2 review of the mesh discovery + reserved internal namespace.

Verification run in this session: go build ./..., go vet ./..., and go test ./... all pass on the branch head (fe016a8). I could not run -race (no C compiler in this container) and racedetector is already dirty on main, so the concurrency findings below are tagged IMPORTANT with the exact code path, not BLOCKING.

High-level: The design is coherent and unusually well documented: relay gating at dispatchFrom, the internal-<name> placement rule, the no-tombstone reconciliation rules, and hidden-listing coverage are all sound and well tested. The per-kind reserved-name rejection and hidden-listings tests give real confidence. Issues concentrate in the shutdown path (reconcile goroutine vs Close) and one unbounded in-memory map.

Phase 2 review of the mesh discovery + reserved internal namespace. **Verification run in this session:** `go build ./...`, `go vet ./...`, and `go test ./...` all pass on the branch head (fe016a8). I could not run `-race` (no C compiler in this container) and `racedetector` is already dirty on `main`, so the concurrency findings below are tagged IMPORTANT with the exact code path, not BLOCKING. **High-level:** The design is coherent and unusually well documented: relay gating at `dispatchFrom`, the `internal-<name>` placement rule, the no-tombstone reconciliation rules, and hidden-listing coverage are all sound and well tested. The per-kind reserved-name rejection and hidden-listings tests give real confidence. Issues concentrate in the shutdown path (reconcile goroutine vs `Close`) and one unbounded in-memory map.
@ -0,0 +159,4 @@
}
if leader == id {
s.mesh.confirm(name, id)
Member

🟡 SUGGESTION — confirmed (and replicas) map entries accumulate without bound in a churning mesh. confirm(name, id) here is called for every stream the peer lists (when it leads it), but entries are only removed by forget (line 313), which runs only in deleteLocalStream — i.e. when this node holds a local copy that it later deletes. A stream the peer lists that this node never manages to adopt locally (e.g. ensureReplica's maxAge read or createStream failed transiently, or the stream was deleted on the peer first) leaves a confirmed entry that is never cleaned. Over a long-lived mesh with ephemeral streams these maps grow without bound. Consider pruning confirmed/replicas entries for streams not present in the local listing after a pass, or keying cleanup to the local stream set.

🟡 SUGGESTION — `confirmed` (and `replicas`) map entries accumulate without bound in a churning mesh. `confirm(name, id)` here is called for every stream the peer lists (when it leads it), but entries are only removed by `forget` (line 313), which runs only in `deleteLocalStream` — i.e. when this node *holds* a local copy that it later deletes. A stream the peer lists that this node never manages to adopt locally (e.g. `ensureReplica`'s `maxAge` read or `createStream` failed transiently, or the stream was deleted on the peer first) leaves a `confirmed` entry that is never cleaned. Over a long-lived mesh with ephemeral streams these maps grow without bound. Consider pruning `confirmed`/`replicas` entries for streams not present in the local listing after a pass, or keying cleanup to the local stream set.
replicate.go Outdated
@ -0,0 +379,4 @@
return true
}
return s.store != nil && s.store.streamExists(name)
Member

🟠 IMPORTANT — data race on s.store between the reconcile goroutine and Close(). hasStream reads s.store here (and streamMaxAge does at lines 407/411) WITHOUT holding s.lock, while Server.Close() writes s.store = nil under s.lock.Lock() (server.go:442). The reconcileLoop goroutine started at server.go:383 is not stopped by Close() — it only exits when the NewServer ctx is cancelled, but the documented use of Close() is before the ctx ends — so a concurrent reconcileOnce → ensureReplica → hasStream/streamMaxAge can read s.store at the same moment Close nils it. This is exactly the race the existing acquire method documents and defends against ('Capture the store under the lock: Close nils s.store under the same lock'). Fix: capture s.store into a local under s.lock (like acquire) before the nil/store-usage check, or stop the reconcile goroutine (a done channel) before decommission/nil in Close(). Reproduced-by-reading only; could not run -race in this container (no C compiler).

🟠 IMPORTANT — data race on `s.store` between the reconcile goroutine and `Close()`. `hasStream` reads `s.store` here (and `streamMaxAge` does at lines 407/411) WITHOUT holding `s.lock`, while `Server.Close()` writes `s.store = nil` under `s.lock.Lock()` (server.go:442). The `reconcileLoop` goroutine started at server.go:383 is not stopped by `Close()` — it only exits when the NewServer ctx is cancelled, but the documented use of `Close()` is *before* the ctx ends — so a concurrent `reconcileOnce` → `ensureReplica` → `hasStream`/`streamMaxAge` can read `s.store` at the same moment Close nils it. This is exactly the race the existing `acquire` method documents and defends against ('Capture the store under the lock: Close nils s.store under the same lock'). Fix: capture `s.store` into a local under `s.lock` (like `acquire`) before the nil/store-usage check, or stop the reconcile goroutine (a done channel) before decommission/nil in `Close()`. Reproduced-by-reading only; could not run `-race` in this container (no C compiler).
replicate.go Outdated
@ -0,0 +390,4 @@
rs, resident := s.streams[name]
s.lock.RUnlock()
if resident && rs.stream == nil {
Member

🟡 SUGGESTION — behavior change in getStream: a resident stream whose first open is still in flight (rs.stream == nil) now reports ErrStreamNotFound transiently (streamMaxAge returns (0,false)). Previously a durable server answered the getStream existence check from the directory (streamExists), which stayed true during a concurrent load. A client's Conn.Stream existence check (conn.go:116) can now transiently fail during a concurrent first open. For the mesh's own maxAge read this is benign (retried next pass), but for a client it is a regression. Consider answering existence from the directory/presence when rs.stream == nil rather than treating it as not-found.

🟡 SUGGESTION — behavior change in `getStream`: a resident stream whose first open is still in flight (`rs.stream == nil`) now reports `ErrStreamNotFound` transiently (streamMaxAge returns `(0,false)`). Previously a durable server answered the getStream existence check from the directory (`streamExists`), which stayed true during a concurrent load. A client's `Conn.Stream` existence check (conn.go:116) can now transiently fail during a concurrent first open. For the mesh's own `maxAge` read this is benign (retried next pass), but for a client it is a regression. Consider answering existence from the directory/presence when `rs.stream == nil` rather than treating it as not-found.
server.go Outdated
@ -375,0 +380,4 @@
// starting has no handler to negotiate yet. A peer that is slower still is
// retried on the next pass rather than waited for.
if server.mesh != nil {
go server.reconcileLoop(ctx)
Member

🟠 IMPORTANT — the reconcileLoop goroutine is never joined or stopped by Close(). It runs until the NewServer ctx is cancelled, but Close() is documented as being for shutdown before that ctx ends (server.go:394-395). After an explicit Close() the goroutine keeps ticking every 2s; during Close() (between decommission and mesh.close()) it can still dial and reach hasStream/streamMaxAge, which is the race flagged on replicate.go:382. It also keeps calling reconcileOnce on a server whose store is closed/nil'd. Recommend a done/stop channel that Close() closes and reconcileLoop selects on (in addition to ctx.Done()), so an explicit early Close actually halts reconciliation.

🟠 IMPORTANT — the `reconcileLoop` goroutine is never joined or stopped by `Close()`. It runs until the NewServer ctx is cancelled, but `Close()` is documented as being for shutdown *before* that ctx ends (server.go:394-395). After an explicit `Close()` the goroutine keeps ticking every 2s; during `Close()` (between decommission and `mesh.close()`) it can still dial and reach `hasStream`/`streamMaxAge`, which is the race flagged on replicate.go:382. It also keeps calling `reconcileOnce` on a server whose store is closed/nil'd. Recommend a `done`/`stop` channel that `Close()` closes and `reconcileLoop` selects on (in addition to `ctx.Done()`), so an explicit early Close actually halts reconciliation.
Findings from pi-reviewer's review (review 28) on PR #7.

- IMPORTANT (server.go:383, replicate.go:382): reconciliation now runs on the
  server's own child context, which Close cancels, and Close waits (bounded by
  meshShutdownTimeout, since a pass can be waiting on an unreachable peer) for a
  pass in flight to return. A pass can no longer dial or create streams after
  the mesh Conns close or the store does. `hasStream` and `streamMaxAge` read
  `s.store` under `s.lock` — the lock Close nils it under — the same rule
  `acquire` documents.
- SUGGESTION (replicate.go:162): the `confirmed` record is written only for a
  stream this node holds, so `confirmed` and `replicas` stay proportional to the
  streams held here rather than to every stream a peer has ever listed.
- SUGGESTION (replicate.go:393): a resident stream whose first open is in flight
  answers existence and its retention window from stream.meta again, so a
  client's `Conn.Stream` does not transiently report the stream missing during a
  concurrent open, and a mesh node never creates a copy with a zero window.

Tests: TestCloseStopsReconciliation (Close halts the loop, a later pass is
inert), TestCloseDuringReconciliation (passes in flight while Close runs, the
path the store race lives on), TestStreamThatCannotBeAdoptedIsNotConfirmed,
TestGetStreamWhileFirstOpenIsInFlight.
Author
Member

Review 28 resolved in 2ced954 (pushed to feat/mesh-internal-namespace). All four findings addressed; each fix carries a test that fails without it except the store-capture one, which is covered by an explicitly concurrent test rather than an assertion.

🟠 server.go:383 — reconcile loop not stopped by Close. Fixed: NewServer gives the loop its own child context (loopCtx, stop := context.WithCancel(ctx), stored as meshPeers.reconcileStop), and Close cancels it first, before decommission and before the mesh Conns or the store are touched. Close waits for a pass in flight (meshPeers.reconcileDone, closed by reconcileLoop's defer) with a bound (meshShutdownTimeout = 2s) because a pass can be waiting on an unreachable peer; a pass that outlives the window only reaches a closed store, whose operations fail rather than corrupt. Tests: TestCloseStopsReconciliation (Close returns with the loop's done channel closed, and a later pass on a closed replica adopts nothing) and TestCloseDuringReconciliation.

🟠 replicate.go:382 — data race on s.store. Fixed both sides: reconcileStop/reconcileDone above stop the loop's own passes before the store closes, and hasStream/streamMaxAge now capture s.store into a local under s.lock — the same lock Close nils it under — before any nil check or use, matching the rule acquire documents. TestCloseDuringReconciliation drives 50 passes concurrently with Close, then one after it; it runs clean under racedetector test -run 'TestCloseDuringReconciliation|...' ./... (0 races in the phase-2 test set, no frames in replicate.go/discovery_test.go).

🟡 replicate.go:162 — unbounded confirmed/replicas maps. Fixed by inverting the order: the pass now calls ensureReplica first and records confirm only when s.hasStream(name) holds afterwards. A stream this node never adopted has no copy to retract, so recording it served no purpose but growth; both maps are now bounded by the streams this node holds (itself bounded by maxStreamsPerServer), and entries still leave via forget on deleteLocalStream. Test: TestStreamThatCannotBeAdoptedIsNotConfirmed — a durable peer holds a stream whose stream.meta is removed after eviction, so its window is unreadable, adoption fails, and the test asserts neither map gained an entry (and the replica holds nothing).

🟡 replicate.go:393 — getStream transiently 404s during a first open. Fixed by reading the window from the directory while a resident stream has no in-memory state yet: streamMaxAge now falls through to store.streamMaxAge(name) for the loading case (a mount writes stream.meta before the stream is materialized) and only reports false when the stream is absent or the mount has not written the window yet. A client's Conn.Stream no longer drops its cached handle during a concurrent open, and a mesh node reads the real window instead of a zero one. Test: TestGetStreamWhileFirstOpenIsInFlight plants the loading marker the way createStream reserves a name and asserts the existence check succeeds with the created window.

Verification on 2ced954: go build ./..., go vet ./..., go test ./... — 8 consecutive clean full-suite runs; golangci-lint run reports no findings in the changed hunks (the branch's remaining findings are the same set main reports).

Review 28 resolved in `2ced954` (pushed to `feat/mesh-internal-namespace`). All four findings addressed; each fix carries a test that fails without it except the store-capture one, which is covered by an explicitly concurrent test rather than an assertion. **🟠 server.go:383 — reconcile loop not stopped by `Close`.** Fixed: `NewServer` gives the loop its own child context (`loopCtx, stop := context.WithCancel(ctx)`, stored as `meshPeers.reconcileStop`), and `Close` cancels it first, before decommission and before the mesh Conns or the store are touched. `Close` waits for a pass in flight (`meshPeers.reconcileDone`, closed by `reconcileLoop`'s `defer`) with a bound (`meshShutdownTimeout = 2s`) because a pass can be waiting on an unreachable peer; a pass that outlives the window only reaches a closed store, whose operations fail rather than corrupt. Tests: `TestCloseStopsReconciliation` (Close returns with the loop's done channel closed, and a later pass on a closed replica adopts nothing) and `TestCloseDuringReconciliation`. **🟠 replicate.go:382 — data race on `s.store`.** Fixed both sides: `reconcileStop`/`reconcileDone` above stop the loop's own passes before the store closes, and `hasStream`/`streamMaxAge` now capture `s.store` into a local under `s.lock` — the same lock `Close` nils it under — before any `nil` check or use, matching the rule `acquire` documents. `TestCloseDuringReconciliation` drives 50 passes concurrently with `Close`, then one after it; it runs clean under `racedetector test -run 'TestCloseDuringReconciliation|...' ./...` (0 races in the phase-2 test set, no frames in `replicate.go`/`discovery_test.go`). **🟡 replicate.go:162 — unbounded `confirmed`/`replicas` maps.** Fixed by inverting the order: the pass now calls `ensureReplica` *first* and records `confirm` only when `s.hasStream(name)` holds afterwards. A stream this node never adopted has no copy to retract, so recording it served no purpose but growth; both maps are now bounded by the streams this node holds (itself bounded by `maxStreamsPerServer`), and entries still leave via `forget` on `deleteLocalStream`. Test: `TestStreamThatCannotBeAdoptedIsNotConfirmed` — a durable peer holds a stream whose `stream.meta` is removed after eviction, so its window is unreadable, adoption fails, and the test asserts neither map gained an entry (and the replica holds nothing). **🟡 replicate.go:393 — `getStream` transiently 404s during a first open.** Fixed by reading the window from the directory while a resident stream has no in-memory state yet: `streamMaxAge` now falls through to `store.streamMaxAge(name)` for the loading case (a mount writes `stream.meta` before the stream is materialized) and only reports false when the stream is absent or the mount has not written the window yet. A client's `Conn.Stream` no longer drops its cached handle during a concurrent open, and a mesh node reads the real window instead of a zero one. Test: `TestGetStreamWhileFirstOpenIsInFlight` plants the loading marker the way `createStream` reserves a name and asserts the existence check succeeds with the created window. Verification on `2ced954`: `go build ./...`, `go vet ./...`, `go test ./...` — 8 consecutive clean full-suite runs; `golangci-lint run` reports no findings in the changed hunks (the branch's remaining findings are the same set `main` reports).
sam merged commit 340af3c2cf into main 2026-09-26 16:43:42 +00:00
sam deleted branch feat/mesh-internal-namespace 2026-09-26 16:43:42 +00:00
Sign in to join this conversation.
No reviewers
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
norobo/webstream!7
No description provided.