docs: server mesh replication plan #2

Merged
sam merged 7 commits from feature/server-mesh-replication into main 2026-09-26 11:42:40 +00:00
Member

Docs-only branch: no Go code changes. The diff is plan.md, README.md and doc.go.

plan.md previously held the single-node server-persistence plan, which is delivered and merged (PR #1, 74c236d), including the Phase 5 publish benchmarks (PR #3, 6b073ef). It is replaced here with the plan for the next milestone: server mesh stream replication. README.md and doc.go are corrected where they still described the pre-persistence state.

Scope

R = N for any node count N >= 1: the replication factor equals the number of server nodes, so every node holds every stream and every stream has exactly as many replicas as there are nodes. Counts 1, 2 and 3 are all in scope, and are the counts the acceptance tests exercise:

  • N = 1 — one replica per stream. Today's server: no mesh configured, no replication goroutine, no forwarding.
  • N = 2 — two replicas per stream, one per node.
  • N = 3 — three replicas per stream, one per node.

Deferred explicitly: placement for R < N (which nodes hold which streams), leader failover, dynamic mesh membership, and the mesh-independent items — auth (libp2p capability envelopes) and the browser/WebTransport client.

Design decisions

One writer per stream, and the writer is derived, not stored.
leader(stream) = argmin hash(streamName, peerID) over the configured node set. A pure function, so every node agrees without state or negotiation, and R < N later changes only the candidate set.

Writes are forwarded, one hop. A mutation that lands on a non-leader is proxied over that node's own peer connection. Clients never need to discover the leader and never see a wrong-node error. Reads are never forwarded.

Records keep the leader's sequence numbers verbatim. With one writer per stream, a sequence number is globally unique for that stream, so replicas share one address space with the leader (base, next, readAt(seq), every consumer watermark) and Msg.Seq stays meaningful when a client changes node.

Webstream replicates webstream. Each user stream <name> gets a hidden companion stream internal-<name>, written only by the leader, holding an ordered command log: Append{seq, ts, data}, Trim{base}, ConsumerUpsert{name, processedBefore}, ConsumerDelete{name}. Every replica consumes it as an ordinary webstream client consumer and applies the commands to its own copy. Replica sync therefore inherits InterestPolicy retention (a lagging replica pins the log), the durable acked watermark (its own sync position) and the consume lease (replica liveness) instead of reinventing them. A replica keeps no copy of the command log, which is also why replication cannot recurse into internal-internal-....

Consumer state is stream state. Watermarks propagate on the command log, and the consume lease is leader-arbitrated, so one consumer name has one live loop across the mesh and a client that reconnects to a different node resumes at its watermark rather than reprocessing. These were the two gaps the previous draft had deferred; they are requirements now.

No clock alignment is needed. libp2p synchronizes nothing (its only clock-adjacent dependencies are local-clock abstractions for testable timers), and host clocks skew. Because each stream has one writer, the leader's append time is the only ingress stamp a record ever gets, so every timestamp in a stream comes from one clock and every sequence from one counter. Clients never supply a timestamp, and replicas never compare their wall clock to a record's — they follow the leader's retained base via Trim{base} commands.

Strict sequential delivery is unchanged. One record in flight, the cursor advances only on a confirmed ack, and advanceIf's strict-equality ack check and the scalar processedBefore watermark are untouched. Because the leader orders everything, a replica has an order to be sequential about: no ordering key, client-side reorder buffer or hole-aware progress structure is required.

Wire delta is deliberately small

  • frameForward — one additive frame kind wrapping an inner request plus a forwarded marker, so any mutation can be forwarded without touching the encoding of existing kinds, and a membership disagreement (a stale peer set during a rolling restart) produces a loud loop error instead of infinite recursion.
  • A reserved internal- stream-name prefix, rejected by CreateStream and filtered out of ListStreams.
  • maxAge added to the frameGetStream reply, so a node discovering a peer's stream creates it with the leader's retention configuration.

Replication itself needs no new frame kind: a replica reads the command log with framePull and confirms with frameAck. streamProtocol stays /norobo/webstream/1.0.0.

Costs this design accepts

  • The leader writes each record twice — once to <name>, once to internal-<name>. One extra copy, not one per replica, since all replicas consume the same command log. The alternative (a command pointing at a sequence range that replicas read directly) removes the copy but requires a second pull loop per replica.
  • A leader outage stops writes to its streams. Reads continue on every replica. Failover needs a liveness rule; a stale live set would produce two leaders and diverging logs.
  • Command logs count against the storage budget and are pinned by replica interest, so a stopped replica holds its leader's command log until its lease lapses.

Phases

  1. Mesh peer set, server-as-client, derived leader, one-hop forwarding (with loop guard).
  2. Reserved internal stream namespace and stream discovery (conservative polled reconciliation; never auto-deletes).
  3. The replication command log for records: ordered, idempotent apply, leader-driven Trim, no age loop on replicas, per-batch residency.
  4. Consumer state replication and mesh-wide lease arbitration.
  5. Multi-node acceptance for N = 2 and N = 3, failure cases (leader restart, leader outage, replica budget) and benchmarks for the forwarding hop, replication throughput, backfill and tail latency. N = 1 is the regression suite and is unchanged.

Decisions worth challenging in review

  1. leader = argmin hash(streamName, peerID) rather than min(peerID) — the latter would put every stream's leader on the lowest-ID node.
  2. Extra copy on the leader vs. the pointer-into-<name> variant.
  3. Polled reconciliation for stream discovery vs. a cluster-state command log (same dogfooding idea applied to metadata), which would retire the polling.

Also fixed in this branch (documentation accuracy)

  • README.md claimed maxNameSize was 1 KiB; it is 64 bytes.
  • README.md said "compaction is next". It is not needed: trimming only ever advances the retained base, and journal.trimTo deletes whole sealed segments below it, so the log never develops interior holes.
  • README.md / doc.go described the server as an early in-memory implementation with deferred retention and Phase 1/2 numbering from the superseded plan.

Verification

Docs and comments only, so the Go suite is unaffected: go build ./... and go vet ./... both pass, and gofmt -l . is clean. The full phase-by-phase verification block is in plan.md under "Verification".

Docs-only branch: **no Go code changes**. The diff is `plan.md`, `README.md` and `doc.go`. `plan.md` previously held the single-node server-persistence plan, which is delivered and merged (PR #1, `74c236d`), including the Phase 5 publish benchmarks (PR #3, `6b073ef`). It is replaced here with the plan for the next milestone: **server mesh stream replication**. `README.md` and `doc.go` are corrected where they still described the pre-persistence state. ## Scope **`R = N` for any node count `N >= 1`**: the replication factor equals the number of server nodes, so every node holds every stream and every stream has exactly as many replicas as there are nodes. Counts 1, 2 and 3 are all in scope, and are the counts the acceptance tests exercise: - **`N = 1`** — one replica per stream. Today's server: no mesh configured, no replication goroutine, no forwarding. - **`N = 2`** — two replicas per stream, one per node. - **`N = 3`** — three replicas per stream, one per node. Deferred explicitly: placement for `R < N` (which nodes hold which streams), leader failover, dynamic mesh membership, and the mesh-independent items — auth (libp2p capability envelopes) and the browser/WebTransport client. ## Design decisions **One writer per stream, and the writer is derived, not stored.** `leader(stream) = argmin hash(streamName, peerID)` over the configured node set. A pure function, so every node agrees without state or negotiation, and `R < N` later changes only the candidate set. **Writes are forwarded, one hop.** A mutation that lands on a non-leader is proxied over that node's own peer connection. Clients never need to discover the leader and never see a wrong-node error. Reads are never forwarded. **Records keep the leader's sequence numbers verbatim.** With one writer per stream, a sequence number is globally unique for that stream, so replicas share one address space with the leader (`base`, `next`, `readAt(seq)`, every consumer watermark) and `Msg.Seq` stays meaningful when a client changes node. **Webstream replicates webstream.** Each user stream ``<name>`` gets a hidden companion stream ``internal-<name>``, written only by the leader, holding an ordered command log: `Append{seq, ts, data}`, `Trim{base}`, `ConsumerUpsert{name, processedBefore}`, `ConsumerDelete{name}`. Every replica consumes it as an **ordinary webstream client consumer** and applies the commands to its own copy. Replica sync therefore inherits `InterestPolicy` retention (a lagging replica pins the log), the durable acked watermark (its own sync position) and the consume lease (replica liveness) instead of reinventing them. A replica keeps no copy of the command log, which is also why replication cannot recurse into `internal-internal-...`. **Consumer state is stream state.** Watermarks propagate on the command log, and the consume lease is leader-arbitrated, so one consumer name has one live loop across the mesh and a client that reconnects to a different node resumes at its watermark rather than reprocessing. These were the two gaps the previous draft had deferred; they are requirements now. **No clock alignment is needed.** libp2p synchronizes nothing (its only clock-adjacent dependencies are local-clock abstractions for testable timers), and host clocks skew. Because each stream has one writer, the leader's append time is the only ingress stamp a record ever gets, so every timestamp in a stream comes from one clock and every sequence from one counter. Clients never supply a timestamp, and replicas never compare their wall clock to a record's — they follow the leader's retained base via `Trim{base}` commands. **Strict sequential delivery is unchanged.** One record in flight, the cursor advances only on a confirmed ack, and `advanceIf`'s strict-equality ack check and the scalar `processedBefore` watermark are untouched. Because the leader orders everything, a replica has an order to be sequential about: no ordering key, client-side reorder buffer or hole-aware progress structure is required. ## Wire delta is deliberately small - `frameForward` — one additive frame kind wrapping an inner request plus a forwarded marker, so any mutation can be forwarded without touching the encoding of existing kinds, and a membership disagreement (a stale peer set during a rolling restart) produces a loud loop error instead of infinite recursion. - A reserved `internal-` stream-name prefix, rejected by `CreateStream` and filtered out of `ListStreams`. - `maxAge` added to the `frameGetStream` reply, so a node discovering a peer's stream creates it with the leader's retention configuration. Replication itself needs **no new frame kind**: a replica reads the command log with `framePull` and confirms with `frameAck`. `streamProtocol` stays `/norobo/webstream/1.0.0`. ## Costs this design accepts - **The leader writes each record twice** — once to `<name>`, once to `internal-<name>`. One extra copy, not one per replica, since all replicas consume the same command log. The alternative (a command pointing at a sequence range that replicas read directly) removes the copy but requires a second pull loop per replica. - **A leader outage stops writes to its streams.** Reads continue on every replica. Failover needs a liveness rule; a stale live set would produce two leaders and diverging logs. - **Command logs count against the storage budget** and are pinned by replica interest, so a stopped replica holds its leader's command log until its lease lapses. ## Phases 1. Mesh peer set, server-as-client, derived leader, one-hop forwarding (with loop guard). 2. Reserved internal stream namespace and stream discovery (conservative polled reconciliation; never auto-deletes). 3. The replication command log for records: ordered, idempotent apply, leader-driven `Trim`, no age loop on replicas, per-batch residency. 4. Consumer state replication and mesh-wide lease arbitration. 5. Multi-node acceptance for `N = 2` and `N = 3`, failure cases (leader restart, leader outage, replica budget) and benchmarks for the forwarding hop, replication throughput, backfill and tail latency. `N = 1` is the regression suite and is unchanged. ## Decisions worth challenging in review 1. `leader = argmin hash(streamName, peerID)` rather than `min(peerID)` — the latter would put every stream's leader on the lowest-ID node. 2. Extra copy on the leader vs. the pointer-into-`<name>` variant. 3. Polled reconciliation for stream discovery vs. a cluster-state command log (same dogfooding idea applied to metadata), which would retire the polling. ## Also fixed in this branch (documentation accuracy) - `README.md` claimed `maxNameSize` was 1 KiB; it is 64 bytes. - `README.md` said "compaction is next". It is not needed: trimming only ever advances the retained base, and `journal.trimTo` deletes whole sealed segments below it, so the log never develops interior holes. - `README.md` / `doc.go` described the server as an early in-memory implementation with deferred retention and Phase 1/2 numbering from the superseded plan. ## Verification Docs and comments only, so the Go suite is unaffected: `go build ./...` and `go vet ./...` both pass, and `gofmt -l .` is clean. The full phase-by-phase verification block is in `plan.md` under "Verification".
The server-persistence plan is delivered (merged as PR #1), so plan.md now
specifies the next milestone: replicating streams across a libp2p server mesh.

- plan.md: phased mesh replication plan scoped to R = N (1 node, 3 nodes).
  Owner = lowest peer ID over the replica set; writes are owner-only; a replica
  reuses the owner's sequence numbers so its log.next is its derived replication
  cursor and no new on-disk file, timestamp or dedup ID is needed on the wire.
  Adds frameReplicate (bounded range read, blocking tail), replica retention
  leases, and one new frame kind; placement for R < N, request forwarding,
  durable replica cursors and owner failover are deferred.
- README.md: drop the stale in-memory/compaction/phase claims, record that
  compaction is unnecessary because trimming only advances the retained base and
  trimTo deletes whole sealed segments, and correct the name cap (maxNameSize is
  64 bytes, not 1 KiB).
- doc.go: drop the 'early, in-memory implementation' preamble.
Adopts the decisions taken in review: writes stay single-writer per stream with a
derived, placed leader and one-hop server-side forwarding, the replication factor
is R = N for now, and consumers keep strict sequential delivery.

- plan.md: replace the frameReplicate range-read design with a hidden command log
  per user stream (internal-<name>) that every replica consumes as an ordinary
  webstream client. One ordered log carries records, trims and consumer state, so
  replica sync inherits InterestPolicy retention, the durable acked watermark and
  the consume lease instead of reinventing them; a replica keeps no copy of the
  command log, which is also why replication cannot recurse.
- plan.md: consumer state is stream state. Watermarks propagate as commands and
  the consume lease is leader-arbitrated, closing both gaps the previous draft
  deferred (mesh-wide single-loop, and a client resuming on another node without
  reprocessing). Delivery stays strictly sequential, so processedBefore,
  advanceIf and Msg.Seq are unchanged.
- plan.md: leader(stream) = argmin hash(streamName, peerID) over the node set,
  rather than min(peerID), so leaders spread across nodes; the rule is a
  placement function already, so R < N does not change it.
- plan.md: one clock per stream means no clock alignment, since libp2p
  synchronizes nothing and a single writer stamps each record once at the leader.
  Replicas follow the leader's base via Trim commands instead of their own clocks.
- plan.md: the wire delta shrinks to one additive kind (frameForward with a loop
  guard), a reserved internal- stream-name prefix, and maxAge on the getStream
  reply; replication itself needs no new frame.
- README.md: drop the branch name from the pointer to plan.md.
The scope section pinned the iteration to two acceptance configurations, 1 node
and 3 nodes. That was too narrow: the rule is R = N, so 2 nodes with 2 replicas
per stream is equally in scope, and so is any other node count. Restates the
scope as R = N for any N >= 1, listing 1, 2 and 3 as the counts the acceptance
tests exercise, and renames Phase 5 from '3-node acceptance' to 'multi-node
acceptance' with a separate N = 2 case.

N = 2 is worth its own acceptance test rather than being treated as a special
case of 3: it is the smallest mesh where a write can land on a node that must
forward it, and the smallest where a stream still has more than one replica.
R < N is confirmed as follow-up work, not part of this iteration, and it should
not be built straight onto the R = N machinery. Placement reintroduces the
questions R = N structurally cannot ask: which nodes hold a stream, what happens
when a holder joins or leaves, and where a request that lands on a non-holder
goes.

- Scope: state the gate, and why the holder-set and forwarding rules get
  prototyped before they get a design. The leader rule stays a pure function of
  the node set so that prototype does not have to change it.
- Deferred: note that R = N is not a special case of R < N that a parameter
  generalizes, since every-node-holds-everything needs no holder set, no
  rebalancing and no request that lands on a non-holder.
- Up next: order the proof of concept first (holder-set function, a holder
  joining and leaving, a request landing on a non-holder), then failover,
  cluster metadata, auth and the browser client.
pi-agent left a comment

Review — server mesh replication plan

Artifact reviewed: plan.md at head 0655d46, branch feature/server-mesh-replication (this PR, base main 6b073ef). Docs-only, so no build/test failure is implied by anything below. The question this review answers is whether the plan, implemented exactly as written, delivers the two guarantees it claims: at-least-once delivery from the server and exactly-once semantics for the client through deduplication.

Scope I read the plan as claiming: the first iteration is R = N (replication factor equals node count, so every node holds every stream) for N ∈ {1, 2, 3}, with placement for R < N, leader failover, and dynamic membership deferred. That matches the PR body, and the N = 1 regression framing is right.

Verdict

The shape is correct — one writer per stream, forwarded mutations, and a replication command log that is itself a webstream stream are the right three decisions, and they are what keep Msg.Seq and processedBefore meaningful across nodes without clocks or ordering keys. But the plan does not yet guarantee what it claims, because of four holes below. B1 breaks at-least-once permanently; B2 and B3 break exactly-once processing in the cases the plan itself asserts as tests. These are design-level, not wording-level: they change the ordering of the write path and two entries in the "reads are never forwarded" rule.

Claims that check out against the merged code

  • Producer deduplication survives forwarding: Server.publish in server.go calls streamState.appendDedup in state.go, which is the only dedup authority, and a forwarded publish lands there.
  • Consumer state really is a good fit for replication: processedBefore is already the durable acked watermark in the consumer slot store, streamState.computeKeepLocked / minConsumerCursorLocked already derive the retained base from it, and consumerState.advanceIf already does the strict-equality cursor move in one critical section.
  • Sequence numbers really do not need a new global ID: journal.base and journal.next are derived from the segment set (store.go), and journal.rollLocked uses rec.seq as the new segment base, so a replica's own roll boundaries are legal and recoverable.
  • The plan is right that libp2p provisions no cross-node clock; see benbjohnson/clock and filecoin-project/go-clock in go.mod, both local-clock abstractions.

Guarantees — please state these explicitly in the plan

The plan currently describes mechanisms but never states, in one place, which mechanism enforces which guarantee and where. Given that the review question is exactly this, add a Guarantees section naming:

  1. At-least-once (server → client). A producer-acked publish is durably present on the leader and applied by every replica (this is what B1 and gap N1 break); a pull advances the cursor only on a confirmed ack (consumerState.advanceIf, unchanged). A redelivery after a lost ack is the documented at-least-once edge.
  2. Exactly-once producer storage. The leader-local streamState.dedup ID→sequence table, retention-bounded (streamState.dropDedupLocked). B4 is required for this to hold on a replica whose durable state is recovered or that later becomes the leader.
  3. Exactly-once client processing. For a live client handle, the client-local watermark in consumer.Consume (raised monotonically by raiseWatermark, never lowered) already covers a node move, because sequence numbers are shared. For a restarted handle the guarantee comes from the replicated watermark read back by Stream.Consumer — that path is B2 and it currently does not hold.

Blocking findings

B1. No crash atomicity between <name> and internal-<name>; this loses data permanently

The plan says the leader appends the Append command "in the same critical section as the user append". A mutex gives ordering, not crash atomicity. Server.publish holds journal.mu (the per-stream writer lock in store.go) across the in-memory append and the durable write to streams/<name>/log/…. The command write is a second durable write to streams/internal-<name>/log/…. A crash between the two fsyncs leaves a record that is durable on the leader and that no replica will ever be told about.

The outcome is worse than a stall, because both plausible apply primitives handle the resulting hole wrongly:

  • memlog.appendRecovered in memlog.go does if rec.seq > log.next { log.next = rec.seq } — it silently adopts gaps. An applier built on it would continue past the missing sequence and diverge; a client pinned to that replica never receives that record, so at-least-once is violated there, not merely delayed.
  • An applier that instead enforces cmd.seq == log.next (which is what the plan's "apply only at its own log.next" implies) stops permanently: every later command is dropped, and the replica never recovers without an explicit resync path, which the plan does not define.

Fix: make internal-<name> the write-ahead log and derive the leader's own copy from it. Order becomes: (1) append and fsync the Append command; (2) apply it to the leader's own <name> through the same applier a replica uses. Then a crash before (1) loses both copies (and the publish never acked, so no client promise is broken), and a crash after (1) is repaired by replaying the leader's own command log from its own apply position (<name>'s log.next) at startup. This also makes the plan's "every timestamp comes from one clock" survive, because both paths store the timestamp the command carries rather than minting one.

B2. Consumer-metadata reads are not read-your-writes, so the Phase 4 headline test cannot pass

The plan classifies frameGetConsumer as a read and says reads are never forwarded. But Stream.Consumer in stream.go seeds the client's watermark from Server.getConsumer in server.go, which answers from local state. Sequence: client acks on node A (the ack is forwarded to the leader, which writes ConsumerUpsert{name, processedBefore}); the client then restarts and calls Stream.Consumer on node B before B's applier has consumed that command; B returns a stale processedBefore; the client re-runs handlers it already completed. That is at-least-once processing, not the exactly-once the plan's Phase 4 test asserts ("reconnects to a different node and continues exactly where it stopped, without reprocessing") and that doc.go promises ("a restarted client resumes where its predecessor stopped").

This is specifically the new-handle case: with a live handle the client-local watermark already skips redeliveries across a node move.

Fix: forward frameGetConsumer to the leader. It is symmetric with the plan already forwarding consumer mutations, and it keeps the record data path local, so "reads are never forwarded" becomes "record reads are never forwarded" — a rule that is both true and enforceable. Gating the pull on the replica's replicated watermark being at least the ack it forwarded is the alternative, but it is strictly more machinery for the same guarantee.

B3. The mesh-wide lease has a window in which two consume loops both run

consumerState.acquireLease in state.go is called on every pull today, and that is what makes the single-loop invariant airtight: the lease is the only thing preventing two loops from serving the same cursor record concurrently (Server.preparePull reads at the consumer's cursor without tracking in-flight delivery, so two admitted loops both run the handler for the same sequence; the loser's ack then fails ErrAckOutOfOrder). The plan replaces the per-pull check with a local lease view refreshed by a heartbeat "well inside consumerLeaseTimeout" (30 s, server.go). Between the leader granting or renewing the lease and a replica receiving that grant on the command log, the replica's view is stale, so a second loop on a third node is admitted. A heartbeat bounds liveness; it cannot bound a stale view, so this is a correctness window, not a tuning question.

Fix — pick one and state the consequence of the choice:

  • (a) Keep the lease check linearizable by forwarding it to the leader on every pull, while the record read stays local. This costs one extra hop per delivered record. It is the honest price of the single-loop invariant, and the plan's benchmarks already cover forwarded-request cost.
  • (b) Keep the heartbeat and explicitly weaken the claim: exactly-once processing becomes conditional on no second loop being started for the same consumer name within the stale-view window. Then the Phase 4 ErrConsumerBusy test becomes "eventually refused", and doc.go's single-loop paragraph must carry the same qualification.

Claiming both the heartbeat and the ErrConsumerBusy guarantee is the one option that is not available.

B4. Append{seq, ts, data} omits dedupID, and neither existing append primitive can be the applier

The on-disk record frame is seq, timestamp, dedupID, data (store.go encodeRecordFrame / decodeRecord), and streamState.rebuildDedup reconstructs the dedup table from the retained records on recovery. A replica applying Append without the ID stores a different record than the leader and recovers an empty dedup table, so a replica that later becomes the leader (the deferred failover) silently loses the producer exactly-once window. Add dedupID to Append.

Related, and needed for the fix in B1: neither existing in-memory append primitive is usable for the applier as-is. memlog.appendOwned mints timestamp: time.Now() and assigns log.next itself, which would break the plan's own "timestamps replicated verbatim / one clock per stream" invariant; memlog.appendRecovered takes an exact record but silently adopts gaps (see B1). The plan needs a strict applyRecord(record) that (i) rejects rec.seq != log.next, (ii) stores the given seq and timestamp, and (iii) registers rec.dedupID in streamState.dedup exactly as appendOwnedLocked does. memlog.go should therefore appear in the plan's "Files touched" list — it is not there today.

Spec gaps that are cheap to close now

  • N1 — command-log trimming can silently skip commands a replica needs. Server.enforceStorageBudget → store.budgetSnapshot walks every directory under streams/ with no name filter, so internal-<name> segments are budget-eligible via store.deleteSegment, advancing the command log's base past a lagging replica. For a user stream, skipping ahead is the documented DiscardOld behaviour; for the command log it means that replica permanently lacks records — an at-least-once violation for clients pinned there. Define one resync(stream) path from the leader's retained window and use it for both this case and the fresh-replica case, and say explicitly what happens to a replica that falls below the command-log base (mark failed, re-register, re-backfill).
  • N2 — the "stopped replica pins the command log until its lease lapses" sentence is incorrect. Interest trimming reads streamState.minConsumerCursorLocked → consumerStore.minLiveWatermark in store.go, which is watermark-based. A lapsed lease frees the consumer for a replacement loop but does not remove it or advance its watermark, so it pins the command log until the consumer is deleted or a DiscardOld trim passes it. Either delete the replica's consumer on decommission, or make the discard path of N1 responsible.
  • N3 — apply Trim{base} monotonically. The plan's own Phase 5 test has a replica evicting its oldest segments under its own storage budget, so a replica's base can legitimately be ahead of the leader's. max(replicaBase, cmd.base), ignoring a stale command; a set would attempt to un-trim.
  • N4 — the internal- reservation is only asserted for CreateStream and ListStreams. There is no authentication in this protocol (doc.go states so), so a peer that guesses internal-<name> can CreateConsumer/Pull on it and pin the leader's command log, a cheap growth vector. Apply the prefix check in Server.dispatch for every client-facing kind (publish, pull, ack, create/delete stream and consumer, get stream/consumer), and put it there rather than deep inside createStream, so the replica's own consumer on the leader's internal-<name> bypasses it. State that the prefix is namespace hygiene, not a security boundary.
  • N5 — replica consumer identity and lifecycle. "The replica registers as a consumer of internal-<name>" needs a distinct name per replica (for example rep-<peerID>); two replicas sharing one name share a watermark and compete for the lease. Also specify the graceful-decommission delete.
  • N6 — WithMeshPeers semantics. State that the configured set includes self, with a deterministic tie-break on hash collisions (compare peer-ID bytes): a node that omits itself computes a leader it will never agree with, and then meets the loop guard.
  • N7 — two protocol details the plan understates. Server.getStream currently returns a bare response{} with no fields, so maxAge on the frameGetStream reply is a real addition and belongs in the "Protocol additions" list. And listStreams is paginated (listStreamsRequest.cursor/limit, Server.listStorePage), so reconciliation must page rather than assume one reply.
  • N8 — concurrent applier and local client on one consumer. Specify that a replicated ConsumerUpsert is applied monotonically (never lowering processedBefore); a move that lowers it would make a local live loop's advanceIf fail ErrAckOutOfOrder.

Tests that would actually catch these

Add fault-injection cases to Phase 3/4/5; a naive implementation passes the current list:

  1. Close the store between the leader's user append and the command append, reopen, and assert no divergence and no permanent replica stall (B1).
  2. Force the command log's base past a replica's position via the storage budget, and assert the resync path fires and the retained window matches the leader's (N1).
  3. Restart a client and call Stream.Consumer on a lagging replica; assert no reprocessing (B2).
  4. Start two loops for one consumer name on two different nodes; assert whichever behaviour the B3 decision commits to.
  5. Apply a duplicate Append and a Trim older than the replica's base; assert strict rejection and no base regression (B4, N3).

The per-line comments below carry the same findings anchored to the text they apply to.

# Review — server mesh replication plan **Artifact reviewed:** `plan.md` at head `0655d46`, branch `feature/server-mesh-replication` (this PR, base `main` `6b073ef`). Docs-only, so no build/test failure is implied by anything below. The question this review answers is whether the plan, implemented exactly as written, delivers the two guarantees it claims: **at-least-once delivery from the server** and **exactly-once semantics for the client through deduplication**. **Scope I read the plan as claiming:** the first iteration is `R = N` (replication factor equals node count, so every node holds every stream) for `N ∈ {1, 2, 3}`, with placement for `R < N`, leader failover, and dynamic membership deferred. That matches the PR body, and the `N = 1` regression framing is right. ## Verdict The shape is correct — one writer per stream, forwarded mutations, and a replication command log that is itself a webstream stream are the right three decisions, and they are what keep `Msg.Seq` and `processedBefore` meaningful across nodes without clocks or ordering keys. But the plan does not yet guarantee what it claims, because of four holes below. **B1 breaks at-least-once permanently; B2 and B3 break exactly-once processing in the cases the plan itself asserts as tests.** These are design-level, not wording-level: they change the ordering of the write path and two entries in the "reads are never forwarded" rule. ## Claims that check out against the merged code - Producer deduplication survives forwarding: `Server.publish` in `server.go` calls `streamState.appendDedup` in `state.go`, which is the only dedup authority, and a forwarded publish lands there. - Consumer state really is a good fit for replication: `processedBefore` is already the durable acked watermark in the consumer slot store, `streamState.computeKeepLocked` / `minConsumerCursorLocked` already derive the retained base from it, and `consumerState.advanceIf` already does the strict-equality cursor move in one critical section. - Sequence numbers really do not need a new global ID: `journal.base` and `journal.next` are derived from the segment set (`store.go`), and `journal.rollLocked` uses `rec.seq` as the new segment base, so a replica's own roll boundaries are legal and recoverable. - The plan is right that libp2p provisions no cross-node clock; see `benbjohnson/clock` and `filecoin-project/go-clock` in `go.mod`, both local-clock abstractions. ## Guarantees — please state these explicitly in the plan The plan currently describes mechanisms but never states, in one place, which mechanism enforces which guarantee and where. Given that the review question is exactly this, add a **Guarantees** section naming: 1. **At-least-once (server → client).** A producer-acked publish is durably present on the leader *and* applied by every replica (this is what B1 and gap N1 break); a pull advances the cursor only on a confirmed ack (`consumerState.advanceIf`, unchanged). A redelivery after a lost ack is the documented at-least-once edge. 2. **Exactly-once producer storage.** The leader-local `streamState.dedup` ID→sequence table, retention-bounded (`streamState.dropDedupLocked`). B4 is required for this to hold on a replica whose durable state is recovered or that later becomes the leader. 3. **Exactly-once client processing.** For a live client handle, the client-local watermark in `consumer.Consume` (raised monotonically by `raiseWatermark`, never lowered) already covers a node move, because sequence numbers are shared. For a *restarted* handle the guarantee comes from the replicated watermark read back by `Stream.Consumer` — that path is B2 and it currently does not hold. ## Blocking findings ### B1. No crash atomicity between `<name>` and `internal-<name>`; this loses data permanently The plan says the leader appends the `Append` command "in the same critical section as the user append". A mutex gives *ordering*, not crash atomicity. `Server.publish` holds `journal.mu` (the per-stream writer lock in `store.go`) across the in-memory append and the durable write to `streams/<name>/log/…`. The command write is a second durable write to `streams/internal-<name>/log/…`. A crash between the two fsyncs leaves a record that is durable on the leader and that no replica will ever be told about. The outcome is worse than a stall, because both plausible apply primitives handle the resulting hole wrongly: - `memlog.appendRecovered` in `memlog.go` does `if rec.seq > log.next { log.next = rec.seq }` — it **silently adopts gaps**. An applier built on it would continue past the missing sequence and diverge; a client pinned to that replica never receives that record, so at-least-once is violated there, not merely delayed. - An applier that instead enforces `cmd.seq == log.next` (which is what the plan's "apply only at its own `log.next`" implies) stops permanently: every later command is dropped, and the replica never recovers without an explicit resync path, which the plan does not define. **Fix:** make `internal-<name>` the write-ahead log and derive the leader's own copy from it. Order becomes: (1) append and fsync the `Append` command; (2) apply it to the leader's own `<name>` through the *same* applier a replica uses. Then a crash before (1) loses both copies (and the publish never acked, so no client promise is broken), and a crash after (1) is repaired by replaying the leader's own command log from its own apply position (`<name>`'s `log.next`) at startup. This also makes the plan's "every timestamp comes from one clock" survive, because both paths store the timestamp the command carries rather than minting one. ### B2. Consumer-metadata reads are not read-your-writes, so the Phase 4 headline test cannot pass The plan classifies `frameGetConsumer` as a read and says reads are never forwarded. But `Stream.Consumer` in `stream.go` seeds the client's watermark from `Server.getConsumer` in `server.go`, which answers from local state. Sequence: client acks on node A (the ack is forwarded to the leader, which writes `ConsumerUpsert{name, processedBefore}`); the client then restarts and calls `Stream.Consumer` on node B before B's applier has consumed that command; B returns a stale `processedBefore`; the client re-runs handlers it already completed. That is at-least-once processing, not the exactly-once the plan's Phase 4 test asserts ("reconnects to a *different* node and continues exactly where it stopped, without reprocessing") and that `doc.go` promises ("a restarted client resumes where its predecessor stopped"). This is specifically the new-handle case: with a live handle the client-local watermark already skips redeliveries across a node move. **Fix:** forward `frameGetConsumer` to the leader. It is symmetric with the plan already forwarding consumer *mutations*, and it keeps the record data path local, so "reads are never forwarded" becomes "record reads are never forwarded" — a rule that is both true and enforceable. Gating the pull on the replica's replicated watermark being at least the ack it forwarded is the alternative, but it is strictly more machinery for the same guarantee. ### B3. The mesh-wide lease has a window in which two consume loops both run `consumerState.acquireLease` in `state.go` is called on **every** pull today, and that is what makes the single-loop invariant airtight: the lease is the only thing preventing two loops from serving the same cursor record concurrently (`Server.preparePull` reads at the consumer's cursor without tracking in-flight delivery, so two admitted loops both run the handler for the same sequence; the loser's ack then fails `ErrAckOutOfOrder`). The plan replaces the per-pull check with a local lease view refreshed by a heartbeat "well inside `consumerLeaseTimeout`" (30 s, `server.go`). Between the leader granting or renewing the lease and a replica receiving that grant on the command log, the replica's view is stale, so a second loop on a third node is admitted. A heartbeat bounds *liveness*; it cannot bound a stale view, so this is a correctness window, not a tuning question. **Fix — pick one and state the consequence of the choice:** - (a) Keep the lease check linearizable by forwarding it to the leader on every pull, while the record read stays local. This costs one extra hop per delivered record. It is the honest price of the single-loop invariant, and the plan's benchmarks already cover forwarded-request cost. - (b) Keep the heartbeat and explicitly weaken the claim: exactly-once processing becomes conditional on no second loop being started for the same consumer name within the stale-view window. Then the Phase 4 `ErrConsumerBusy` test becomes "eventually refused", and `doc.go`'s single-loop paragraph must carry the same qualification. Claiming both the heartbeat and the `ErrConsumerBusy` guarantee is the one option that is not available. ### B4. `Append{seq, ts, data}` omits `dedupID`, and neither existing append primitive can be the applier The on-disk record frame is `seq, timestamp, dedupID, data` (`store.go` `encodeRecordFrame` / `decodeRecord`), and `streamState.rebuildDedup` reconstructs the dedup table from the retained records on recovery. A replica applying `Append` without the ID stores a different record than the leader and recovers an empty dedup table, so a replica that later becomes the leader (the deferred failover) silently loses the producer exactly-once window. Add `dedupID` to `Append`. Related, and needed for the fix in B1: **neither existing in-memory append primitive is usable for the applier as-is.** `memlog.appendOwned` mints `timestamp: time.Now()` and assigns `log.next` itself, which would break the plan's own "timestamps replicated verbatim / one clock per stream" invariant; `memlog.appendRecovered` takes an exact record but silently adopts gaps (see B1). The plan needs a strict `applyRecord(record)` that (i) rejects `rec.seq != log.next`, (ii) stores the given `seq` and `timestamp`, and (iii) registers `rec.dedupID` in `streamState.dedup` exactly as `appendOwnedLocked` does. `memlog.go` should therefore appear in the plan's "Files touched" list — it is not there today. ## Spec gaps that are cheap to close now - **N1 — command-log trimming can silently skip commands a replica needs.** `Server.enforceStorageBudget` → `store.budgetSnapshot` walks every directory under `streams/` with no name filter, so `internal-<name>` segments are budget-eligible via `store.deleteSegment`, advancing the command log's base past a lagging replica. For a user stream, skipping ahead is the documented `DiscardOld` behaviour; for the command log it means that replica permanently lacks records — an at-least-once violation for clients pinned there. Define one `resync(stream)` path from the leader's retained window and use it for both this case and the fresh-replica case, and say explicitly what happens to a replica that falls below the command-log base (mark failed, re-register, re-backfill). - **N2 — the "stopped replica pins the command log until its lease lapses" sentence is incorrect.** Interest trimming reads `streamState.minConsumerCursorLocked` → `consumerStore.minLiveWatermark` in `store.go`, which is watermark-based. A lapsed lease frees the consumer for a replacement loop but does not remove it or advance its watermark, so it pins the command log until the consumer is deleted or a `DiscardOld` trim passes it. Either delete the replica's consumer on decommission, or make the discard path of N1 responsible. - **N3 — apply `Trim{base}` monotonically.** The plan's own Phase 5 test has a replica evicting its oldest segments under its own storage budget, so a replica's base can legitimately be *ahead* of the leader's. `max(replicaBase, cmd.base)`, ignoring a stale command; a set would attempt to un-trim. - **N4 — the `internal-` reservation is only asserted for `CreateStream` and `ListStreams`.** There is no authentication in this protocol (`doc.go` states so), so a peer that guesses `internal-<name>` can `CreateConsumer`/`Pull` on it and pin the leader's command log, a cheap growth vector. Apply the prefix check in `Server.dispatch` for every client-facing kind (publish, pull, ack, create/delete stream and consumer, get stream/consumer), and put it there rather than deep inside `createStream`, so the replica's own consumer on the leader's `internal-<name>` bypasses it. State that the prefix is namespace hygiene, not a security boundary. - **N5 — replica consumer identity and lifecycle.** "The replica registers as a consumer of `internal-<name>`" needs a distinct name per replica (for example `rep-<peerID>`); two replicas sharing one name share a watermark and compete for the lease. Also specify the graceful-decommission delete. - **N6 — `WithMeshPeers` semantics.** State that the configured set **includes self**, with a deterministic tie-break on `hash` collisions (compare peer-ID bytes): a node that omits itself computes a leader it will never agree with, and then meets the loop guard. - **N7 — two protocol details the plan understates.** `Server.getStream` currently returns a bare `response{}` with no fields, so `maxAge` on the `frameGetStream` reply is a real addition and belongs in the "Protocol additions" list. And `listStreams` is paginated (`listStreamsRequest.cursor`/`limit`, `Server.listStorePage`), so reconciliation must page rather than assume one reply. - **N8 — concurrent applier and local client on one consumer.** Specify that a replicated `ConsumerUpsert` is applied monotonically (never lowering `processedBefore`); a move that lowers it would make a local live loop's `advanceIf` fail `ErrAckOutOfOrder`. ## Tests that would actually catch these Add fault-injection cases to Phase 3/4/5; a naive implementation passes the current list: 1. Close the store between the leader's user append and the command append, reopen, and assert no divergence and no permanent replica stall (B1). 2. Force the command log's base past a replica's position via the storage budget, and assert the resync path fires and the retained window matches the leader's (N1). 3. Restart a client and call `Stream.Consumer` on a lagging replica; assert no reprocessing (B2). 4. Start two loops for one consumer name on two different nodes; assert whichever behaviour the B3 decision commits to. 5. Apply a duplicate `Append` and a `Trim` older than the replica's base; assert strict rejection and no base regression (B4, N3). The per-line comments below carry the same findings anchored to the text they apply to.
plan.md Outdated
@ -124,0 +113,4 @@
- **A forwarded request keeps its payload and its deadline**, bounded by
`defaultWriteTimeout` when the caller's deadline is not near, exactly like a
client request write.
- **Reads are never forwarded.** A pull, list or get is answered from the local
Author
Member

B2 — blocking. This rule cannot include frameGetConsumer. Stream.Consumer in stream.go seeds the client's watermark from Server.getConsumer in server.go, which answers from local state. A client that acks on node A (forwarded to the leader, which writes ConsumerUpsert), restarts, and calls Stream.Consumer on node B before B has applied that command gets a stale processedBefore and re-runs handlers it already completed. That contradicts the Phase 4 test at line 374 and the promise in doc.go that a restarted client resumes where its predecessor stopped. Forward frameGetConsumer to the leader — symmetric with forwarding consumer mutations, and the record data path stays local. Suggest restating the rule as "record reads are never forwarded".

**B2 — blocking.** This rule cannot include `frameGetConsumer`. `Stream.Consumer` in `stream.go` seeds the client's watermark from `Server.getConsumer` in `server.go`, which answers from local state. A client that acks on node A (forwarded to the leader, which writes `ConsumerUpsert`), restarts, and calls `Stream.Consumer` on node B before B has applied that command gets a stale `processedBefore` and re-runs handlers it already completed. That contradicts the Phase 4 test at line 374 and the promise in `doc.go` that a restarted client resumes where its predecessor stopped. Forward `frameGetConsumer` to the leader — symmetric with forwarding consumer mutations, and the record data path stays local. Suggest restating the rule as "record reads are never forwarded".
plan.md Outdated
@ -178,3 +157,1 @@
1 nameLen u8
2 name [64]B
66 cursorSlot u64 // which slot of cursors.meta holds this consumer
Append{seq, ts, data} a record was appended to <name>
Author
Member

B4. This command shape omits dedupID, which the on-disk record frame carries: see encodeRecordFrame / decodeRecord in store.go (fields seq, timestamp, dedupID, data), and streamState.rebuildDedup in state.go, which rebuilds the dedup table from the retained records on recovery. A replica applying this Append stores a record that differs from the leader's and recovers an empty dedup table, so any later leader role for that node silently loses the producer exactly-once window. Add dedupID to the command.

**B4.** This command shape omits `dedupID`, which the on-disk record frame carries: see `encodeRecordFrame` / `decodeRecord` in `store.go` (fields `seq`, `timestamp`, `dedupID`, `data`), and `streamState.rebuildDedup` in `state.go`, which rebuilds the dedup table from the retained records on recovery. A replica applying this `Append` stores a record that differs from the leader's and recovers an empty dedup table, so any later leader role for that node silently loses the producer exactly-once window. Add `dedupID` to the command.
@ -197,2 +167,3 @@
**`cursors.meta`** — the values, written once per ack:
- **A replica's sync position is a real consumer watermark.** The replica
registers as a consumer of the leader's `internal-<name>` through the normal
Author
Member

N5. The replica's consumer name on the leader's internal-<name> must be distinct per replica (e.g. rep-<peerID>): a shared name shares one watermark and puts two replicas in competition for one consume lease. Also specify the graceful-decommission path, which is what actually unpins the command log (see the comment on line 226).

**N5.** The replica's consumer name on the leader's `internal-<name>` must be distinct per replica (e.g. `rep-<peerID>`): a shared name shares one watermark and puts two replicas in competition for one consume lease. Also specify the graceful-decommission path, which is what actually unpins the command log (see the comment on line 226).
plan.md Outdated
@ -199,0 +169,4 @@
registers as a consumer of the leader's `internal-<name>` through the normal
`CreateConsumer` path, so `InterestPolicy` holds commands for a replica that is
behind, and a stopped replica simply stops acking. No new retention logic.
- **Apply is idempotent and ordered.** A replica applies `Append` only at its own
Author
Member

B4 / B1. "Apply only at its own log.next" is right, but neither existing primitive can implement it as-is. memlog.appendOwned in memlog.go mints timestamp: time.Now() and assigns log.next itself, which breaks the plan's own one-clock-per-stream invariant. memlog.appendRecovered takes an exact record but does if rec.seq > log.next { log.next = rec.seq }, i.e. it silently adopts gaps. A new strict applyRecord(record) is required: reject rec.seq != log.next, store the given seq and timestamp, and register rec.dedupID in streamState.dedup as appendOwnedLocked does. memlog.go is missing from the "Files touched" list.

**B4 / B1.** "Apply only at its own `log.next`" is right, but neither existing primitive can implement it as-is. `memlog.appendOwned` in `memlog.go` mints `timestamp: time.Now()` and assigns `log.next` itself, which breaks the plan's own one-clock-per-stream invariant. `memlog.appendRecovered` takes an exact record but does `if rec.seq > log.next { log.next = rec.seq }`, i.e. it **silently adopts gaps**. A new strict `applyRecord(record)` is required: reject `rec.seq != log.next`, store the given `seq` and `timestamp`, and register `rec.dedupID` in `streamState.dedup` as `appendOwnedLocked` does. `memlog.go` is missing from the "Files touched" list.
plan.md Outdated
@ -199,0 +174,4 @@
beyond its position resets it. It holds the journal's writer lock across the
in-memory append *and* the durable write, exactly as `Server.publish` does, or
out-of-order frames reach disk and refuse to load.
- **The leader writes the command in the same critical section as the user
Author
Member

B1 — blocking. A mutex gives ordering, not crash atomicity. Server.publish in server.go holds journal.mu across the in-memory append and the fsync of streams/<name>/log/...; this command is a second durable write to streams/internal-<name>/log/.... A crash between the two leaves a record durable on the leader that no replica is ever told about. Because memlog.appendRecovered adopts gaps silently (see the comment on line 172), an applier would diverge rather than stall, and a client pinned to that replica never receives the record — at-least-once fails, not just exactly-once.

Fix: make internal-<name> the write-ahead log. (1) append + fsync the command, (2) apply it to the leader's own <name> through the same applier a replica uses, so recovery replays the leader's command log from its own apply position. A crash before (1) loses both copies and no publish acked; a crash after (1) is repaired by replay.

**B1 — blocking.** A mutex gives ordering, not crash atomicity. `Server.publish` in `server.go` holds `journal.mu` across the in-memory append and the fsync of `streams/<name>/log/...`; this command is a second durable write to `streams/internal-<name>/log/...`. A crash between the two leaves a record durable on the leader that no replica is ever told about. Because `memlog.appendRecovered` adopts gaps silently (see the comment on line 172), an applier would diverge rather than stall, and a client pinned to that replica never receives the record — at-least-once fails, not just exactly-once. Fix: make `internal-<name>` the write-ahead log. (1) append + fsync the command, (2) apply it to the leader's own `<name>` through the same applier a replica uses, so recovery replays the leader's command log from its own apply position. A crash before (1) loses both copies and no publish acked; a crash after (1) is repaired by replay.
plan.md Outdated
@ -232,0 +201,4 @@
assigned `seq`, the new `processedBefore`, the granted lease) and the receiving
replica applies it immediately. Without that, a replica would redeliver a
record between its local ack and the arrival of the propagated watermark.
- **Lease renewal is a heartbeat, not a per-pull round trip.** A replica holds
Author
Member

B3 — blocking. A heartbeat bounds liveness, not stale reads. Today consumerState.acquireLease in state.go runs on every pull, and that is what makes the single-loop invariant airtight; Server.preparePull serves at the consumer cursor without tracking in-flight delivery, so two admitted loops both run the handler for the same sequence and the loser's ack fails ErrAckOutOfOrder. Between the leader granting or renewing the lease and a replica receiving that grant, the replica's view is stale, so a second loop on a third node is admitted.

Pick one: (a) forward the lease check to the leader on every pull while the record read stays local (one extra hop per record; the honest price), or (b) keep the heartbeat and state explicitly that exactly-once processing is conditional on no second loop starting within the stale-view window — in which case the Phase 4 ErrConsumerBusy test becomes "eventually refused" and doc.go's single-loop paragraph needs the same qualification. Claiming both is not available.

**B3 — blocking.** A heartbeat bounds liveness, not stale reads. Today `consumerState.acquireLease` in `state.go` runs on every pull, and that is what makes the single-loop invariant airtight; `Server.preparePull` serves at the consumer cursor without tracking in-flight delivery, so two admitted loops both run the handler for the same sequence and the loser's ack fails `ErrAckOutOfOrder`. Between the leader granting or renewing the lease and a replica receiving that grant, the replica's view is stale, so a second loop on a third node is admitted. Pick one: (a) forward the lease check to the leader on every pull while the record read stays local (one extra hop per record; the honest price), or (b) keep the heartbeat and state explicitly that exactly-once processing is conditional on no second loop starting within the stale-view window — in which case the Phase 4 `ErrConsumerBusy` test becomes "eventually refused" and `doc.go`'s single-loop paragraph needs the same qualification. Claiming both is not available.
plan.md Outdated
@ -255,0 +223,4 @@
- **A leader outage stops writes to its streams.** Reads continue on every
replica. Failover needs a liveness rule and is deferred; a stale live set would
produce two leaders and diverging logs.
- **Command logs count against the storage budget** and are bounded by replica
Author
Member

N1 and N2. Two corrections. (1) "until its lease lapses" is not how interest trimming works: streamState.minConsumerCursorLocked in state.go reads consumerStore.minLiveWatermark in store.go, which is watermark-based. A lapsed lease frees the consumer for a replacement loop but neither removes it nor advances its watermark, so it pins the command log until the consumer is deleted or a DiscardOld trim passes it. (2) Server.enforceStorageBudget calls store.budgetSnapshot, which sweeps every directory under streams/ with no name filter, so internal-<name> segments are budget-eligible through store.deleteSegment and can be dropped ahead of a lagging replica. For a user stream that is documented skip-ahead; for the command log the replica permanently lacks records, which is an at-least-once violation for clients pinned to it. Define a single resync(stream) from the leader's retained window and use it for both this and the fresh-replica case.

**N1 and N2.** Two corrections. (1) "until its lease lapses" is not how interest trimming works: `streamState.minConsumerCursorLocked` in `state.go` reads `consumerStore.minLiveWatermark` in `store.go`, which is watermark-based. A lapsed lease frees the consumer for a replacement loop but neither removes it nor advances its watermark, so it pins the command log until the consumer is deleted or a `DiscardOld` trim passes it. (2) `Server.enforceStorageBudget` calls `store.budgetSnapshot`, which sweeps every directory under `streams/` with no name filter, so `internal-<name>` segments are budget-eligible through `store.deleteSegment` and can be dropped ahead of a lagging replica. For a user stream that is documented skip-ahead; for the command log the replica permanently lacks records, which is an at-least-once violation for clients pinned to it. Define a single `resync(stream)` from the leader's retained window and use it for both this and the fresh-replica case.
plan.md Outdated
@ -270,0 +236,4 @@
touching the encoding of the existing kinds, and a membership disagreement is
an error rather than a loop. The reply echoes `frameForward` and carries the
inner reply payload.
- **A reserved internal stream-name prefix** (`internal-`), rejected by
Author
Member

N4. The reservation as stated covers only CreateStream and ListStreams. There is no authentication in this protocol (doc.go says so), so a peer that guesses internal-<name> can CreateConsumer/Pull on it and pin the leader's command log — a cheap growth vector. Enforce the prefix for every client-facing kind in Server.dispatch (publish, pull, ack, create/delete stream and consumer, get stream/consumer), and put the check there rather than inside createStream, so the replica's own consumer on the leader's internal-<name> bypasses it. Call it namespace hygiene, not a security boundary.

**N4.** The reservation as stated covers only `CreateStream` and `ListStreams`. There is no authentication in this protocol (`doc.go` says so), so a peer that guesses `internal-<name>` can `CreateConsumer`/`Pull` on it and pin the leader's command log — a cheap growth vector. Enforce the prefix for every client-facing kind in `Server.dispatch` (publish, pull, ack, create/delete stream and consumer, get stream/consumer), and put the check there rather than inside `createStream`, so the replica's own consumer on the leader's `internal-<name>` bypasses it. Call it namespace hygiene, not a security boundary.
@ -270,0 +241,4 @@
store-backed listing). Internal names are server-generated and may exceed
`maxNameSize` by the prefix width; they remain safe path components and are
never client-supplied.
- **The `frameGetStream` reply gains `maxAge`**, so a node discovering a peer's
Author
Member

N7. Server.getStream in server.go currently returns a bare response{} with no fields, so maxAge on this reply is a real protocol addition and belongs in the "Protocol additions" list next to frameForward. While there: listStreams is paginated (listStreamsRequest.cursor/limit, Server.listStorePage), so the reconciliation described in Phase 2 must page rather than assume one reply.

**N7.** `Server.getStream` in `server.go` currently returns a bare `response{}` with no fields, so `maxAge` on this reply is a real protocol addition and belongs in the "Protocol additions" list next to `frameForward`. While there: `listStreams` is paginated (`listStreamsRequest.cursor`/`limit`, `Server.listStorePage`), so the reconciliation described in Phase 2 must page rather than assume one reply.
@ -446,3 +268,1 @@
because Phase 1 lives on an unmerged feature branch; no migration reader is
written. If any Phase 1 data must be preserved across this branch, a one-time
reader must be added before merge.
- `WithMeshPeers(ids ...peer.ID)` configures the node set; absence of the option
Author
Member

N6. State that the configured set includes self, and add a deterministic tie-break for hash collisions (compare peer-ID bytes). A node that omits itself computes a leader it will never agree with, and then meets the loop guard instead of making progress.

**N6.** State that the configured set **includes self**, and add a deterministic tie-break for `hash` collisions (compare peer-ID bytes). A node that omits itself computes a leader it will never agree with, and then meets the loop guard instead of making progress.
@ -527,0 +333,4 @@
reset to the leader's base on a `Trim` past its position, repairing any local
consumer cursor the reset outran through the existing `repairCursor` (without
raising its watermark — those records were never processed).
- A `Trim` command drives the replica's base, and a replica never runs an age
Author
Member

N3. Apply this monotonically: max(replicaBase, cmd.base), ignoring a stale command. The plan's own Phase 5 test at line ~415 has a replica evicting its oldest segments under its own storage budget, so a replica's base can legitimately be ahead of the leader's; applying Trim{base} as a set would attempt to un-trim.

**N3.** Apply this monotonically: `max(replicaBase, cmd.base)`, ignoring a stale command. The plan's own Phase 5 test at line ~415 has a replica evicting its oldest segments under its own storage budget, so a replica's base can legitimately be *ahead* of the leader's; applying `Trim{base}` as a set would attempt to un-trim.
@ -577,3 +374,1 @@
not materialize a cold stream.
- `bench_test.go` — a budget sweep over thousands of non-resident streams does
not load their records.
- A client consumes on one replica, acks, then reconnects to a *different* node
Author
Member

B2. This test cannot pass as specified while frameGetConsumer is answered locally: Stream.Consumer seeds the client watermark from Server.getConsumer, and the replica may not yet have applied the ConsumerUpsert from the client's acknowledged record. Either forward the consumer-metadata read to the leader (recommended; see the comment on line 116) or restate this test as "continues exactly where it stopped once the replica has caught up", which is a weaker and quite different assertion.

**B2.** This test cannot pass as specified while `frameGetConsumer` is answered locally: `Stream.Consumer` seeds the client watermark from `Server.getConsumer`, and the replica may not yet have applied the `ConsumerUpsert` from the client's acknowledged record. Either forward the consumer-metadata read to the leader (recommended; see the comment on line 116) or restate this test as "continues exactly where it stopped once the replica has caught up", which is a weaker and quite different assertion.
Addresses the review on PR #2. Four blocking findings and eight spec gaps, all
design-level, so the fixes are to plan.md.

Blocking:

- B1: a mutex gives ordering, not crash atomicity. A crash between the fsync of
  <name> and the fsync of internal-<name> left a record durable on the leader
  that no replica was told about, and because memlog.appendRecovered adopts gaps
  silently, a replica diverged rather than stalled — a silent at-least-once
  failure. The command log is now the write-ahead log: append and fsync the
  command, then apply it to the leader's own <name> through the same applier a
  replica uses, and replay from the leader's own apply position at startup.
- B2: frameGetConsumer was classified as a local read, so a restarted handle on a
  lagging replica seeded a stale processedBefore and re-ran completed handlers.
  It is now forwarded to the leader, and the rule is restated as 'record reads
  are never forwarded'.
- B3: a heartbeat-bounded lease cannot bound a stale view, so a second loop on a
  third node was admitted. A pull now validates its lease on the leader every
  time (option (a)); the reply also carries the authoritative watermark and
  cursor, so the replica serves at the leader's cursor. One extra hop per
  delivered record is the stated price, and the ErrConsumerBusy test is now
  immediate rather than eventual.
- B4: Append gains dedupID, and the applier is a new strict memlog.applyRecord
  that rejects a gap, stores the given seq and timestamp, and registers dedupID;
  neither appendOwned (mints time.Now) nor appendRecovered (adopts gaps) can
  serve. memlog.go added to Files touched.

Gaps: N1 one resync path for a replica below the command log's base and for a
fresh replica; N2 the retention claim corrected (watermark-based, so decommission
must delete the replica consumer) and applied in the trade-off, Phase 2 and the
tests; N3 Trim applied as max(replicaBase, cmd.base); N4 the internal- prefix
enforced in Server.dispatch for every client-facing kind, called namespace
hygiene rather than security; N5 replica consumer named rep-<peerID> with a
decommission path; N6 WithMeshPeers includes self and leader breaks hash
collisions on peer-ID bytes; N7 maxAge on the getStream reply stated as a real
addition and listStreams pagination noted for reconciliation; N8 ConsumerUpsert
applied monotonically.

Adds a 'Guarantees, and what enforces each' section naming the three promises and
the single place each is enforced, and fault-injection tests: crash atomicity
between the two copies, command-log truncation below a replica, a stale Trim, and
a duplicate and gapped Append.
sam merged commit 216df662a5 into main 2026-09-26 11:42:40 +00:00
sam deleted branch feature/server-mesh-replication 2026-09-26 11:42:40 +00:00
Sign in to join this conversation.
No reviewers
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
norobo/webstream!2
No description provided.