docs: server mesh replication plan #2
No reviewers
Labels
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
norobo/webstream!2
Loading…
Reference in a new issue
No description provided.
Delete branch "feature/server-mesh-replication"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Docs-only branch: no Go code changes. The diff is
plan.md,README.mdanddoc.go.plan.mdpreviously held the single-node server-persistence plan, which is delivered and merged (PR #1,74c236d), including the Phase 5 publish benchmarks (PR #3,6b073ef). It is replaced here with the plan for the next milestone: server mesh stream replication.README.mdanddoc.goare corrected where they still described the pre-persistence state.Scope
R = Nfor any node countN >= 1: the replication factor equals the number of server nodes, so every node holds every stream and every stream has exactly as many replicas as there are nodes. Counts 1, 2 and 3 are all in scope, and are the counts the acceptance tests exercise:N = 1— one replica per stream. Today's server: no mesh configured, no replication goroutine, no forwarding.N = 2— two replicas per stream, one per node.N = 3— three replicas per stream, one per node.Deferred explicitly: placement for
R < N(which nodes hold which streams), leader failover, dynamic mesh membership, and the mesh-independent items — auth (libp2p capability envelopes) and the browser/WebTransport client.Design decisions
One writer per stream, and the writer is derived, not stored.
leader(stream) = argmin hash(streamName, peerID)over the configured node set. A pure function, so every node agrees without state or negotiation, andR < Nlater changes only the candidate set.Writes are forwarded, one hop. A mutation that lands on a non-leader is proxied over that node's own peer connection. Clients never need to discover the leader and never see a wrong-node error. Reads are never forwarded.
Records keep the leader's sequence numbers verbatim. With one writer per stream, a sequence number is globally unique for that stream, so replicas share one address space with the leader (
base,next,readAt(seq), every consumer watermark) andMsg.Seqstays meaningful when a client changes node.Webstream replicates webstream. Each user stream
<name>gets a hidden companion streaminternal-<name>, written only by the leader, holding an ordered command log:Append{seq, ts, data},Trim{base},ConsumerUpsert{name, processedBefore},ConsumerDelete{name}. Every replica consumes it as an ordinary webstream client consumer and applies the commands to its own copy. Replica sync therefore inheritsInterestPolicyretention (a lagging replica pins the log), the durable acked watermark (its own sync position) and the consume lease (replica liveness) instead of reinventing them. A replica keeps no copy of the command log, which is also why replication cannot recurse intointernal-internal-....Consumer state is stream state. Watermarks propagate on the command log, and the consume lease is leader-arbitrated, so one consumer name has one live loop across the mesh and a client that reconnects to a different node resumes at its watermark rather than reprocessing. These were the two gaps the previous draft had deferred; they are requirements now.
No clock alignment is needed. libp2p synchronizes nothing (its only clock-adjacent dependencies are local-clock abstractions for testable timers), and host clocks skew. Because each stream has one writer, the leader's append time is the only ingress stamp a record ever gets, so every timestamp in a stream comes from one clock and every sequence from one counter. Clients never supply a timestamp, and replicas never compare their wall clock to a record's — they follow the leader's retained base via
Trim{base}commands.Strict sequential delivery is unchanged. One record in flight, the cursor advances only on a confirmed ack, and
advanceIf's strict-equality ack check and the scalarprocessedBeforewatermark are untouched. Because the leader orders everything, a replica has an order to be sequential about: no ordering key, client-side reorder buffer or hole-aware progress structure is required.Wire delta is deliberately small
frameForward— one additive frame kind wrapping an inner request plus a forwarded marker, so any mutation can be forwarded without touching the encoding of existing kinds, and a membership disagreement (a stale peer set during a rolling restart) produces a loud loop error instead of infinite recursion.internal-stream-name prefix, rejected byCreateStreamand filtered out ofListStreams.maxAgeadded to theframeGetStreamreply, so a node discovering a peer's stream creates it with the leader's retention configuration.Replication itself needs no new frame kind: a replica reads the command log with
framePulland confirms withframeAck.streamProtocolstays/norobo/webstream/1.0.0.Costs this design accepts
<name>, once tointernal-<name>. One extra copy, not one per replica, since all replicas consume the same command log. The alternative (a command pointing at a sequence range that replicas read directly) removes the copy but requires a second pull loop per replica.Phases
Trim, no age loop on replicas, per-batch residency.N = 2andN = 3, failure cases (leader restart, leader outage, replica budget) and benchmarks for the forwarding hop, replication throughput, backfill and tail latency.N = 1is the regression suite and is unchanged.Decisions worth challenging in review
leader = argmin hash(streamName, peerID)rather thanmin(peerID)— the latter would put every stream's leader on the lowest-ID node.<name>variant.Also fixed in this branch (documentation accuracy)
README.mdclaimedmaxNameSizewas 1 KiB; it is 64 bytes.README.mdsaid "compaction is next". It is not needed: trimming only ever advances the retained base, andjournal.trimTodeletes whole sealed segments below it, so the log never develops interior holes.README.md/doc.godescribed the server as an early in-memory implementation with deferred retention and Phase 1/2 numbering from the superseded plan.Verification
Docs and comments only, so the Go suite is unaffected:
go build ./...andgo vet ./...both pass, andgofmt -l .is clean. The full phase-by-phase verification block is inplan.mdunder "Verification".Review — server mesh replication plan
Artifact reviewed:
plan.mdat head0655d46, branchfeature/server-mesh-replication(this PR, basemain6b073ef). Docs-only, so no build/test failure is implied by anything below. The question this review answers is whether the plan, implemented exactly as written, delivers the two guarantees it claims: at-least-once delivery from the server and exactly-once semantics for the client through deduplication.Scope I read the plan as claiming: the first iteration is
R = N(replication factor equals node count, so every node holds every stream) forN ∈ {1, 2, 3}, with placement forR < N, leader failover, and dynamic membership deferred. That matches the PR body, and theN = 1regression framing is right.Verdict
The shape is correct — one writer per stream, forwarded mutations, and a replication command log that is itself a webstream stream are the right three decisions, and they are what keep
Msg.SeqandprocessedBeforemeaningful across nodes without clocks or ordering keys. But the plan does not yet guarantee what it claims, because of four holes below. B1 breaks at-least-once permanently; B2 and B3 break exactly-once processing in the cases the plan itself asserts as tests. These are design-level, not wording-level: they change the ordering of the write path and two entries in the "reads are never forwarded" rule.Claims that check out against the merged code
Server.publishinserver.gocallsstreamState.appendDedupinstate.go, which is the only dedup authority, and a forwarded publish lands there.processedBeforeis already the durable acked watermark in the consumer slot store,streamState.computeKeepLocked/minConsumerCursorLockedalready derive the retained base from it, andconsumerState.advanceIfalready does the strict-equality cursor move in one critical section.journal.baseandjournal.nextare derived from the segment set (store.go), andjournal.rollLockedusesrec.seqas the new segment base, so a replica's own roll boundaries are legal and recoverable.benbjohnson/clockandfilecoin-project/go-clockingo.mod, both local-clock abstractions.Guarantees — please state these explicitly in the plan
The plan currently describes mechanisms but never states, in one place, which mechanism enforces which guarantee and where. Given that the review question is exactly this, add a Guarantees section naming:
consumerState.advanceIf, unchanged). A redelivery after a lost ack is the documented at-least-once edge.streamState.dedupID→sequence table, retention-bounded (streamState.dropDedupLocked). B4 is required for this to hold on a replica whose durable state is recovered or that later becomes the leader.consumer.Consume(raised monotonically byraiseWatermark, never lowered) already covers a node move, because sequence numbers are shared. For a restarted handle the guarantee comes from the replicated watermark read back byStream.Consumer— that path is B2 and it currently does not hold.Blocking findings
B1. No crash atomicity between
<name>andinternal-<name>; this loses data permanentlyThe plan says the leader appends the
Appendcommand "in the same critical section as the user append". A mutex gives ordering, not crash atomicity.Server.publishholdsjournal.mu(the per-stream writer lock instore.go) across the in-memory append and the durable write tostreams/<name>/log/…. The command write is a second durable write tostreams/internal-<name>/log/…. A crash between the two fsyncs leaves a record that is durable on the leader and that no replica will ever be told about.The outcome is worse than a stall, because both plausible apply primitives handle the resulting hole wrongly:
memlog.appendRecoveredinmemlog.godoesif rec.seq > log.next { log.next = rec.seq }— it silently adopts gaps. An applier built on it would continue past the missing sequence and diverge; a client pinned to that replica never receives that record, so at-least-once is violated there, not merely delayed.cmd.seq == log.next(which is what the plan's "apply only at its ownlog.next" implies) stops permanently: every later command is dropped, and the replica never recovers without an explicit resync path, which the plan does not define.Fix: make
internal-<name>the write-ahead log and derive the leader's own copy from it. Order becomes: (1) append and fsync theAppendcommand; (2) apply it to the leader's own<name>through the same applier a replica uses. Then a crash before (1) loses both copies (and the publish never acked, so no client promise is broken), and a crash after (1) is repaired by replaying the leader's own command log from its own apply position (<name>'slog.next) at startup. This also makes the plan's "every timestamp comes from one clock" survive, because both paths store the timestamp the command carries rather than minting one.B2. Consumer-metadata reads are not read-your-writes, so the Phase 4 headline test cannot pass
The plan classifies
frameGetConsumeras a read and says reads are never forwarded. ButStream.Consumerinstream.goseeds the client's watermark fromServer.getConsumerinserver.go, which answers from local state. Sequence: client acks on node A (the ack is forwarded to the leader, which writesConsumerUpsert{name, processedBefore}); the client then restarts and callsStream.Consumeron node B before B's applier has consumed that command; B returns a staleprocessedBefore; the client re-runs handlers it already completed. That is at-least-once processing, not the exactly-once the plan's Phase 4 test asserts ("reconnects to a different node and continues exactly where it stopped, without reprocessing") and thatdoc.gopromises ("a restarted client resumes where its predecessor stopped").This is specifically the new-handle case: with a live handle the client-local watermark already skips redeliveries across a node move.
Fix: forward
frameGetConsumerto the leader. It is symmetric with the plan already forwarding consumer mutations, and it keeps the record data path local, so "reads are never forwarded" becomes "record reads are never forwarded" — a rule that is both true and enforceable. Gating the pull on the replica's replicated watermark being at least the ack it forwarded is the alternative, but it is strictly more machinery for the same guarantee.B3. The mesh-wide lease has a window in which two consume loops both run
consumerState.acquireLeaseinstate.gois called on every pull today, and that is what makes the single-loop invariant airtight: the lease is the only thing preventing two loops from serving the same cursor record concurrently (Server.preparePullreads at the consumer's cursor without tracking in-flight delivery, so two admitted loops both run the handler for the same sequence; the loser's ack then failsErrAckOutOfOrder). The plan replaces the per-pull check with a local lease view refreshed by a heartbeat "well insideconsumerLeaseTimeout" (30 s,server.go). Between the leader granting or renewing the lease and a replica receiving that grant on the command log, the replica's view is stale, so a second loop on a third node is admitted. A heartbeat bounds liveness; it cannot bound a stale view, so this is a correctness window, not a tuning question.Fix — pick one and state the consequence of the choice:
ErrConsumerBusytest becomes "eventually refused", anddoc.go's single-loop paragraph must carry the same qualification.Claiming both the heartbeat and the
ErrConsumerBusyguarantee is the one option that is not available.B4.
Append{seq, ts, data}omitsdedupID, and neither existing append primitive can be the applierThe on-disk record frame is
seq, timestamp, dedupID, data(store.goencodeRecordFrame/decodeRecord), andstreamState.rebuildDedupreconstructs the dedup table from the retained records on recovery. A replica applyingAppendwithout the ID stores a different record than the leader and recovers an empty dedup table, so a replica that later becomes the leader (the deferred failover) silently loses the producer exactly-once window. AdddedupIDtoAppend.Related, and needed for the fix in B1: neither existing in-memory append primitive is usable for the applier as-is.
memlog.appendOwnedmintstimestamp: time.Now()and assignslog.nextitself, which would break the plan's own "timestamps replicated verbatim / one clock per stream" invariant;memlog.appendRecoveredtakes an exact record but silently adopts gaps (see B1). The plan needs a strictapplyRecord(record)that (i) rejectsrec.seq != log.next, (ii) stores the givenseqandtimestamp, and (iii) registersrec.dedupIDinstreamState.dedupexactly asappendOwnedLockeddoes.memlog.goshould therefore appear in the plan's "Files touched" list — it is not there today.Spec gaps that are cheap to close now
Server.enforceStorageBudget→store.budgetSnapshotwalks every directory understreams/with no name filter, sointernal-<name>segments are budget-eligible viastore.deleteSegment, advancing the command log's base past a lagging replica. For a user stream, skipping ahead is the documentedDiscardOldbehaviour; for the command log it means that replica permanently lacks records — an at-least-once violation for clients pinned there. Define oneresync(stream)path from the leader's retained window and use it for both this case and the fresh-replica case, and say explicitly what happens to a replica that falls below the command-log base (mark failed, re-register, re-backfill).streamState.minConsumerCursorLocked→consumerStore.minLiveWatermarkinstore.go, which is watermark-based. A lapsed lease frees the consumer for a replacement loop but does not remove it or advance its watermark, so it pins the command log until the consumer is deleted or aDiscardOldtrim passes it. Either delete the replica's consumer on decommission, or make the discard path of N1 responsible.Trim{base}monotonically. The plan's own Phase 5 test has a replica evicting its oldest segments under its own storage budget, so a replica's base can legitimately be ahead of the leader's.max(replicaBase, cmd.base), ignoring a stale command; a set would attempt to un-trim.internal-reservation is only asserted forCreateStreamandListStreams. There is no authentication in this protocol (doc.gostates so), so a peer that guessesinternal-<name>canCreateConsumer/Pullon it and pin the leader's command log, a cheap growth vector. Apply the prefix check inServer.dispatchfor every client-facing kind (publish, pull, ack, create/delete stream and consumer, get stream/consumer), and put it there rather than deep insidecreateStream, so the replica's own consumer on the leader'sinternal-<name>bypasses it. State that the prefix is namespace hygiene, not a security boundary.internal-<name>" needs a distinct name per replica (for examplerep-<peerID>); two replicas sharing one name share a watermark and compete for the lease. Also specify the graceful-decommission delete.WithMeshPeerssemantics. State that the configured set includes self, with a deterministic tie-break onhashcollisions (compare peer-ID bytes): a node that omits itself computes a leader it will never agree with, and then meets the loop guard.Server.getStreamcurrently returns a bareresponse{}with no fields, somaxAgeon theframeGetStreamreply is a real addition and belongs in the "Protocol additions" list. AndlistStreamsis paginated (listStreamsRequest.cursor/limit,Server.listStorePage), so reconciliation must page rather than assume one reply.ConsumerUpsertis applied monotonically (never loweringprocessedBefore); a move that lowers it would make a local live loop'sadvanceIffailErrAckOutOfOrder.Tests that would actually catch these
Add fault-injection cases to Phase 3/4/5; a naive implementation passes the current list:
Stream.Consumeron a lagging replica; assert no reprocessing (B2).Appendand aTrimolder than the replica's base; assert strict rejection and no base regression (B4, N3).The per-line comments below carry the same findings anchored to the text they apply to.
@ -124,0 +113,4 @@- **A forwarded request keeps its payload and its deadline**, bounded by`defaultWriteTimeout` when the caller's deadline is not near, exactly like aclient request write.- **Reads are never forwarded.** A pull, list or get is answered from the localB2 — blocking. This rule cannot include
frameGetConsumer.Stream.Consumerinstream.goseeds the client's watermark fromServer.getConsumerinserver.go, which answers from local state. A client that acks on node A (forwarded to the leader, which writesConsumerUpsert), restarts, and callsStream.Consumeron node B before B has applied that command gets a staleprocessedBeforeand re-runs handlers it already completed. That contradicts the Phase 4 test at line 374 and the promise indoc.gothat a restarted client resumes where its predecessor stopped. ForwardframeGetConsumerto the leader — symmetric with forwarding consumer mutations, and the record data path stays local. Suggest restating the rule as "record reads are never forwarded".@ -178,3 +157,1 @@1 nameLen u82 name [64]B66 cursorSlot u64 // which slot of cursors.meta holds this consumerAppend{seq, ts, data} a record was appended to <name>B4. This command shape omits
dedupID, which the on-disk record frame carries: seeencodeRecordFrame/decodeRecordinstore.go(fieldsseq,timestamp,dedupID,data), andstreamState.rebuildDedupinstate.go, which rebuilds the dedup table from the retained records on recovery. A replica applying thisAppendstores a record that differs from the leader's and recovers an empty dedup table, so any later leader role for that node silently loses the producer exactly-once window. AdddedupIDto the command.@ -197,2 +167,3 @@**`cursors.meta`** — the values, written once per ack:- **A replica's sync position is a real consumer watermark.** The replicaregisters as a consumer of the leader's `internal-<name>` through the normalN5. The replica's consumer name on the leader's
internal-<name>must be distinct per replica (e.g.rep-<peerID>): a shared name shares one watermark and puts two replicas in competition for one consume lease. Also specify the graceful-decommission path, which is what actually unpins the command log (see the comment on line 226).@ -199,0 +169,4 @@registers as a consumer of the leader's `internal-<name>` through the normal`CreateConsumer` path, so `InterestPolicy` holds commands for a replica that isbehind, and a stopped replica simply stops acking. No new retention logic.- **Apply is idempotent and ordered.** A replica applies `Append` only at its ownB4 / B1. "Apply only at its own
log.next" is right, but neither existing primitive can implement it as-is.memlog.appendOwnedinmemlog.gomintstimestamp: time.Now()and assignslog.nextitself, which breaks the plan's own one-clock-per-stream invariant.memlog.appendRecoveredtakes an exact record but doesif rec.seq > log.next { log.next = rec.seq }, i.e. it silently adopts gaps. A new strictapplyRecord(record)is required: rejectrec.seq != log.next, store the givenseqandtimestamp, and registerrec.dedupIDinstreamState.dedupasappendOwnedLockeddoes.memlog.gois missing from the "Files touched" list.@ -199,0 +174,4 @@beyond its position resets it. It holds the journal's writer lock across thein-memory append *and* the durable write, exactly as `Server.publish` does, orout-of-order frames reach disk and refuse to load.- **The leader writes the command in the same critical section as the userB1 — blocking. A mutex gives ordering, not crash atomicity.
Server.publishinserver.goholdsjournal.muacross the in-memory append and the fsync ofstreams/<name>/log/...; this command is a second durable write tostreams/internal-<name>/log/.... A crash between the two leaves a record durable on the leader that no replica is ever told about. Becausememlog.appendRecoveredadopts gaps silently (see the comment on line 172), an applier would diverge rather than stall, and a client pinned to that replica never receives the record — at-least-once fails, not just exactly-once.Fix: make
internal-<name>the write-ahead log. (1) append + fsync the command, (2) apply it to the leader's own<name>through the same applier a replica uses, so recovery replays the leader's command log from its own apply position. A crash before (1) loses both copies and no publish acked; a crash after (1) is repaired by replay.@ -232,0 +201,4 @@assigned `seq`, the new `processedBefore`, the granted lease) and the receivingreplica applies it immediately. Without that, a replica would redeliver arecord between its local ack and the arrival of the propagated watermark.- **Lease renewal is a heartbeat, not a per-pull round trip.** A replica holdsB3 — blocking. A heartbeat bounds liveness, not stale reads. Today
consumerState.acquireLeaseinstate.goruns on every pull, and that is what makes the single-loop invariant airtight;Server.preparePullserves at the consumer cursor without tracking in-flight delivery, so two admitted loops both run the handler for the same sequence and the loser's ack failsErrAckOutOfOrder. Between the leader granting or renewing the lease and a replica receiving that grant, the replica's view is stale, so a second loop on a third node is admitted.Pick one: (a) forward the lease check to the leader on every pull while the record read stays local (one extra hop per record; the honest price), or (b) keep the heartbeat and state explicitly that exactly-once processing is conditional on no second loop starting within the stale-view window — in which case the Phase 4
ErrConsumerBusytest becomes "eventually refused" anddoc.go's single-loop paragraph needs the same qualification. Claiming both is not available.@ -255,0 +223,4 @@- **A leader outage stops writes to its streams.** Reads continue on everyreplica. Failover needs a liveness rule and is deferred; a stale live set wouldproduce two leaders and diverging logs.- **Command logs count against the storage budget** and are bounded by replicaN1 and N2. Two corrections. (1) "until its lease lapses" is not how interest trimming works:
streamState.minConsumerCursorLockedinstate.goreadsconsumerStore.minLiveWatermarkinstore.go, which is watermark-based. A lapsed lease frees the consumer for a replacement loop but neither removes it nor advances its watermark, so it pins the command log until the consumer is deleted or aDiscardOldtrim passes it. (2)Server.enforceStorageBudgetcallsstore.budgetSnapshot, which sweeps every directory understreams/with no name filter, sointernal-<name>segments are budget-eligible throughstore.deleteSegmentand can be dropped ahead of a lagging replica. For a user stream that is documented skip-ahead; for the command log the replica permanently lacks records, which is an at-least-once violation for clients pinned to it. Define a singleresync(stream)from the leader's retained window and use it for both this and the fresh-replica case.@ -270,0 +236,4 @@touching the encoding of the existing kinds, and a membership disagreement isan error rather than a loop. The reply echoes `frameForward` and carries theinner reply payload.- **A reserved internal stream-name prefix** (`internal-`), rejected byN4. The reservation as stated covers only
CreateStreamandListStreams. There is no authentication in this protocol (doc.gosays so), so a peer that guessesinternal-<name>canCreateConsumer/Pullon it and pin the leader's command log — a cheap growth vector. Enforce the prefix for every client-facing kind inServer.dispatch(publish, pull, ack, create/delete stream and consumer, get stream/consumer), and put the check there rather than insidecreateStream, so the replica's own consumer on the leader'sinternal-<name>bypasses it. Call it namespace hygiene, not a security boundary.@ -270,0 +241,4 @@store-backed listing). Internal names are server-generated and may exceed`maxNameSize` by the prefix width; they remain safe path components and arenever client-supplied.- **The `frameGetStream` reply gains `maxAge`**, so a node discovering a peer'sN7.
Server.getStreaminserver.gocurrently returns a bareresponse{}with no fields, somaxAgeon this reply is a real protocol addition and belongs in the "Protocol additions" list next toframeForward. While there:listStreamsis paginated (listStreamsRequest.cursor/limit,Server.listStorePage), so the reconciliation described in Phase 2 must page rather than assume one reply.@ -446,3 +268,1 @@because Phase 1 lives on an unmerged feature branch; no migration reader iswritten. If any Phase 1 data must be preserved across this branch, a one-timereader must be added before merge.- `WithMeshPeers(ids ...peer.ID)` configures the node set; absence of the optionN6. State that the configured set includes self, and add a deterministic tie-break for
hashcollisions (compare peer-ID bytes). A node that omits itself computes a leader it will never agree with, and then meets the loop guard instead of making progress.@ -527,0 +333,4 @@reset to the leader's base on a `Trim` past its position, repairing any localconsumer cursor the reset outran through the existing `repairCursor` (withoutraising its watermark — those records were never processed).- A `Trim` command drives the replica's base, and a replica never runs an ageN3. Apply this monotonically:
max(replicaBase, cmd.base), ignoring a stale command. The plan's own Phase 5 test at line ~415 has a replica evicting its oldest segments under its own storage budget, so a replica's base can legitimately be ahead of the leader's; applyingTrim{base}as a set would attempt to un-trim.@ -577,3 +374,1 @@not materialize a cold stream.- `bench_test.go` — a budget sweep over thousands of non-resident streams doesnot load their records.- A client consumes on one replica, acks, then reconnects to a *different* nodeB2. This test cannot pass as specified while
frameGetConsumeris answered locally:Stream.Consumerseeds the client watermark fromServer.getConsumer, and the replica may not yet have applied theConsumerUpsertfrom the client's acknowledged record. Either forward the consumer-metadata read to the leader (recommended; see the comment on line 116) or restate this test as "continues exactly where it stopped once the replica has caught up", which is a weaker and quite different assertion.