feat: relay leader-owned requests to a stream's leader #5
No reviewers
Labels
No labels
code-review
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
norobo/webstream!5
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/mesh-request-relay"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Phase 1 of
plan.md(server mesh replication): the mesh's node set, the derived stream leader, and one-hop forwarding of writes. No record replication yet — that is Phases 2 and 3.What is here
WithMeshPeers(ids ...peer.ID)on the server names the mesh's nodes. A stream's writer isargmin hash(streamName, peerID)over that set, so every node agrees on the leader from the same inputs and nothing is stored or negotiated. A leader-owned request that lands on another node is relayed to that stream's leader over the node's own peerConn— a node is an ordinary webstream client of its peers, so there is no second wire path — and the client receives the leader's reply.frameForwardis one additive frame kind wrapping an inner request (its kind plus payload) plus the outer reply carrying the leader's inner reply. Existing kinds keep their encoding, andstreamProtocolstays/norobo/webstream/1.0.0.ErrForwardLoop, never relayed again, so two nodes that disagree about membership (a rolling restart with a stale node set) fail loudly instead of ping-ponging. An unreachable leader fails the request withErrLeaderUnavailableand applies nothing locally.framePull), listings (frameListStreams) and the existence check (frameGetStream) are answered locally: a record is read from the local copy, while consumer metadata is leader-owned state that lives only on the leader until Phase 4 replicates it.N = 1is unchanged. WithoutWithMeshPeersthere is no node set, no relay path and no outboundConn;WithMeshPeers()with no peers is also a no-op set. A mesh deployment must give the option a set that includes the node itself and is identical on every node — the leader is a pure function of it — and must combine it withWithAllowedPeersand a host gater, becauseWithMeshPeersis not an inbound allowlist.Two entry points own the two routing policies and no boolean or enum is threaded between them:
clientDispatchrelays a leader-owned request this node does not lead,peerDispatchfails a request a peer handed it that it does not lead.servedecodes and runs the handlers with no routing policy in it at all. Routing reads the stream name from the request's first field (frameStreamName), which every routed kind carries first;TestFrameStreamNameMatchesDecoderspins that against the full per-kind decoders so a decoder that reorders fields fails a test instead of mis-routing.Two consequences worth knowing before review
maxRecordSizeand the publish frame budget now reserveforwardOverhead(the wrapper's kind byte plus the inner length prefix), so a publish that fits locally is also relayable. Without the reservation a maximum-size publish would be rejected only when it had to be relayed, which would break "any node accepts a write" at the boundary.rpcStream.delivernow keeps the read buffer for aframeForwardreply. A forwarded reply'srawinner payload is decoded by the forwarder afterdeliverreturns, exactly like a record aliases the payload, so reusing the buffer would let the next frame overwrite a reply still being decoded. This bug was found while writing the relay tests and is covered byTestDeliverKeepsAliasedPayloads.Deliberately not in this PR
plan.mdlists that clause under Phase 1, but the mechanism it needs ("a pull validates its lease on the leader every time", with the reply carrying the authoritative watermark) is Phase 4's design. Phase 1 forwards consumer mutations andgetConsumer; the record read and the lease stay local, soErrConsumerBusyis not yet a cross-node guarantee.Conn.Streamon that node reports not-found until Phase 2 discovery and Phase 3 backfill. That is the plan's phase boundary, not a regression —getStreamwas never routed.R < N, leader failover, dynamic membership: deferred inplan.md.Verification
The new tests are in
mesh_test.go: two nodes in one process (separate libp2p hosts, because libp2p refuses to self-dial) covering create and publish relayed to the leader with the leader's sequence numbers and no local state on the non-leader, the leader serving locally even with an unreachable peer in its set, a pull staying local whilegetConsumeris answered by the leader, a membership disagreement yieldingErrForwardLoop, an unreachable leader yieldingErrLeaderUnavailablewith no local write, and the no-mesh option forwarding nothing.fuzz_test.gogainsFuzzDecodeForwardand theframeForwardreply seed.Pre-existing failures this PR does not introduce
Both reproduce on the parent commit in a separate worktree, so a red run is not from this change:
racedetector test ./...already reports ~1.5kWARNING: DATA RACE(dominated byTestConcurrentDurablePublishOnDiskOrder, inServer.acquire/publish) andTestConsumeSurvivesEvictionalready panics on an unlockedserver.streams[name].streamread inintegration_test.go.go test ./...is roughly 1-in-25 flaky at the base commit too (TestConsumeSurvivesEviction,TestMinLiveWatermarkTracksCreateDeletePut), load-dependent.golangci-lint runwith the workspace's current go-linters default config reports ~84 issues on the base commit;--new-from-revis the clean signal used above.Reviewed PR #5 (feat/mesh-request-relay) against the samber/cc-skills-golang skill set, verified against the PR head (
df33ac5) in this session:go build ./...clean,go vet ./...clean, fullgo test -count=1 ./...passes,racedetector test -run 'TestMesh|TestDeliverKeepsAliasedPayloads|TestConcurrentDurablePublishOnDiskOrder'reports no races, and the mesh + fuzz seed tests pass.High quality overall: the routing split (clientDispatch / peerDispatch / serve) with no flag threaded between entry points, the single-source routed-kind list pinned by TestFrameStreamNameMatchesDecoders, the frameForward read-buffer aliasing fix with TestDeliverKeepsAliasedPayloads, and the meshPeers connsLock/closed/drop-by-identity concurrency are all sound and well-documented. I checked the forward-overhead reservation end to end: a max-size publish stays relayable and the leader's read path never exceeds maxFrameSize. I found no blocking correctness, safety, or data-race issue.
Three non-blocking findings (two IMPORTANT, one SUGGESTION), all in meshPeers.forward/route, each about the relay's timeout and failure handling, each with a concrete fix.
@ -467,0 +588,4 @@// forward relays one request to the stream's leader and returns the leader's// reply to that request.func (m *meshPeers) forward(ctx context.Context, leader peer.ID, kind frameKind, payload []byte) (response, error) {conn, err := m.conn(ctx, leader)🟠 IMPORTANT — the relay dial is not bounded by the server's write timeout.
m.conn(ctx, leader)at line 591 runsNewConn→host.NewStream(ctx, ...)(server.go:555) before thecontext.WithTimeout(ctx, defaultWriteTimeout)at line 602 is applied. Thectxhere is the inbound handler's long-lived stream context (no deadline), so the dial to an unreachable leader that still advertises addresses is bounded only by libp2p's implicit dial timeout, not bydefaultWriteTimeout(30s) — contradicting the inline comment at line 596 ("an unreachable leader must fail the request rather than hang it"). Verified:defaultDialTimeout = 15 * time.Secondin$GOMODCACHE/github.com/libp2p/go-libp2p@v0.49.0/p2p/net/swarm/swarm.go:29;Host.NewStream(ctx, p, ...)is ctx-driven. The handler goroutine is pinned for that dial, and because the dial runs outsideconnsLock(server.go:553) with no in-flight dedup, a burst of writes to a dead-led stream spins up N independent concurrent dials.Fix: build the timeout context before
m.connand pass it through bothm.connandconn.forward. NoteTestMeshUnreachableLeaderLeavesStateUntoucheduses an "absent-node" with no addresses, so it fails instantly ("no addresses") and does not exercise this path.@ -467,0 +607,4 @@if err != nil {// A failed relay leaves a broken Conn behind; drop it so the next relay// redials instead of reusing it.m.drop(ctx, leader, conn)🟠 IMPORTANT — any relay error (including a pure per-request timeout) tears down the shared peer Conn and fails concurrent sibling relays, while the write may already be applied at the leader.
m.drop(ctx, leader, conn)on every error path callsConn.Close, andConn.Close(conn.go:290) runsc.control.fail(ErrConnClosed)→releaseAll, failing every in-flight request on that Conn. So when two relays to the same leader are in flight and one times out on a healthy-but-slow leader — a timeout is not a broken Conn — the shared cached Conn is closed and the other relay fails too. Compounding this, the timed-out write was already applied at the leader, so the client receivesErrLeaderUnavailablefor a write that landed; a retry duplicates it (the relay path is at-least-once).Fix: drop only on a genuinely broken Conn (transport error /
ErrConnClosed), not on a request-cancel/timeout, and treat a timed-out relay as ambiguous rather than definitively "nothing applied locally".@ -467,0 +748,4 @@slog.String("leader", leader.String()),slog.String("error", err.Error()))return s.errorResponse(ctx, fmt.Errorf("%w: peer %s: %w", ErrLeaderUnavailable, leader, err)), true🟡 SUGGESTION — a leader's loop-guard refusal is double-wrapped, so the error matches both
ErrLeaderUnavailableandErrForwardLoop.routewraps the relay error asfmt.Errorf("%w: peer %s: %w", ErrLeaderUnavailable, leader, err). When the leader refuses withErrForwardLoop,errors.Is(res, ErrLeaderUnavailable)anderrors.Is(res, ErrForwardLoop)are both true — the test passes only becauseerrors.Isunwraps the trailing%w; the log confirms it:relaying to the stream leader failed ... error="forward loop: stream \"s0\" ...". A caller that checksErrLeaderUnavailablefirst misclassifies a membership disagreement as an unreachable leader.Fix: when
errors.Is(err, ErrForwardLoop), surface the refusal as the primary error (e.g.fmt.Errorf("%w: peer %s", err, leader)) instead of wrapping it underErrLeaderUnavailable.All three findings resolved in
20e1c28(pushed tofeat/mesh-request-relay). Two of them now have regression tests that I verified fail against the pre-fix code (df33ac5) in a scratch worktree.1. 🟠 The relay dial was not bounded — comment 573
Fixed as suggested: the
defaultWriteTimeoutcontext is now built before the dial and flows through bothm.connandconn.forward, soNewConn→host.NewStreamis bounded with the rest of the relay instead of by libp2p's own dial timeout.The no-in-flight-dedup half is also addressed: the dial now goes through a
singleflight.Groupkeyed on the peer ID, so a burst of relays to a cold or stalled leader opens one application stream, not one per request. The flight is detached from the caller's cancellation (context.WithoutCancel+ its owndefaultWriteTimeout) so a caller that gives up waiting does not abort the dial for the others, and each caller still returns on its ownctxviaDoChan+select. That mirrors the existing flight inConn.createStream.You were right that
TestMeshUnreachableLeaderLeavesStateUntoucheddoes not exercise this: its absent peer has no addresses, so the dial fails instantly. I did not add a slow-dial test — it needs either a non-routable address or a stalled peer handler and would cost the full 30s window — so this part is covered by inspection, not by a test. Say the word if you want one behind a shorter configurable bound.2. 🟠 Any relay error dropped the shared Conn — comment 574
Fixed: the Conn is now dropped only when the failure is not the caller's own context (
isContextError, i.e.context.Canceled/context.DeadlineExceeded). A relay that failed on its deadline or a cancelled caller leaves the cached Conn usable for its concurrent siblings; a broken stream, a failed write or an undecodable reply still drops it so the next relay redials.The ambiguity is now documented rather than implied away:
ErrLeaderUnavailablestates that nothing was applied locally, that a timed-out relay may already have reached the leader, and that a retry is therefore at-least-once — pair it withWithDeduplicationIDwhen the publish must stay idempotent.New test
TestMeshRelayFailureDropsOnlyABrokenConn: primes a peer Conn with a successful relay, fails a second relay with an already-cancelled context and asserts the Conn is still cached, then closes the Conn and asserts it is dropped. It fails againstdf33ac5(a relay that failed on its caller's context dropped the shared peer Conn).3. 🟡 A refusal matched both sentinels — comment 575
Fixed: the classification is now its own function,
relayError, which surfaces a refusal unchanged and wraps underErrLeaderUnavailableonly when the leader genuinely could not be reached, so the two sentinels are mutually exclusive.One correction to the severity, in your favour: on the wire this was latent rather than client-visible, because
codeForhappens to checkErrForwardLoopbeforeErrLeaderUnavailable, so the double-wrapped error still encoded ascodeForwardLoop. The defect was that the Go error value matched both and the correct answer depended on that case order — reordercodeForand it would have misreported. The fix removes the ambiguity at the source, andTestRelayErrorClassifiesARefusalasserts both directions; it fails against a shim reproducing the old wrapping (refusal err also matched ErrLeaderUnavailable).TestMeshForwardLoopIsLoudadditionally asserts now that the client-visible refusal does not matchErrLeaderUnavailable; that one passes pre-fix for exactly thecodeForordering reason above, so therelayErrortest is the real regression test for this finding.Verification (at
20e1c28)Also folded in:
unparamflagged the test helpers' now-unusedopts ...ServerOptionparameters (newMesh,startMesh), which are dropped; Phase 5 can re-add them when a durable mesh node needsWithStore.One process note: this Forgejo version's API has no endpoint to resolve review conversations (its swagger exposes only the
resolverfield), so these three inline threads need resolving in the UI. My summary comment above is the closest thing the API allows.