feat: relay leader-owned requests to a stream's leader #5

Merged
sam merged 2 commits from feat/mesh-request-relay into main 2026-09-26 13:17:30 +00:00
Member

Phase 1 of plan.md (server mesh replication): the mesh's node set, the derived stream leader, and one-hop forwarding of writes. No record replication yet — that is Phases 2 and 3.

What is here

WithMeshPeers(ids ...peer.ID) on the server names the mesh's nodes. A stream's writer is argmin hash(streamName, peerID) over that set, so every node agrees on the leader from the same inputs and nothing is stored or negotiated. A leader-owned request that lands on another node is relayed to that stream's leader over the node's own peer Conn — a node is an ordinary webstream client of its peers, so there is no second wire path — and the client receives the leader's reply.

  • frameForward is one additive frame kind wrapping an inner request (its kind plus payload) plus the outer reply carrying the leader's inner reply. Existing kinds keep their encoding, and streamProtocol stays /norobo/webstream/1.0.0.
  • One hop, with a loop guard. A relayed request is served by the leader or failed with ErrForwardLoop, never relayed again, so two nodes that disagree about membership (a rolling restart with a stale node set) fail loudly instead of ping-ponging. An unreachable leader fails the request with ErrLeaderUnavailable and applies nothing locally.
  • Routed kinds are the mutations plus consumer metadata: create and delete stream, create and delete consumer, publish, ack, consume-release, get-consumer. Record reads (framePull), listings (frameListStreams) and the existence check (frameGetStream) are answered locally: a record is read from the local copy, while consumer metadata is leader-owned state that lives only on the leader until Phase 4 replicates it.
  • N = 1 is unchanged. Without WithMeshPeers there is no node set, no relay path and no outbound Conn; WithMeshPeers() with no peers is also a no-op set. A mesh deployment must give the option a set that includes the node itself and is identical on every node — the leader is a pure function of it — and must combine it with WithAllowedPeers and a host gater, because WithMeshPeers is not an inbound allowlist.

Two entry points own the two routing policies and no boolean or enum is threaded between them: clientDispatch relays a leader-owned request this node does not lead, peerDispatch fails a request a peer handed it that it does not lead. serve decodes and runs the handlers with no routing policy in it at all. Routing reads the stream name from the request's first field (frameStreamName), which every routed kind carries first; TestFrameStreamNameMatchesDecoders pins that against the full per-kind decoders so a decoder that reorders fields fails a test instead of mis-routing.

Two consequences worth knowing before review

  1. The largest publishable record shrinks by 5 bytes. maxRecordSize and the publish frame budget now reserve forwardOverhead (the wrapper's kind byte plus the inner length prefix), so a publish that fits locally is also relayable. Without the reservation a maximum-size publish would be rejected only when it had to be relayed, which would break "any node accepts a write" at the boundary.
  2. rpcStream.deliver now keeps the read buffer for a frameForward reply. A forwarded reply's raw inner payload is decoded by the forwarder after deliver returns, exactly like a record aliases the payload, so reusing the buffer would let the next frame overwrite a reply still being decoded. This bug was found while writing the relay tests and is covered by TestDeliverKeepsAliasedPayloads.

Deliberately not in this PR

  • The per-pull consume-lease check is not forwarded. plan.md lists that clause under Phase 1, but the mechanism it needs ("a pull validates its lease on the leader every time", with the reply carrying the authoritative watermark) is Phase 4's design. Phase 1 forwards consumer mutations and getConsumer; the record read and the lease stay local, so ErrConsumerBusy is not yet a cross-node guarantee.
  • No record replication yet. Phase 1 relays writes, so a stream created through a relay exists on its leader alone: a client pinned to a non-leader can create, publish and consume through the cached handle, but Conn.Stream on that node reports not-found until Phase 2 discovery and Phase 3 backfill. That is the plan's phase boundary, not a regression — getStream was never routed.
  • Placement for R < N, leader failover, dynamic membership: deferred in plan.md.

Verification

gofmt -l .                                   # clean
go build ./...                               # ok
go vet ./...                                 # ok
go test -count=1 ./...                       # ok, three consecutive runs
golangci-lint run --new-from-rev=origin/main # 0 issues
racedetector test -run 'TestMesh|TestHashLeader|TestDeliverKeeps|TestFrameStreamName' .  # 0 races
go test -run '^$' -bench 'BenchmarkServerPublish$' -benchtime=10x .  # ~2.0 us/op, 1 alloc/op

The new tests are in mesh_test.go: two nodes in one process (separate libp2p hosts, because libp2p refuses to self-dial) covering create and publish relayed to the leader with the leader's sequence numbers and no local state on the non-leader, the leader serving locally even with an unreachable peer in its set, a pull staying local while getConsumer is answered by the leader, a membership disagreement yielding ErrForwardLoop, an unreachable leader yielding ErrLeaderUnavailable with no local write, and the no-mesh option forwarding nothing. fuzz_test.go gains FuzzDecodeForward and the frameForward reply seed.

Pre-existing failures this PR does not introduce

Both reproduce on the parent commit in a separate worktree, so a red run is not from this change:

  • racedetector test ./... already reports ~1.5k WARNING: DATA RACE (dominated by TestConcurrentDurablePublishOnDiskOrder, in Server.acquire/publish) and TestConsumeSurvivesEviction already panics on an unlocked server.streams[name].stream read in integration_test.go.
  • go test ./... is roughly 1-in-25 flaky at the base commit too (TestConsumeSurvivesEviction, TestMinLiveWatermarkTracksCreateDeletePut), load-dependent.
  • golangci-lint run with the workspace's current go-linters default config reports ~84 issues on the base commit; --new-from-rev is the clean signal used above.
Phase 1 of `plan.md` (server mesh replication): the mesh's node set, the derived stream leader, and one-hop forwarding of writes. No record replication yet — that is Phases 2 and 3. ## What is here `WithMeshPeers(ids ...peer.ID)` on the server names the mesh's nodes. A stream's writer is `argmin hash(streamName, peerID)` over that set, so every node agrees on the leader from the same inputs and nothing is stored or negotiated. A leader-owned request that lands on another node is relayed to that stream's leader over the node's own peer `Conn` — a node is an ordinary webstream client of its peers, so there is no second wire path — and the client receives the leader's reply. - **`frameForward`** is one additive frame kind wrapping an inner request (its kind plus payload) plus the outer reply carrying the leader's inner reply. Existing kinds keep their encoding, and `streamProtocol` stays `/norobo/webstream/1.0.0`. - **One hop, with a loop guard.** A relayed request is served by the leader or failed with `ErrForwardLoop`, never relayed again, so two nodes that disagree about membership (a rolling restart with a stale node set) fail loudly instead of ping-ponging. An unreachable leader fails the request with `ErrLeaderUnavailable` and applies nothing locally. - **Routed kinds** are the mutations plus consumer metadata: create and delete stream, create and delete consumer, publish, ack, consume-release, get-consumer. Record reads (`framePull`), listings (`frameListStreams`) and the existence check (`frameGetStream`) are answered locally: a record is read from the local copy, while consumer metadata is leader-owned state that lives only on the leader until Phase 4 replicates it. - **`N = 1` is unchanged.** Without `WithMeshPeers` there is no node set, no relay path and no outbound `Conn`; `WithMeshPeers()` with no peers is also a no-op set. A mesh deployment must give the option a set that includes the node itself and is identical on every node — the leader is a pure function of it — and must combine it with `WithAllowedPeers` and a host gater, because `WithMeshPeers` is not an inbound allowlist. Two entry points own the two routing policies and no boolean or enum is threaded between them: `clientDispatch` relays a leader-owned request this node does not lead, `peerDispatch` fails a request a peer handed it that it does not lead. `serve` decodes and runs the handlers with no routing policy in it at all. Routing reads the stream name from the request's first field (`frameStreamName`), which every routed kind carries first; `TestFrameStreamNameMatchesDecoders` pins that against the full per-kind decoders so a decoder that reorders fields fails a test instead of mis-routing. ## Two consequences worth knowing before review 1. **The largest publishable record shrinks by 5 bytes.** `maxRecordSize` and the publish frame budget now reserve `forwardOverhead` (the wrapper's kind byte plus the inner length prefix), so a publish that fits locally is also relayable. Without the reservation a maximum-size publish would be rejected only when it had to be relayed, which would break "any node accepts a write" at the boundary. 2. **`rpcStream.deliver` now keeps the read buffer for a `frameForward` reply.** A forwarded reply's `raw` inner payload is decoded by the forwarder *after* `deliver` returns, exactly like a record aliases the payload, so reusing the buffer would let the next frame overwrite a reply still being decoded. This bug was found while writing the relay tests and is covered by `TestDeliverKeepsAliasedPayloads`. ## Deliberately not in this PR - **The per-pull consume-lease check is not forwarded.** `plan.md` lists that clause under Phase 1, but the mechanism it needs ("a pull validates its lease on the leader every time", with the reply carrying the authoritative watermark) is Phase 4's design. Phase 1 forwards consumer *mutations* and `getConsumer`; the record read and the lease stay local, so `ErrConsumerBusy` is not yet a cross-node guarantee. - **No record replication yet.** Phase 1 relays writes, so a stream created through a relay exists on its leader alone: a client pinned to a non-leader can create, publish and consume through the cached handle, but `Conn.Stream` on that node reports not-found until Phase 2 discovery and Phase 3 backfill. That is the plan's phase boundary, not a regression — `getStream` was never routed. - Placement for `R < N`, leader failover, dynamic membership: deferred in `plan.md`. ## Verification ``` gofmt -l . # clean go build ./... # ok go vet ./... # ok go test -count=1 ./... # ok, three consecutive runs golangci-lint run --new-from-rev=origin/main # 0 issues racedetector test -run 'TestMesh|TestHashLeader|TestDeliverKeeps|TestFrameStreamName' . # 0 races go test -run '^$' -bench 'BenchmarkServerPublish$' -benchtime=10x . # ~2.0 us/op, 1 alloc/op ``` The new tests are in `mesh_test.go`: two nodes in one process (separate libp2p hosts, because libp2p refuses to self-dial) covering create and publish relayed to the leader with the leader's sequence numbers and no local state on the non-leader, the leader serving locally even with an unreachable peer in its set, a pull staying local while `getConsumer` is answered by the leader, a membership disagreement yielding `ErrForwardLoop`, an unreachable leader yielding `ErrLeaderUnavailable` with no local write, and the no-mesh option forwarding nothing. `fuzz_test.go` gains `FuzzDecodeForward` and the `frameForward` reply seed. ## Pre-existing failures this PR does not introduce Both reproduce on the parent commit in a separate worktree, so a red run is not from this change: - `racedetector test ./...` already reports ~1.5k `WARNING: DATA RACE` (dominated by `TestConcurrentDurablePublishOnDiskOrder`, in `Server.acquire`/`publish`) and `TestConsumeSurvivesEviction` already panics on an unlocked `server.streams[name].stream` read in `integration_test.go`. - `go test ./...` is roughly 1-in-25 flaky at the base commit too (`TestConsumeSurvivesEviction`, `TestMinLiveWatermarkTracksCreateDeletePut`), load-dependent. - `golangci-lint run` with the workspace's current go-linters default config reports ~84 issues on the base commit; `--new-from-rev` is the clean signal used above.
feat: relay leader-owned requests to a stream's leader
All checks were successful
code-review / Agent review (pull_request) Successful in 10s
df33ac5918
Phase 1 of plan.md (server mesh replication). WithMeshPeers(ids...) names the
mesh's nodes, and a stream's writer is derived from them as
argmin hash(streamName, peerID), so every node agrees on the leader without
storing or negotiating one. A leader-owned request that lands on another node
is relayed to that stream's leader over the node's own peer Conn and the client
receives the leader's reply, so a client never has to discover the leader and
never sees a wrong-node error.

- frameForward is one additive frame kind wrapping an inner request, so the
  encoding of the existing kinds is untouched. A relayed request is served by
  the leader or failed with ErrForwardLoop, never relayed again, which turns a
  membership disagreement (a rolling restart with a stale node set) into a loud
  error instead of a ping-pong loop. An unreachable leader fails the request
  with ErrLeaderUnavailable and applies nothing locally.
- The two routing policies are two entry points: clientDispatch relays a
  leader-owned request this node does not lead, peerDispatch fails a request a
  peer handed it that it does not lead. serve decodes and runs the handlers
  with no routing policy at all.
- Routed kinds are the mutations plus consumer metadata: create and delete
  stream and consumer, publish, ack, consume-release and get-consumer. Record
  reads and listings stay local, because a record is read from the local copy
  while consumer metadata is leader-owned state.
- maxRecordSize and the publish frame budget reserve the wrapper's five bytes,
  so a publish that fits locally stays relayable; the largest publishable
  record shrinks by those five bytes.
- rpcStream.deliver keeps the read buffer for a frameForward reply, whose raw
  inner payload aliases it and is decoded after deliver returns.

Records are not replicated yet (Phase 3), so a stream created through a relay
exists on its leader alone until Phase 2 discovery and Phase 3 backfill land.
pi-agent left a comment

Reviewed PR #5 (feat/mesh-request-relay) against the samber/cc-skills-golang skill set, verified against the PR head (df33ac5) in this session: go build ./... clean, go vet ./... clean, full go test -count=1 ./... passes, racedetector test -run 'TestMesh|TestDeliverKeepsAliasedPayloads|TestConcurrentDurablePublishOnDiskOrder' reports no races, and the mesh + fuzz seed tests pass.

High quality overall: the routing split (clientDispatch / peerDispatch / serve) with no flag threaded between entry points, the single-source routed-kind list pinned by TestFrameStreamNameMatchesDecoders, the frameForward read-buffer aliasing fix with TestDeliverKeepsAliasedPayloads, and the meshPeers connsLock/closed/drop-by-identity concurrency are all sound and well-documented. I checked the forward-overhead reservation end to end: a max-size publish stays relayable and the leader's read path never exceeds maxFrameSize. I found no blocking correctness, safety, or data-race issue.

Three non-blocking findings (two IMPORTANT, one SUGGESTION), all in meshPeers.forward/route, each about the relay's timeout and failure handling, each with a concrete fix.

Reviewed PR #5 (feat/mesh-request-relay) against the samber/cc-skills-golang skill set, verified against the PR head (df33ac5) in this session: `go build ./...` clean, `go vet ./...` clean, full `go test -count=1 ./...` passes, `racedetector test -run 'TestMesh|TestDeliverKeepsAliasedPayloads|TestConcurrentDurablePublishOnDiskOrder'` reports no races, and the mesh + fuzz seed tests pass. High quality overall: the routing split (clientDispatch / peerDispatch / serve) with no flag threaded between entry points, the single-source routed-kind list pinned by TestFrameStreamNameMatchesDecoders, the frameForward read-buffer aliasing fix with TestDeliverKeepsAliasedPayloads, and the meshPeers connsLock/closed/drop-by-identity concurrency are all sound and well-documented. I checked the forward-overhead reservation end to end: a max-size publish stays relayable and the leader's read path never exceeds maxFrameSize. I found no blocking correctness, safety, or data-race issue. Three non-blocking findings (two IMPORTANT, one SUGGESTION), all in meshPeers.forward/route, each about the relay's timeout and failure handling, each with a concrete fix.
server.go Outdated
@ -467,0 +588,4 @@
// forward relays one request to the stream's leader and returns the leader's
// reply to that request.
func (m *meshPeers) forward(ctx context.Context, leader peer.ID, kind frameKind, payload []byte) (response, error) {
conn, err := m.conn(ctx, leader)
Author
Member

🟠 IMPORTANT — the relay dial is not bounded by the server's write timeout.

m.conn(ctx, leader) at line 591 runs NewConn → host.NewStream(ctx, ...) (server.go:555) before the context.WithTimeout(ctx, defaultWriteTimeout) at line 602 is applied. The ctx here is the inbound handler's long-lived stream context (no deadline), so the dial to an unreachable leader that still advertises addresses is bounded only by libp2p's implicit dial timeout, not by defaultWriteTimeout (30s) — contradicting the inline comment at line 596 ("an unreachable leader must fail the request rather than hang it"). Verified: defaultDialTimeout = 15 * time.Second in $GOMODCACHE/github.com/libp2p/go-libp2p@v0.49.0/p2p/net/swarm/swarm.go:29; Host.NewStream(ctx, p, ...) is ctx-driven. The handler goroutine is pinned for that dial, and because the dial runs outside connsLock (server.go:553) with no in-flight dedup, a burst of writes to a dead-led stream spins up N independent concurrent dials.

Fix: build the timeout context before m.conn and pass it through both m.conn and conn.forward. Note TestMeshUnreachableLeaderLeavesStateUntouched uses an "absent-node" with no addresses, so it fails instantly ("no addresses") and does not exercise this path.

🟠 IMPORTANT — the relay dial is not bounded by the server's write timeout. `m.conn(ctx, leader)` at line 591 runs `NewConn` → `host.NewStream(ctx, ...)` (server.go:555) *before* the `context.WithTimeout(ctx, defaultWriteTimeout)` at line 602 is applied. The `ctx` here is the inbound handler's long-lived stream context (no deadline), so the dial to an unreachable leader that still advertises addresses is bounded only by libp2p's implicit dial timeout, not by `defaultWriteTimeout` (30s) — contradicting the inline comment at line 596 ("an unreachable leader must fail the request rather than hang it"). Verified: `defaultDialTimeout = 15 * time.Second` in `$GOMODCACHE/github.com/libp2p/go-libp2p@v0.49.0/p2p/net/swarm/swarm.go:29`; `Host.NewStream(ctx, p, ...)` is ctx-driven. The handler goroutine is pinned for that dial, and because the dial runs outside `connsLock` (server.go:553) with no in-flight dedup, a burst of writes to a dead-led stream spins up N independent concurrent dials. Fix: build the timeout context before `m.conn` and pass it through both `m.conn` and `conn.forward`. Note `TestMeshUnreachableLeaderLeavesStateUntouched` uses an "absent-node" with no addresses, so it fails instantly ("no addresses") and does not exercise this path.
server.go Outdated
@ -467,0 +607,4 @@
if err != nil {
// A failed relay leaves a broken Conn behind; drop it so the next relay
// redials instead of reusing it.
m.drop(ctx, leader, conn)
Author
Member

🟠 IMPORTANT — any relay error (including a pure per-request timeout) tears down the shared peer Conn and fails concurrent sibling relays, while the write may already be applied at the leader.

m.drop(ctx, leader, conn) on every error path calls Conn.Close, and Conn.Close (conn.go:290) runs c.control.fail(ErrConnClosed) → releaseAll, failing every in-flight request on that Conn. So when two relays to the same leader are in flight and one times out on a healthy-but-slow leader — a timeout is not a broken Conn — the shared cached Conn is closed and the other relay fails too. Compounding this, the timed-out write was already applied at the leader, so the client receives ErrLeaderUnavailable for a write that landed; a retry duplicates it (the relay path is at-least-once).

Fix: drop only on a genuinely broken Conn (transport error / ErrConnClosed), not on a request-cancel/timeout, and treat a timed-out relay as ambiguous rather than definitively "nothing applied locally".

🟠 IMPORTANT — any relay error (including a pure per-request timeout) tears down the shared peer Conn and fails concurrent sibling relays, while the write may already be applied at the leader. `m.drop(ctx, leader, conn)` on *every* error path calls `Conn.Close`, and `Conn.Close` (conn.go:290) runs `c.control.fail(ErrConnClosed)` → `releaseAll`, failing every in-flight request on that Conn. So when two relays to the same leader are in flight and one times out on a healthy-but-slow leader — a timeout is not a broken Conn — the shared cached Conn is closed and the other relay fails too. Compounding this, the timed-out write was already applied at the leader, so the client receives `ErrLeaderUnavailable` for a write that landed; a retry duplicates it (the relay path is at-least-once). Fix: drop only on a genuinely broken Conn (transport error / `ErrConnClosed`), not on a request-cancel/timeout, and treat a timed-out relay as ambiguous rather than definitively "nothing applied locally".
server.go Outdated
@ -467,0 +748,4 @@
slog.String("leader", leader.String()),
slog.String("error", err.Error()))
return s.errorResponse(ctx, fmt.Errorf("%w: peer %s: %w", ErrLeaderUnavailable, leader, err)), true
Author
Member

🟡 SUGGESTION — a leader's loop-guard refusal is double-wrapped, so the error matches both ErrLeaderUnavailable and ErrForwardLoop.

route wraps the relay error as fmt.Errorf("%w: peer %s: %w", ErrLeaderUnavailable, leader, err). When the leader refuses with ErrForwardLoop, errors.Is(res, ErrLeaderUnavailable) and errors.Is(res, ErrForwardLoop) are both true — the test passes only because errors.Is unwraps the trailing %w; the log confirms it: relaying to the stream leader failed ... error="forward loop: stream \"s0\" ...". A caller that checks ErrLeaderUnavailable first misclassifies a membership disagreement as an unreachable leader.

Fix: when errors.Is(err, ErrForwardLoop), surface the refusal as the primary error (e.g. fmt.Errorf("%w: peer %s", err, leader)) instead of wrapping it under ErrLeaderUnavailable.

🟡 SUGGESTION — a leader's loop-guard refusal is double-wrapped, so the error matches both `ErrLeaderUnavailable` and `ErrForwardLoop`. `route` wraps the relay error as `fmt.Errorf("%w: peer %s: %w", ErrLeaderUnavailable, leader, err)`. When the leader refuses with `ErrForwardLoop`, `errors.Is(res, ErrLeaderUnavailable)` *and* `errors.Is(res, ErrForwardLoop)` are both true — the test passes only because `errors.Is` unwraps the trailing `%w`; the log confirms it: `relaying to the stream leader failed ... error="forward loop: stream \"s0\" ..."`. A caller that checks `ErrLeaderUnavailable` first misclassifies a membership disagreement as an unreachable leader. Fix: when `errors.Is(err, ErrForwardLoop)`, surface the refusal as the primary error (e.g. `fmt.Errorf("%w: peer %s", err, leader)`) instead of wrapping it under `ErrLeaderUnavailable`.
Resolves the three findings of the PR review on the Phase 1 relay.

- The relay's defaultWriteTimeout is now imposed before the peer dial, so the
  dial (NewConn -> host.NewStream) is bounded with the rest of the relay and an
  unreachable leader cannot pin a handler goroutine for libp2p's own dial
  timeout. Concurrent first dials to one peer are coalesced with singleflight,
  so a burst of relays to a cold leader opens one application stream instead of
  one per request, each of which costs the leader a handler goroutine and a
  streamsPerPeer slot.
- A relay that failed on its caller's context or deadline no longer drops the
  cached peer Conn. Dropping closes it and fails every concurrent relay to the
  same leader, and a relay that timed out may already have reached the leader,
  so ErrLeaderUnavailable now documents that outcome as ambiguous rather than
  "nothing applied".
- A leader's refusal is no longer wrapped under ErrLeaderUnavailable, so the
  two sentinels are mutually exclusive and a caller checking
  ErrLeaderUnavailable first cannot classify a membership disagreement as a
  dead node. The classification lives in relayError.

Tests: TestRelayErrorClassifiesARefusal and
TestMeshRelayFailureDropsOnlyABrokenConn both fail against the pre-fix code.
Author
Member

All three findings resolved in 20e1c28 (pushed to feat/mesh-request-relay). Two of them now have regression tests that I verified fail against the pre-fix code (df33ac5) in a scratch worktree.

1. 🟠 The relay dial was not bounded — comment 573

Fixed as suggested: the defaultWriteTimeout context is now built before the dial and flows through both m.conn and conn.forward, so NewConn → host.NewStream is bounded with the rest of the relay instead of by libp2p's own dial timeout.

The no-in-flight-dedup half is also addressed: the dial now goes through a singleflight.Group keyed on the peer ID, so a burst of relays to a cold or stalled leader opens one application stream, not one per request. The flight is detached from the caller's cancellation (context.WithoutCancel + its own defaultWriteTimeout) so a caller that gives up waiting does not abort the dial for the others, and each caller still returns on its own ctx via DoChan + select. That mirrors the existing flight in Conn.createStream.

You were right that TestMeshUnreachableLeaderLeavesStateUntouched does not exercise this: its absent peer has no addresses, so the dial fails instantly. I did not add a slow-dial test — it needs either a non-routable address or a stalled peer handler and would cost the full 30s window — so this part is covered by inspection, not by a test. Say the word if you want one behind a shorter configurable bound.

2. 🟠 Any relay error dropped the shared Conn — comment 574

Fixed: the Conn is now dropped only when the failure is not the caller's own context (isContextError, i.e. context.Canceled / context.DeadlineExceeded). A relay that failed on its deadline or a cancelled caller leaves the cached Conn usable for its concurrent siblings; a broken stream, a failed write or an undecodable reply still drops it so the next relay redials.

The ambiguity is now documented rather than implied away: ErrLeaderUnavailable states that nothing was applied locally, that a timed-out relay may already have reached the leader, and that a retry is therefore at-least-once — pair it with WithDeduplicationID when the publish must stay idempotent.

New test TestMeshRelayFailureDropsOnlyABrokenConn: primes a peer Conn with a successful relay, fails a second relay with an already-cancelled context and asserts the Conn is still cached, then closes the Conn and asserts it is dropped. It fails against df33ac5 (a relay that failed on its caller's context dropped the shared peer Conn).

3. 🟡 A refusal matched both sentinels — comment 575

Fixed: the classification is now its own function, relayError, which surfaces a refusal unchanged and wraps under ErrLeaderUnavailable only when the leader genuinely could not be reached, so the two sentinels are mutually exclusive.

One correction to the severity, in your favour: on the wire this was latent rather than client-visible, because codeFor happens to check ErrForwardLoop before ErrLeaderUnavailable, so the double-wrapped error still encoded as codeForwardLoop. The defect was that the Go error value matched both and the correct answer depended on that case order — reorder codeFor and it would have misreported. The fix removes the ambiguity at the source, and TestRelayErrorClassifiesARefusal asserts both directions; it fails against a shim reproducing the old wrapping (refusal err also matched ErrLeaderUnavailable).

TestMeshForwardLoopIsLoud additionally asserts now that the client-visible refusal does not match ErrLeaderUnavailable; that one passes pre-fix for exactly the codeFor ordering reason above, so the relayError test is the real regression test for this finding.

Verification (at 20e1c28)

gofmt -l .                                    # clean
go build ./...                                # ok
go vet ./...                                  # ok
go test -count=1 ./...                        # ok (3 runs)
golangci-lint run --new-from-rev=origin/main  # 0 issues
racedetector test -run 'TestMesh|TestHashLeader|TestDeliver|TestFrameStreamName|TestRelayError' .  # 0 races

Also folded in: unparam flagged the test helpers' now-unused opts ...ServerOption parameters (newMesh, startMesh), which are dropped; Phase 5 can re-add them when a durable mesh node needs WithStore.

One process note: this Forgejo version's API has no endpoint to resolve review conversations (its swagger exposes only the resolver field), so these three inline threads need resolving in the UI. My summary comment above is the closest thing the API allows.

All three findings resolved in `20e1c28` (pushed to `feat/mesh-request-relay`). Two of them now have regression tests that I verified fail against the pre-fix code (`df33ac5`) in a scratch worktree. ### 1. 🟠 The relay dial was not bounded — [comment 573](https://git.norobo.chat/norobo/webstream/pulls/5#issuecomment-573) Fixed as suggested: the `defaultWriteTimeout` context is now built **before** the dial and flows through both `m.conn` and `conn.forward`, so `NewConn` → `host.NewStream` is bounded with the rest of the relay instead of by libp2p's own dial timeout. The no-in-flight-dedup half is also addressed: the dial now goes through a `singleflight.Group` keyed on the peer ID, so a burst of relays to a cold or stalled leader opens **one** application stream, not one per request. The flight is detached from the caller's cancellation (`context.WithoutCancel` + its own `defaultWriteTimeout`) so a caller that gives up waiting does not abort the dial for the others, and each caller still returns on its own `ctx` via `DoChan` + `select`. That mirrors the existing flight in `Conn.createStream`. You were right that `TestMeshUnreachableLeaderLeavesStateUntouched` does not exercise this: its absent peer has no addresses, so the dial fails instantly. I did not add a slow-dial test — it needs either a non-routable address or a stalled peer handler and would cost the full 30s window — so this part is covered by inspection, not by a test. Say the word if you want one behind a shorter configurable bound. ### 2. 🟠 Any relay error dropped the shared Conn — [comment 574](https://git.norobo.chat/norobo/webstream/pulls/5#issuecomment-574) Fixed: the Conn is now dropped only when the failure is not the caller's own context (`isContextError`, i.e. `context.Canceled` / `context.DeadlineExceeded`). A relay that failed on its deadline or a cancelled caller leaves the cached Conn usable for its concurrent siblings; a broken stream, a failed write or an undecodable reply still drops it so the next relay redials. The ambiguity is now documented rather than implied away: `ErrLeaderUnavailable` states that nothing was applied **locally**, that a timed-out relay may already have reached the leader, and that a retry is therefore at-least-once — pair it with `WithDeduplicationID` when the publish must stay idempotent. New test `TestMeshRelayFailureDropsOnlyABrokenConn`: primes a peer Conn with a successful relay, fails a second relay with an already-cancelled context and asserts the Conn is still cached, then closes the Conn and asserts it *is* dropped. It fails against `df33ac5` (`a relay that failed on its caller's context dropped the shared peer Conn`). ### 3. 🟡 A refusal matched both sentinels — [comment 575](https://git.norobo.chat/norobo/webstream/pulls/5#issuecomment-575) Fixed: the classification is now its own function, `relayError`, which surfaces a refusal unchanged and wraps under `ErrLeaderUnavailable` only when the leader genuinely could not be reached, so the two sentinels are mutually exclusive. One correction to the severity, in your favour: on the wire this was latent rather than client-visible, because `codeFor` happens to check `ErrForwardLoop` before `ErrLeaderUnavailable`, so the double-wrapped error still encoded as `codeForwardLoop`. The defect was that the Go error value matched both and the correct answer depended on that case order — reorder `codeFor` and it would have misreported. The fix removes the ambiguity at the source, and `TestRelayErrorClassifiesARefusal` asserts both directions; it fails against a shim reproducing the old wrapping (`refusal err also matched ErrLeaderUnavailable`). `TestMeshForwardLoopIsLoud` additionally asserts now that the client-visible refusal does not match `ErrLeaderUnavailable`; that one passes pre-fix for exactly the `codeFor` ordering reason above, so the `relayError` test is the real regression test for this finding. ### Verification (at `20e1c28`) ``` gofmt -l . # clean go build ./... # ok go vet ./... # ok go test -count=1 ./... # ok (3 runs) golangci-lint run --new-from-rev=origin/main # 0 issues racedetector test -run 'TestMesh|TestHashLeader|TestDeliver|TestFrameStreamName|TestRelayError' . # 0 races ``` Also folded in: `unparam` flagged the test helpers' now-unused `opts ...ServerOption` parameters (`newMesh`, `startMesh`), which are dropped; Phase 5 can re-add them when a durable mesh node needs `WithStore`. One process note: this Forgejo version's API has no endpoint to resolve review conversations (its swagger exposes only the `resolver` field), so these three inline threads need resolving in the UI. My summary comment above is the closest thing the API allows.
sam merged commit 183d2ea455 into main 2026-09-26 13:17:30 +00:00
sam deleted branch feat/mesh-request-relay 2026-09-26 13:17:30 +00:00
Sign in to join this conversation.
No reviewers
No labels
code-review
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
norobo/webstream!5
No description provided.