test: add multi-node mesh acceptance tests and benchmarks #12

Merged
sam merged 6 commits from feature/mesh-acceptance-and-benchmarks into main 2026-10-11 19:52:13 +00:00
Member

Adds the multi-node mesh coverage the plan called for, and fixes the two bugs that
coverage exposed.

Tests (acceptance_test.go, replication_test.go)

  • TestMeshAcceptance / TestMeshAcceptanceDurable: 2- and 3-node meshes, in
    memory and on durable stores, with every request driven through a node that does
    not lead the stream — create, publish and consume, a restarted replica that
    catches up on the commits it missed, a restarted leader that resumes at the next
    sequence number, and a delete that removes the stream, its command log, and the
    replicas' loops.
  • TestLeaderOutageServesReadsEverywhere: with the leader stopped, a write fails
    with ErrLeaderUnavailable while every replica still serves its own copy.
  • TestReplicaBudgetEvictionKeepsTailing: a replica's own storage budget evicts
    its oldest segments; its base moves and its next does not, so the retained window
    stays contiguous and it keeps tailing.

The mesh test helpers in mesh_test.go and discovery_test.go now take
testing.TB and build N-node meshes, so the benchmarks reuse the same hosts,
servers and stream-name placement as the tests.

Benchmarks (mesh_bench_test.go)

BenchmarkMeshForwardedPublish (a local against a relayed publish),
BenchmarkMeshReplicationThroughput (records per second for the replicating
path), BenchmarkMeshBackfill (records per second for a replica adopting a stream
it has never held), and BenchmarkMeshPublishLatency (p50 and p99 of a relayed
publish at 2 and 3 nodes).

Indicative numbers from short runs (arm64): relayed publish 140µs against 63µs
local; replication ~1.4k records/s, which is the run loop's own pace with one
command in flight; backfill ~4.4k records/s; p99 178µs at 2 nodes against 412µs at
3.

Fixes

  1. codeFor classified a relay failure as a malformed request. It checked
    io.EOF / io.ErrUnexpectedEOF before ErrLeaderUnavailable, and relayError
    wraps both the sentinel and the transport error. A relay to a leader whose
    server has stopped fails when the stream closes, so it reads as EOF and was
    reported as codeInvalidRequest. That also meant Server.replicaPull in
    server.go never took its fallback — the path that serves a read from a
    replica's own copy when no leader can be reached — so reads during a leader
    outage failed. The mesh sentinels are now classified before the
    truncated-payload mapping, which still covers a genuinely truncated request
    because such an error carries no relay sentinel.

  2. reconcilePeer adopted a replicated stream from any peer. It adopted any
    non-reserved stream a peer listed, as long as this node did not lead it, so a
    replica's copy was as good an authority as its leader's. At three or more nodes
    that lets two replicas resurrect a stream the leader has deleted: each still
    holds a copy for a pass or two, each lists it, and each re-creates it from the
    other's listing. The function's own comment and its retraction path already
    required the listing peer to be the leader; the adoption path now does too.

Verification

go build ./...
go vet ./...
golangci-lint run          # 0 issues
go test ./...
go test -race ./...
go test -count=3 -run 'TestMeshAcceptance|TestLeaderOutageServesReadsEverywhere|TestReplicaBudgetEvictionKeepsTailing' .
go test -run '^$' -bench 'BenchmarkMesh' .

Completes the acceptance, failure-case and benchmark coverage described in
plan.md Phase 5.

Adds the multi-node mesh coverage the plan called for, and fixes the two bugs that coverage exposed. ## Tests (`acceptance_test.go`, `replication_test.go`) - `TestMeshAcceptance` / `TestMeshAcceptanceDurable`: 2- and 3-node meshes, in memory and on durable stores, with every request driven through a node that does not lead the stream — create, publish and consume, a restarted replica that catches up on the commits it missed, a restarted leader that resumes at the next sequence number, and a delete that removes the stream, its command log, and the replicas' loops. - `TestLeaderOutageServesReadsEverywhere`: with the leader stopped, a write fails with `ErrLeaderUnavailable` while every replica still serves its own copy. - `TestReplicaBudgetEvictionKeepsTailing`: a replica's own storage budget evicts its oldest segments; its base moves and its next does not, so the retained window stays contiguous and it keeps tailing. The mesh test helpers in `mesh_test.go` and `discovery_test.go` now take `testing.TB` and build N-node meshes, so the benchmarks reuse the same hosts, servers and stream-name placement as the tests. ## Benchmarks (`mesh_bench_test.go`) `BenchmarkMeshForwardedPublish` (a local against a relayed publish), `BenchmarkMeshReplicationThroughput` (records per second for the replicating path), `BenchmarkMeshBackfill` (records per second for a replica adopting a stream it has never held), and `BenchmarkMeshPublishLatency` (p50 and p99 of a relayed publish at 2 and 3 nodes). Indicative numbers from short runs (arm64): relayed publish 140µs against 63µs local; replication ~1.4k records/s, which is the run loop's own pace with one command in flight; backfill ~4.4k records/s; p99 178µs at 2 nodes against 412µs at 3. ## Fixes 1. **`codeFor` classified a relay failure as a malformed request.** It checked `io.EOF` / `io.ErrUnexpectedEOF` before `ErrLeaderUnavailable`, and `relayError` wraps both the sentinel and the transport error. A relay to a leader whose server has stopped fails when the stream closes, so it reads as EOF and was reported as `codeInvalidRequest`. That also meant `Server.replicaPull` in `server.go` never took its fallback — the path that serves a read from a replica's own copy when no leader can be reached — so reads during a leader outage failed. The mesh sentinels are now classified before the truncated-payload mapping, which still covers a genuinely truncated request because such an error carries no relay sentinel. 2. **`reconcilePeer` adopted a replicated stream from any peer.** It adopted any non-reserved stream a peer listed, as long as this node did not lead it, so a replica's copy was as good an authority as its leader's. At three or more nodes that lets two replicas resurrect a stream the leader has deleted: each still holds a copy for a pass or two, each lists it, and each re-creates it from the other's listing. The function's own comment and its retraction path already required the listing peer to be the leader; the adoption path now does too. ## Verification ``` go build ./... go vet ./... golangci-lint run # 0 issues go test ./... go test -race ./... go test -count=3 -run 'TestMeshAcceptance|TestLeaderOutageServesReadsEverywhere|TestReplicaBudgetEvictionKeepsTailing' . go test -run '^$' -bench 'BenchmarkMesh' . ``` Completes the acceptance, failure-case and benchmark coverage described in `plan.md` Phase 5.
codeFor checked io.EOF and io.ErrUnexpectedEOF before ErrLeaderUnavailable, and
relayError wraps both the sentinel and the transport error. A relay to a leader
whose server has stopped therefore matched the EOF case first and was reported
as codeInvalidRequest, not codeLeaderUnavailable. A client read a malformed
request instead of a dead leader, and Server.replicaPull's fallback — the one
that serves a read from a replica's own copy when no leader can be reached —
never ran, so reads during a leader outage failed. The mesh sentinels are now
classified before the truncated-payload mapping, which still covers a genuinely
truncated request because such an error carries no relay sentinel.
reconcilePeer adopted any non-reserved stream a peer listed, as long as this
node did not lead it, so a replica's copy was as good an authority as its
leader's. With three or more nodes that lets two replicas resurrect a stream the
leader has deleted: each still holds a copy for a pass or two, each lists it,
and each re-creates it from the other's listing, while the copy is also recorded
as confirmed so the next pass may retract it again. The stream can ping-pong
indefinitely instead of converging.

reconcilePeer's own comment already stated the rule — a peer's listing
authorises a stream only for that peer to lead — and the retraction path applied
it; the adoption path now does too. A stream whose leader this node cannot reach
is no longer adopted from a replica, which matches the deletion rule: a copy is
derived state, never the authority, so it is only adopted from the writer.
test: add the multi-node mesh acceptance cases and benchmarks
All checks were successful
code-review / Agent review (pull_request) Successful in 8s
4609c99dd2
The mesh had unit tests for its parts but no end-to-end case covering several
nodes at once, and no benchmark pricing what the mesh costs. Both are added
here.

TestMeshAcceptance and TestMeshAcceptanceDurable run 2- and 3-node meshes, in
memory and on durable stores, and drive every request through a node that does
not lead the stream: create, publish and consume, a restarted replica that
catches up on the commits it missed, a restarted leader that resumes at the next
sequence number, and a delete that removes the stream, its command log, and the
replicas' loops.

TestLeaderOutageServesReadsEverywhere stops the leader and asserts a write fails
with ErrLeaderUnavailable while every replica still serves its own copy.
TestReplicaBudgetEvictionKeepsTailing squeezes only the replica's storage budget
and asserts the replica's own eviction moves its base and never its next,
leaving a contiguous window that keeps tailing.

The benchmarks price the extra hop (a local against a relayed publish),
replication throughput in records per second, backfill time for a stream a
replica has just adopted, and the p50 and p99 of a relayed publish at 2 and 3
nodes.

The mesh test helpers take testing.TB and build N-node meshes, so the benchmarks
reuse the same hosts, servers and stream-name placement as the tests instead of
duplicating them.
pi-reviewer left a comment

AI code review (samber/cc-skills-golang, verified in-container)

Reviewed against a clean clone of feature/mesh-acceptance-and-benchmarks at 4609c99 (Go 1.27.1, linux/arm64). All claims below were verified by running the commands in this session.

Pass 1 — style / naming / docs

No findings. The *testing.T → testing.TB conversion in mesh_test.go/discovery_test.go/replication_test.go/store_test.go is exactly right for benchmark reuse; doc comments are precise and the streamLedBy → streamLedByPrefix split is clean.

Pass 2 — correctness / safety / concurrency

Both production fixes verified correct:

  1. codeFor reorder (protocol.go) — correct. relayError (server.go:990-996) wraps ErrLeaderUnavailable around the transport error, so the old io.EOF-first order classified a relayed outage as codeInvalidRequest; replicaPull (server.go:2436-2445) then bailed on !errors.Is(err, ErrLeaderUnavailable) instead of falling back to serveLocalPull. New order + remoteError.Is (protocol.go:1199-1212, codeLeaderUnavailable → ErrLeaderUnavailable) make the outage read path work. go test -run TestLeaderOutageServesReadsEverywhere passes (when not flaking — see below).
  2. reconcilePeer leader-only adoption (replicate.go) — correct. The if leader != id { continue } guard makes the simplified confirm condition if s.hasStream(name) equivalent to the old leader == id && s.hasStream(name), and the retraction path below already required leader == id. Consistent with the function's documented rules.

The stopForOutage idempotent-close claim holds: Conn.Close is closeOnce-guarded (conn.go:255-269), meshPeers.close is closed.Swap-guarded (server.go:812-817), and Server.Close short-circuits on a nil store.

Pass 3 — security / dependencies

No findings. go.mod is unchanged by the PR (libp2p v0.49.0, go-multiaddr, x/sync; all else indirect); no new imports, no replace directives, no auth/crypto/input-handling changes in the diff.

Pass 4 — tests / performance / observability / modernization

  • BLOCKING: the new acceptance tests flake under load — see the inline comments on acceptance_test.go. I ran the PR's own verification commands and they fail roughly 1 in 3 runs: go test ./... (FAIL ×2), go test -count=3 -run 'TestMeshAcceptance|TestLeaderOutageServesReadsEverywhere|TestReplicaBudgetEvictionKeepsTailing' . (FAIL ×2), go test -count=1 -run TestLeaderOutageServesReadsEverywhere (FAIL). Base main passes cleanly; the durable tests pass at -count=5 in isolation. The failures cluster in the 3-node durable case (restart + outage windows).
  • go build ./..., go vet ./..., golangci-lint run ./... all clean — matches the PR's claims.
  • Benchmarks run (go test -run '^$' -bench BenchmarkMesh -benchtime=1x . PASS) but have two measurement-hygiene issues — see inline suggestions on mesh_bench_test.go.
  • Modernization: uses Go 1.27 idioms throughout (for range n, b.Loop(), tb.Context(), os.Root) — good.
## AI code review (samber/cc-skills-golang, verified in-container) Reviewed against a clean clone of `feature/mesh-acceptance-and-benchmarks` at 4609c99 (Go 1.27.1, linux/arm64). All claims below were verified by running the commands in this session. ### Pass 1 — style / naming / docs No findings. The `*testing.T` → `testing.TB` conversion in `mesh_test.go`/`discovery_test.go`/`replication_test.go`/`store_test.go` is exactly right for benchmark reuse; doc comments are precise and the `streamLedBy` → `streamLedByPrefix` split is clean. ### Pass 2 — correctness / safety / concurrency Both production fixes verified correct: 1. `codeFor` reorder (protocol.go) — correct. `relayError` (server.go:990-996) wraps `ErrLeaderUnavailable` around the transport error, so the old `io.EOF`-first order classified a relayed outage as `codeInvalidRequest`; `replicaPull` (server.go:2436-2445) then bailed on `!errors.Is(err, ErrLeaderUnavailable)` instead of falling back to `serveLocalPull`. New order + `remoteError.Is` (protocol.go:1199-1212, `codeLeaderUnavailable → ErrLeaderUnavailable`) make the outage read path work. `go test -run TestLeaderOutageServesReadsEverywhere` passes (when not flaking — see below). 2. `reconcilePeer` leader-only adoption (replicate.go) — correct. The `if leader != id { continue }` guard makes the simplified confirm condition `if s.hasStream(name)` equivalent to the old `leader == id && s.hasStream(name)`, and the retraction path below already required `leader == id`. Consistent with the function's documented rules. The `stopForOutage` idempotent-close claim holds: `Conn.Close` is `closeOnce`-guarded (conn.go:255-269), `meshPeers.close` is `closed.Swap`-guarded (server.go:812-817), and `Server.Close` short-circuits on a nil store. ### Pass 3 — security / dependencies No findings. `go.mod` is unchanged by the PR (libp2p v0.49.0, go-multiaddr, x/sync; all else indirect); no new imports, no `replace` directives, no auth/crypto/input-handling changes in the diff. ### Pass 4 — tests / performance / observability / modernization - **BLOCKING: the new acceptance tests flake under load** — see the inline comments on `acceptance_test.go`. I ran the PR's own verification commands and they fail roughly 1 in 3 runs: `go test ./...` (FAIL ×2), `go test -count=3 -run 'TestMeshAcceptance|TestLeaderOutageServesReadsEverywhere|TestReplicaBudgetEvictionKeepsTailing' .` (FAIL ×2), `go test -count=1 -run TestLeaderOutageServesReadsEverywhere` (FAIL). Base `main` passes cleanly; the durable tests pass at `-count=5` in isolation. The failures cluster in the 3-node durable case (restart + outage windows). - `go build ./...`, `go vet ./...`, `golangci-lint run ./...` all clean — matches the PR's claims. - Benchmarks run (`go test -run '^$' -bench BenchmarkMesh -benchtime=1x .` PASS) but have two measurement-hygiene issues — see inline suggestions on `mesh_bench_test.go`. - Modernization: uses Go 1.27 idioms throughout (`for range n`, `b.Loop()`, `tb.Context()`, `os.Root`) — good.
@ -0,0 +126,4 @@
case <-ctx.Done():
t.Fatalf("the consume loop on node %s was stopped before it delivered every record", node.id)
case <-timeout:
t.Fatalf("the consume loop on node %s delivered %d of %d records", node.id, len(got), count)
Member

🔴 BLOCKING — the new acceptance tests flake under load; the PR's own verification claim does not hold. Reproduced in this session, ~1 in 3 runs of the PR's exact command:

  • go test ./... → FAIL TestMeshAcceptanceDurable/3_nodes: acceptance_test.go:365 "pull on node ... with its leader down = {message:consumer not found: "readers" ...}"
  • go test ./... → FAIL TestMeshAcceptanceDurable/3_nodes: acceptance_test.go:268 "the leader kept the deleted stream's command log"
  • go test -count=3 -run 'TestMeshAcceptance|TestLeaderOutageServesReadsEverywhere|TestReplicaBudgetEvictionKeepsTailing' . → FAIL TestMeshAcceptanceDurable/3_nodes: this line — the consume loop delivered 2 of 3 records and hit the full 10s convergenceTimeout (test took 10.13s)
  • go test -count=1 -run TestLeaderOutageServesReadsEverywhere -v . → FAIL /2_nodes (0.07s)

Isolation passes (-count=5 -run TestMeshAcceptanceDurable ok), base main passes cleanly — the flakiness arrives with this PR. The assertions check point-in-time invariants after convergence windows (deleted command log on the leader, local read during the outage, in-flight consume) and race the mesh's own reconcile/retraction/replication loops. Whether this is pure test timing or exposes a real production race (e.g. command-log recreation racing the delete, or a replica pull stalling on a lagging local copy), the suite as submitted cannot gate CI. Fix: poll each invariant (wait for hasStream(internalName(name)) to clear, retry the outage read, keep the consume timeout as a last resort) and re-run the verification until -count=3 holds repeatedly.

🔴 BLOCKING — the new acceptance tests flake under load; the PR's own verification claim does not hold. Reproduced in this session, ~1 in 3 runs of the PR's exact command: - `go test ./...` → FAIL `TestMeshAcceptanceDurable/3_nodes`: acceptance_test.go:365 "pull on node ... with its leader down = {message:consumer not found: "readers" ...}" - `go test ./...` → FAIL `TestMeshAcceptanceDurable/3_nodes`: acceptance_test.go:268 "the leader kept the deleted stream's command log" - `go test -count=3 -run 'TestMeshAcceptance|TestLeaderOutageServesReadsEverywhere|TestReplicaBudgetEvictionKeepsTailing' .` → FAIL `TestMeshAcceptanceDurable/3_nodes`: this line — the consume loop delivered 2 of 3 records and hit the full 10s convergenceTimeout (test took 10.13s) - `go test -count=1 -run TestLeaderOutageServesReadsEverywhere -v .` → FAIL `/2_nodes` (0.07s) Isolation passes (`-count=5 -run TestMeshAcceptanceDurable` ok), base `main` passes cleanly — the flakiness arrives with this PR. The assertions check point-in-time invariants after convergence windows (deleted command log on the leader, local read during the outage, in-flight consume) and race the mesh's own reconcile/retraction/replication loops. Whether this is pure test timing or exposes a real production race (e.g. command-log recreation racing the delete, or a replica pull stalling on a lagging local copy), the suite as submitted cannot gate CI. Fix: poll each invariant (wait for `hasStream(internalName(name))` to clear, retry the outage read, keep the consume timeout as a last resort) and re-run the verification until `-count=3` holds repeatedly.
@ -0,0 +362,4 @@
resp := dispatchRequest(t, node.server, framePull, pullRequest{stream: name, consumer: outageConsumer})
if !resp.hasRecord || resp.record.seq != 0 {
t.Fatalf("pull on node %s with its leader down = %+v, want the first retained record", node.id, resp)
Member

🟡 SUGGESTION — the failure message is misleading for the pre-outage calls. assertReplicaServesRead is also invoked while the leader is up (the loops at lines ~214 and ~349), yet every failure reads "pull on node ... with its leader down". When a pre-outage read fails the message asserts something false, which will misdirect debugging of exactly the flake above. Consider wording it neutrally (e.g. "pull on node %s = %+v, want the first retained record").

🟡 SUGGESTION — the failure message is misleading for the pre-outage calls. `assertReplicaServesRead` is also invoked while the leader is up (the loops at lines ~214 and ~349), yet every failure reads "pull on node ... with its leader down". When a pre-outage read fails the message asserts something false, which will misdirect debugging of exactly the flake above. Consider wording it neutrally (e.g. "pull on node %s = %+v, want the first retained record").
@ -0,0 +63,4 @@
b.Helper()
ctx, cancel := context.WithTimeout(b.Context(), convergenceTimeout)
b.Cleanup(func() { cancel() })
Member

🟡 SUGGESTION — b.Cleanup + a fresh 10s timeout context are registered per awaitApplied call, i.e. once per benchmark iteration (lines 143, 185). With b.N iterations that accumulates b.N cleanup closures and live timer contexts (only released when the benchmark ends), and the per-call context allocation lands inside the timed loop, polluting the allocs/op reading. Fix: create one timeout context per benchmark before the loop and pass it in (or check ctx.Done() in the spin loop), dropping the per-iteration Cleanup.

🟡 SUGGESTION — `b.Cleanup` + a fresh 10s timeout context are registered per `awaitApplied` call, i.e. once per benchmark iteration (lines 143, 185). With b.N iterations that accumulates b.N cleanup closures and live timer contexts (only released when the benchmark ends), and the per-call context allocation lands inside the timed loop, polluting the allocs/op reading. Fix: create one timeout context per benchmark before the loop and pass it in (or check `ctx.Done()` in the spin loop), dropping the per-iteration Cleanup.
@ -0,0 +131,4 @@
func BenchmarkMeshReplicationThroughput(b *testing.B) {
b.ReportAllocs()
nodes, name := benchMesh(b, 2)
Member

🟡 SUGGESTION — mesh setup runs inside the timed region. benchMesh (host startup + create + convergence waits) is called before the loop with the timer running in all four benchmarks (lines 111, 134, 159, 201); b.Elapsed() (verified: go doc testing.B.Elapsed — "matches the one measured by B.StartTimer, B.StopTimer") includes it, so ns/op and the reported records/s are polluted by setup. Measured at -benchtime=1x: BenchmarkMeshReplicationThroughput-3 1 44733791 ns/op 22.35 records/s — at b.N=1 the metric is almost entirely setup; at the default 1s benchtime (N≈1000) the ~100-300ms setup still skews records/s by 10-30%. Fix: b.StopTimer() before benchMesh / b.StartTimer() (or b.ResetTimer()) after setup, so the metric is the replication path alone.

🟡 SUGGESTION — mesh setup runs inside the timed region. `benchMesh` (host startup + create + convergence waits) is called before the loop with the timer running in all four benchmarks (lines 111, 134, 159, 201); `b.Elapsed()` (verified: `go doc testing.B.Elapsed` — "matches the one measured by B.StartTimer, B.StopTimer") includes it, so ns/op and the reported records/s are polluted by setup. Measured at -benchtime=1x: `BenchmarkMeshReplicationThroughput-3 1 44733791 ns/op 22.35 records/s` — at b.N=1 the metric is almost entirely setup; at the default 1s benchtime (N≈1000) the ~100-300ms setup still skews records/s by 10-30%. Fix: `b.StopTimer()` before `benchMesh` / `b.StartTimer()` (or `b.ResetTimer()`) after setup, so the metric is the replication path alone.
consumerStore.minLiveWatermark used a zero watermark as its "no minimum yet"
sentinel. Zero is a real watermark, so when the live consumers were visited with
an unacked one first, every later consumer's higher watermark overwrote the
minimum. The interest trim then dropped records a consumer had never read, and
Go's map iteration order decided whether that happened on any given pass, so the
same stream kept or lost its window from run to run. The minimum is now computed
with an unset marker that cannot be a watermark, and the cache seeded at load
uses the same computation. A stream whose consumers have all acked is unaffected.
Four concurrency faults in the same area, each found by running the new
acceptance tests under load:

- A create, a delete and a first load each span the resident map and the disk,
  and nothing ordered them against each other for one name. A delete that swept
  between a create's durable write and its mount left a stream resident with no
  directory, and the leader could keep a command log for a stream it had already
  deleted. A per-name lifecycle lock, refcounted so its map grows only with the
  operations in flight, now covers all three. A holder must not take the same
  name's lock again; a load that replays its own command log takes the log's
  name, which is a different entry.

- Server.leaseCheck reported any failure to open the stream as
  ErrConsumerNotFound, including the context cancellation of a server that is
  shutting down. A replica treats that sentinel as an authoritative refusal, so
  it returned the error instead of serving the read from its own copy, and a
  client's consume loop would stop for good. Only a stream that is really gone
  is a missing consumer now; anything else is an unavailable leader, which a
  caller retries and a replica answers from its own copy.

- Server.streamBase and Server.streamMaxAge read the mounted stream after
  releasing the server lock, racing the mount that writes it.

- ensureCommandLog re-checks that the user stream still exists after creating
  its log, and ensureReplica re-checks the leader after adopting a copy: both
  are statements about a name other than the one being locked, so the lifecycle
  lock alone cannot order them.
The acceptance tests sampled state that the mesh's own goroutines write, and the
residency test read a resident entry's refcount after releasing the server lock,
which the race detector caught. Reads now go through the locked helpers or hold
the lock, and two store reads that the flush loops also guard are taken under the
store lock.

Two assertions that a client retries are now retried rather than sampled once:
deleting through a node whose peer Conn to a freshly restarted leader is stale (a
delete is idempotent), and the pre-outage read that warms a replica's mirror of
the consumer. The command-log assertion polls, because hasStream counts a load
that is still in flight. The outage read itself is still asserted in one shot.

The relayed publish and latency benchmarks already excluded their setup: b.Loop
resets the timer on its first call. The b.N-based replication-throughput and
backfill benchmarks measured it, so they stop the timer around the mesh build
now, and the wait for a replica uses one context per benchmark instead of a
context and a cleanup per iteration. At -benchtime=50x, replication throughput
was 1.05ms/op, 950 records/s and 1316 allocs/op, and is now 145us/op, 6917
records/s and 78 allocs/op.
Author
Member

Review findings resolved

All six are addressed in 7685a96, 684756a, 7669c25. The blocking flake was not test timing: each failure had a production cause, and I reproduced every one before fixing it (four concurrent test processes was enough to hit them reliably).

BLOCKING — the acceptance tests flake under load: fixed

Three separate production faults:

  1. A shutting-down leader was reported as a missing consumer. leaseCheck in server.go mapped any failure to open the stream to ErrConsumerNotFound, including the context canceled an in-flight request gets when the leader stops serving. replicaPull treats that sentinel as an authoritative refusal, so it returned the error instead of serving from its own copy — the read during a leader outage that the plan promises — and a client consume loop treats it as terminal. Only a stream that is really gone is a missing consumer now; anything else is ErrLeaderUnavailable, which the caller retries and the replica falls back from. (This was the pull on node ... = consumer not found: "readers" failure.)

  2. The durable interest trim dropped records a consumer had not read. consumerStore.minLiveWatermark used a zero watermark as its "no minimum yet" sentinel. Zero is a real watermark, so when the live consumers were visited with an unacked one first, a later consumer's higher watermark overwrote the minimum; Go's map iteration order decided whether that happened on any given pass. That is the consume loop delivered 2 of 3 records failure: the leader published Trim{base=3} while a consumer still had a record to read.

  3. A create/delete/first-load interleaving left a stream resident with no directory. createStreamLocal inserts a placeholder, writes the directory, then mounts; deleteStreamLocal removes the map entry and then the directory. A delete that swept in between left the command log mounted with nothing on disk — storeHasLog=false, resident=true — so the leader kept a command log for a stream it had deleted. A per-name, refcounted lifecycle lock now serializes create, delete and first load. ensureCommandLog and ensureReplica also re-check the other name they depend on (the user stream, the leader), which a per-name lock cannot order.

SUGGESTIONS

  1. Misleading failure message — assertReplicaServesRead no longer claims the leader is down. The pre-outage read is a separate helper (warmReplicaRead) that retries a transient relay failure; the outage read itself is still asserted in one shot.

  2. Mesh setup inside the timed region — fixed for the two b.N benchmarks (throughput, backfill) by stopping the timer around the mesh build. At -benchtime=50x: replication throughput 1.05 ms/op → 145 µs/op, 950 → 6917 records/s. This did not apply to BenchmarkMeshForwardedPublish or BenchmarkMeshPublishLatency: they use b.Loop(), whose first call resets the timer, so their setup was already outside the measurement (their numbers are unchanged, ~64 µs local vs ~138 µs relayed).

  3. Per-iteration context and cleanup — one timeout context per benchmark now, so the allocation is out of the measured loop: throughput allocs/op 1316 → 78.

Also fixed, found while verifying

  • A test data race: TestReplicatedStreamIsEvictableBetweenBatches called residentStream.idleEvictable after releasing the server lock. Resident state in the tests is now read through the locked helpers or under the lock.
  • Server.streamBase and Server.streamMaxAge read the mounted stream after releasing the lock, racing the mount that writes it. Both now read under the lock. (Pre-existing; -race caught it under load.)

Verification

go build ./... && go vet ./... && golangci-lint run        # 0 issues
go test ./...                                              # x2
go test -count=3 -run 'TestMeshAcceptance|TestLeaderOutageServesReadsEverywhere|TestReplicaBudgetEvictionKeepsTailing' .   # x2
go test -count=1 -run TestLeaderOutageServesReadsEverywhere -v .
go test -race -count=3 ./...                               # 2 processes, x2 rounds
# plus four concurrent processes x -count=5 of the acceptance/replication set

One caveat for the record: three concurrent full -race suites exhaust this 4-CPU box (dial failures in unrelated tests, and a data race in the baseline). I confirmed the same failures on main at the same parallelism, so they are environmental; every result above is from 1–2 concurrent suites.

Ready for another review pass at 7669c25.

## Review findings resolved All six are addressed in `7685a96`, `684756a`, `7669c25`. The blocking flake was not test timing: each failure had a production cause, and I reproduced every one before fixing it (four concurrent test processes was enough to hit them reliably). ### BLOCKING — the acceptance tests flake under load: fixed Three separate production faults: 1. **A shutting-down leader was reported as a missing consumer.** `leaseCheck` in `server.go` mapped *any* failure to open the stream to `ErrConsumerNotFound`, including the `context canceled` an in-flight request gets when the leader stops serving. `replicaPull` treats that sentinel as an authoritative refusal, so it returned the error instead of serving from its own copy — the read during a leader outage that the plan promises — and a client consume loop treats it as terminal. Only a stream that is really gone is a missing consumer now; anything else is `ErrLeaderUnavailable`, which the caller retries and the replica falls back from. (This was the `pull on node ... = consumer not found: "readers"` failure.) 2. **The durable interest trim dropped records a consumer had not read.** `consumerStore.minLiveWatermark` used a zero watermark as its "no minimum yet" sentinel. Zero is a real watermark, so when the live consumers were visited with an unacked one first, a later consumer's higher watermark overwrote the minimum; Go's map iteration order decided whether that happened on any given pass. That is the `consume loop delivered 2 of 3 records` failure: the leader published `Trim{base=3}` while a consumer still had a record to read. 3. **A create/delete/first-load interleaving left a stream resident with no directory.** `createStreamLocal` inserts a placeholder, writes the directory, then mounts; `deleteStreamLocal` removes the map entry and then the directory. A delete that swept in between left the command log mounted with nothing on disk — `storeHasLog=false`, `resident=true` — so the leader kept a command log for a stream it had deleted. A per-name, refcounted lifecycle lock now serializes create, delete and first load. `ensureCommandLog` and `ensureReplica` also re-check the other name they depend on (the user stream, the leader), which a per-name lock cannot order. ### SUGGESTIONS 4. **Misleading failure message** — `assertReplicaServesRead` no longer claims the leader is down. The pre-outage read is a separate helper (`warmReplicaRead`) that retries a transient relay failure; the outage read itself is still asserted in one shot. 5. **Mesh setup inside the timed region** — fixed for the two `b.N` benchmarks (throughput, backfill) by stopping the timer around the mesh build. At `-benchtime=50x`: replication throughput 1.05 ms/op → 145 µs/op, 950 → 6917 records/s. This did **not** apply to `BenchmarkMeshForwardedPublish` or `BenchmarkMeshPublishLatency`: they use `b.Loop()`, whose first call resets the timer, so their setup was already outside the measurement (their numbers are unchanged, ~64 µs local vs ~138 µs relayed). 6. **Per-iteration context and cleanup** — one timeout context per benchmark now, so the allocation is out of the measured loop: throughput allocs/op 1316 → 78. ### Also fixed, found while verifying - A **test data race**: `TestReplicatedStreamIsEvictableBetweenBatches` called `residentStream.idleEvictable` after releasing the server lock. Resident state in the tests is now read through the locked helpers or under the lock. - `Server.streamBase` and `Server.streamMaxAge` read the mounted stream after releasing the lock, racing the mount that writes it. Both now read under the lock. (Pre-existing; `-race` caught it under load.) ### Verification ``` go build ./... && go vet ./... && golangci-lint run # 0 issues go test ./... # x2 go test -count=3 -run 'TestMeshAcceptance|TestLeaderOutageServesReadsEverywhere|TestReplicaBudgetEvictionKeepsTailing' . # x2 go test -count=1 -run TestLeaderOutageServesReadsEverywhere -v . go test -race -count=3 ./... # 2 processes, x2 rounds # plus four concurrent processes x -count=5 of the acceptance/replication set ``` One caveat for the record: three concurrent full `-race` suites exhaust this 4-CPU box (dial failures in unrelated tests, and a data race in the baseline). I confirmed the same failures on `main` at the same parallelism, so they are environmental; every result above is from 1–2 concurrent suites. Ready for another review pass at `7669c25`.
sam merged commit 0549f5dfe2 into main 2026-10-11 19:52:13 +00:00
sam deleted branch feature/mesh-acceptance-and-benchmarks 2026-10-11 19:52:13 +00:00
Sign in to join this conversation.
No reviewers
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
norobo/webstream!12
No description provided.