test: add multi-node mesh acceptance tests and benchmarks #12
No reviewers
Labels
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
norobo/webstream!12
Loading…
Reference in a new issue
No description provided.
Delete branch "feature/mesh-acceptance-and-benchmarks"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Adds the multi-node mesh coverage the plan called for, and fixes the two bugs that
coverage exposed.
Tests (
acceptance_test.go,replication_test.go)TestMeshAcceptance/TestMeshAcceptanceDurable: 2- and 3-node meshes, inmemory and on durable stores, with every request driven through a node that does
not lead the stream â create, publish and consume, a restarted replica that
catches up on the commits it missed, a restarted leader that resumes at the next
sequence number, and a delete that removes the stream, its command log, and the
replicas' loops.
TestLeaderOutageServesReadsEverywhere: with the leader stopped, a write failswith
ErrLeaderUnavailablewhile every replica still serves its own copy.TestReplicaBudgetEvictionKeepsTailing: a replica's own storage budget evictsits oldest segments; its base moves and its next does not, so the retained window
stays contiguous and it keeps tailing.
The mesh test helpers in
mesh_test.goanddiscovery_test.gonow taketesting.TBand build N-node meshes, so the benchmarks reuse the same hosts,servers and stream-name placement as the tests.
Benchmarks (
mesh_bench_test.go)BenchmarkMeshForwardedPublish(a local against a relayed publish),BenchmarkMeshReplicationThroughput(records per second for the replicatingpath),
BenchmarkMeshBackfill(records per second for a replica adopting a streamit has never held), and
BenchmarkMeshPublishLatency(p50 and p99 of a relayedpublish at 2 and 3 nodes).
Indicative numbers from short runs (arm64): relayed publish 140µs against 63µs
local; replication ~1.4k records/s, which is the run loop's own pace with one
command in flight; backfill ~4.4k records/s; p99 178µs at 2 nodes against 412µs at
3.
Fixes
codeForclassified a relay failure as a malformed request. It checkedio.EOF/io.ErrUnexpectedEOFbeforeErrLeaderUnavailable, andrelayErrorwraps both the sentinel and the transport error. A relay to a leader whose
server has stopped fails when the stream closes, so it reads as EOF and was
reported as
codeInvalidRequest. That also meantServer.replicaPullinserver.gonever took its fallback â the path that serves a read from areplica's own copy when no leader can be reached â so reads during a leader
outage failed. The mesh sentinels are now classified before the
truncated-payload mapping, which still covers a genuinely truncated request
because such an error carries no relay sentinel.
reconcilePeeradopted a replicated stream from any peer. It adopted anynon-reserved stream a peer listed, as long as this node did not lead it, so a
replica's copy was as good an authority as its leader's. At three or more nodes
that lets two replicas resurrect a stream the leader has deleted: each still
holds a copy for a pass or two, each lists it, and each re-creates it from the
other's listing. The function's own comment and its retraction path already
required the listing peer to be the leader; the adoption path now does too.
Verification
Completes the acceptance, failure-case and benchmark coverage described in
plan.mdPhase 5.AI code review (samber/cc-skills-golang, verified in-container)
Reviewed against a clean clone of
feature/mesh-acceptance-and-benchmarksat4609c99(Go 1.27.1, linux/arm64). All claims below were verified by running the commands in this session.Pass 1 — style / naming / docs
No findings. The
*testing.T→testing.TBconversion inmesh_test.go/discovery_test.go/replication_test.go/store_test.gois exactly right for benchmark reuse; doc comments are precise and thestreamLedBy→streamLedByPrefixsplit is clean.Pass 2 — correctness / safety / concurrency
Both production fixes verified correct:
codeForreorder (protocol.go) — correct.relayError(server.go:990-996) wrapsErrLeaderUnavailablearound the transport error, so the oldio.EOF-first order classified a relayed outage ascodeInvalidRequest;replicaPull(server.go:2436-2445) then bailed on!errors.Is(err, ErrLeaderUnavailable)instead of falling back toserveLocalPull. New order +remoteError.Is(protocol.go:1199-1212,codeLeaderUnavailable → ErrLeaderUnavailable) make the outage read path work.go test -run TestLeaderOutageServesReadsEverywherepasses (when not flaking — see below).reconcilePeerleader-only adoption (replicate.go) — correct. Theif leader != id { continue }guard makes the simplified confirm conditionif s.hasStream(name)equivalent to the oldleader == id && s.hasStream(name), and the retraction path below already requiredleader == id. Consistent with the function's documented rules.The
stopForOutageidempotent-close claim holds:Conn.CloseiscloseOnce-guarded (conn.go:255-269),meshPeers.closeisclosed.Swap-guarded (server.go:812-817), andServer.Closeshort-circuits on a nil store.Pass 3 — security / dependencies
No findings.
go.modis unchanged by the PR (libp2p v0.49.0, go-multiaddr, x/sync; all else indirect); no new imports, noreplacedirectives, no auth/crypto/input-handling changes in the diff.Pass 4 — tests / performance / observability / modernization
acceptance_test.go. I ran the PR's own verification commands and they fail roughly 1 in 3 runs:go test ./...(FAIL ×2),go test -count=3 -run 'TestMeshAcceptance|TestLeaderOutageServesReadsEverywhere|TestReplicaBudgetEvictionKeepsTailing' .(FAIL ×2),go test -count=1 -run TestLeaderOutageServesReadsEverywhere(FAIL). Basemainpasses cleanly; the durable tests pass at-count=5in isolation. The failures cluster in the 3-node durable case (restart + outage windows).go build ./...,go vet ./...,golangci-lint run ./...all clean — matches the PR's claims.go test -run '^$' -bench BenchmarkMesh -benchtime=1x .PASS) but have two measurement-hygiene issues — see inline suggestions onmesh_bench_test.go.for range n,b.Loop(),tb.Context(),os.Root) — good.@ -0,0 +126,4 @@case <-ctx.Done():t.Fatalf("the consume loop on node %s was stopped before it delivered every record", node.id)case <-timeout:t.Fatalf("the consume loop on node %s delivered %d of %d records", node.id, len(got), count)🔴 BLOCKING — the new acceptance tests flake under load; the PR's own verification claim does not hold. Reproduced in this session, ~1 in 3 runs of the PR's exact command:
go test ./...→ FAILTestMeshAcceptanceDurable/3_nodes: acceptance_test.go:365 "pull on node ... with its leader down = {message:consumer not found: "readers" ...}"go test ./...→ FAILTestMeshAcceptanceDurable/3_nodes: acceptance_test.go:268 "the leader kept the deleted stream's command log"go test -count=3 -run 'TestMeshAcceptance|TestLeaderOutageServesReadsEverywhere|TestReplicaBudgetEvictionKeepsTailing' .→ FAILTestMeshAcceptanceDurable/3_nodes: this line — the consume loop delivered 2 of 3 records and hit the full 10s convergenceTimeout (test took 10.13s)go test -count=1 -run TestLeaderOutageServesReadsEverywhere -v .→ FAIL/2_nodes(0.07s)Isolation passes (
-count=5 -run TestMeshAcceptanceDurableok), basemainpasses cleanly — the flakiness arrives with this PR. The assertions check point-in-time invariants after convergence windows (deleted command log on the leader, local read during the outage, in-flight consume) and race the mesh's own reconcile/retraction/replication loops. Whether this is pure test timing or exposes a real production race (e.g. command-log recreation racing the delete, or a replica pull stalling on a lagging local copy), the suite as submitted cannot gate CI. Fix: poll each invariant (wait forhasStream(internalName(name))to clear, retry the outage read, keep the consume timeout as a last resort) and re-run the verification until-count=3holds repeatedly.@ -0,0 +362,4 @@resp := dispatchRequest(t, node.server, framePull, pullRequest{stream: name, consumer: outageConsumer})if !resp.hasRecord || resp.record.seq != 0 {t.Fatalf("pull on node %s with its leader down = %+v, want the first retained record", node.id, resp)🟡 SUGGESTION — the failure message is misleading for the pre-outage calls.
assertReplicaServesReadis also invoked while the leader is up (the loops at lines ~214 and ~349), yet every failure reads "pull on node ... with its leader down". When a pre-outage read fails the message asserts something false, which will misdirect debugging of exactly the flake above. Consider wording it neutrally (e.g. "pull on node %s = %+v, want the first retained record").@ -0,0 +63,4 @@b.Helper()ctx, cancel := context.WithTimeout(b.Context(), convergenceTimeout)b.Cleanup(func() { cancel() })🟡 SUGGESTION —
b.Cleanup+ a fresh 10s timeout context are registered perawaitAppliedcall, i.e. once per benchmark iteration (lines 143, 185). With b.N iterations that accumulates b.N cleanup closures and live timer contexts (only released when the benchmark ends), and the per-call context allocation lands inside the timed loop, polluting the allocs/op reading. Fix: create one timeout context per benchmark before the loop and pass it in (or checkctx.Done()in the spin loop), dropping the per-iteration Cleanup.@ -0,0 +131,4 @@func BenchmarkMeshReplicationThroughput(b *testing.B) {b.ReportAllocs()nodes, name := benchMesh(b, 2)🟡 SUGGESTION — mesh setup runs inside the timed region.
benchMesh(host startup + create + convergence waits) is called before the loop with the timer running in all four benchmarks (lines 111, 134, 159, 201);b.Elapsed()(verified:go doc testing.B.Elapsed— "matches the one measured by B.StartTimer, B.StopTimer") includes it, so ns/op and the reported records/s are polluted by setup. Measured at -benchtime=1x:BenchmarkMeshReplicationThroughput-3 1 44733791 ns/op 22.35 records/s— at b.N=1 the metric is almost entirely setup; at the default 1s benchtime (N≈1000) the ~100-300ms setup still skews records/s by 10-30%. Fix:b.StopTimer()beforebenchMesh/b.StartTimer()(orb.ResetTimer()) after setup, so the metric is the replication path alone.Review findings resolved
All six are addressed in
7685a96,684756a,7669c25. The blocking flake was not test timing: each failure had a production cause, and I reproduced every one before fixing it (four concurrent test processes was enough to hit them reliably).BLOCKING â the acceptance tests flake under load: fixed
Three separate production faults:
A shutting-down leader was reported as a missing consumer.
leaseCheckinserver.gomapped any failure to open the stream toErrConsumerNotFound, including thecontext canceledan in-flight request gets when the leader stops serving.replicaPulltreats that sentinel as an authoritative refusal, so it returned the error instead of serving from its own copy â the read during a leader outage that the plan promises â and a client consume loop treats it as terminal. Only a stream that is really gone is a missing consumer now; anything else isErrLeaderUnavailable, which the caller retries and the replica falls back from. (This was thepull on node ... = consumer not found: "readers"failure.)The durable interest trim dropped records a consumer had not read.
consumerStore.minLiveWatermarkused a zero watermark as its "no minimum yet" sentinel. Zero is a real watermark, so when the live consumers were visited with an unacked one first, a later consumer's higher watermark overwrote the minimum; Go's map iteration order decided whether that happened on any given pass. That is theconsume loop delivered 2 of 3 recordsfailure: the leader publishedTrim{base=3}while a consumer still had a record to read.A create/delete/first-load interleaving left a stream resident with no directory.
createStreamLocalinserts a placeholder, writes the directory, then mounts;deleteStreamLocalremoves the map entry and then the directory. A delete that swept in between left the command log mounted with nothing on disk âstoreHasLog=false,resident=trueâ so the leader kept a command log for a stream it had deleted. A per-name, refcounted lifecycle lock now serializes create, delete and first load.ensureCommandLogandensureReplicaalso re-check the other name they depend on (the user stream, the leader), which a per-name lock cannot order.SUGGESTIONS
Misleading failure message â
assertReplicaServesReadno longer claims the leader is down. The pre-outage read is a separate helper (warmReplicaRead) that retries a transient relay failure; the outage read itself is still asserted in one shot.Mesh setup inside the timed region â fixed for the two
b.Nbenchmarks (throughput, backfill) by stopping the timer around the mesh build. At-benchtime=50x: replication throughput 1.05 ms/op â 145 µs/op, 950 â 6917 records/s. This did not apply toBenchmarkMeshForwardedPublishorBenchmarkMeshPublishLatency: they useb.Loop(), whose first call resets the timer, so their setup was already outside the measurement (their numbers are unchanged, ~64 µs local vs ~138 µs relayed).Per-iteration context and cleanup â one timeout context per benchmark now, so the allocation is out of the measured loop: throughput allocs/op 1316 â 78.
Also fixed, found while verifying
TestReplicatedStreamIsEvictableBetweenBatchescalledresidentStream.idleEvictableafter releasing the server lock. Resident state in the tests is now read through the locked helpers or under the lock.Server.streamBaseandServer.streamMaxAgeread the mounted stream after releasing the lock, racing the mount that writes it. Both now read under the lock. (Pre-existing;-racecaught it under load.)Verification
One caveat for the record: three concurrent full
-racesuites exhaust this 4-CPU box (dial failures in unrelated tests, and a data race in the baseline). I confirmed the same failures onmainat the same parallelism, so they are environmental; every result above is from 1â2 concurrent suites.Ready for another review pass at
7669c25.