feat: quorum-durable publishes (plan.md Phase 2) #16
No reviewers
Labels
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
norobo/webstream!16
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/phase-2-quorum-durable-publishes"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Phase 2 of
plan.md: a publish is acked only once its record is durable on a quorum of the stream's holders, so an ack is a promise a later promotion can check instead of a fact about one node. Until now the ack landed after one local fsync of theAppendcommand, and no election could repair that — a promoted leader could mint sequence numbers below records clients already hold aPublishAck{Seq}for. The write path had to change first.The write quorum (promise P1)
quorum(R) = R/2 + 1over the stream's holders (every mesh node today, soR = N). The ack waits for the replica confirmations beyond this node's own durable append —needed = R/2distinct replicas, so 1/2/3/5 holders need 0/1/1/2. Neither number is configurable: a promotion later takes a read quorum of the same set and the two majorities must intersect.Server.ackrecords that replica's confirmation in the stream'squorumTrackeronce the command has been applied durably, andServer.publishwaits on it (Server.awaitQuorum) before replying.quorumTrackerholds one watermark per replica consumer, never a per-sequence tally: a replica applies and acks the log in order, so a confirmation of a later record covers every earlier one. That is what makes waiting for one sequence a comparison, and what lets the retry of an already-confirmed publish check the same condition — the state is bounded by the mesh, not by the number of publishes.quorumTimeoutfailsErrNoQuorum(new sentinel, new wire codecodeNoQuorum) instead of hanging. Nothing is undone: the record'sAppendcommand is durable on the leader, so the record may still be readable there and may still replicate. A retry that must not duplicate it carries the sameWithDeduplicationIDand reads the original sequence number back. A duplicate reply is itself an ack for that sequence, so it waits on the same quorum — a dedup hit is never granted on one copy.meshPeers.forwardOwnConn). A node serves one application stream's frames one at a time, so a publish waiting out its quorum on the shared per-peer connection would stall that node's own reconciliation and consumer registration — exactly the work that produces the confirmation it waits for. I hit that cycle while writing the tests: the shared connection meansnode1 -> node0publish relays andnode1's reconciliation withnode0are the same serialized stream, so the replica could never be asked for the confirmation.quorum(R)is over the configured set and this phase has no liveness rule, soquorum(R)of the configured members must be up for a stream to be writable. With two nodes, either one being down stops writes (durability and reads survive). This is the trade-off phase 4 states, arriving early, and phase 3's live view is what restores write availability. Recorded inplan.md,README.mdanddoc.go.Group commit, mandatory and automatic
syncSegmentLockedjoins the fsync already in flight on the record's segment and returns when that segment's durable end has passed the record — often by an fsync a later writer started, so a covered writer pays no fsync of its own.startSegmentSyncLockedruns the fsync with the journal lock released; a failed fsync fails every writer in the batch and rolls the segment back to its durable end, so no writer is ever told a record is durable after a failed fsync.end,synced,batch,pending) because the writer lock can no longer be held across the fsync. On-disk order must stay sequence order, so a roll cannot wait for the fsync in flight on the segment it moves past: it queues the segment for finalization, which happens at a batch boundary once no fsync and no writer still need it (pending). A per-append seek is gone too — the segment tracks its own end.WithFlushIntervalis deleted, as the plan says: once the ack waits for the covering fsync, its documented tradeoff (an early ack against a crash window) no longer exists, and an idle server pays exactly the one fsync it always has.BenchmarkPublishGroupCommitreports 0.5 fsyncs/op againstBenchmarkPublishDurable's 1, andBenchmarkQuorumPublishreports a publish on a one-node mesh at 61 µs against 182 µs on three nodes — the quorum round trip, which is the number the plan asks to be published. Both are appended tobench.logbymake bench-lognow (the Makefile's target gained them).rapid-tagged journal differential now pays one real fsync per append (it used to run in batched mode); it measured 16.5 s here, so it stays behind the tag andTESTING_PLAN.md/README.mdsay so without a fixed number.Two pre-existing data races, fixed here
Both were reproduced on a copy of the unmodified base tree (
git archive HEAD) before being touched, and both become easy to hit inmake test— which runs-race— because this phase adds longer-lived concurrent traffic:Server.listStreamNames,listStorePage,enforceStorageBudget,enforceDurableStorageBudget,evictColdDurableToBudgetanddeleteStreamLocalreads.storeoutside the lockClosenils it under (the existing comment claimed the field was only read under the lock —acquiredoes, those did not). They now go through onestoreRefaccessor that takes the lock the write uses.streamState.trimToFloorreads.log.baseafter unlocking, racing the ack path's trim. The base is now read under the lock.Tests
New:
quorum_test.go,quorum_property_test.go,store_group_commit_test.go.ErrNoQuorumwhen no replica can confirm: the unacked record stays readable on the leader, and a same-ID retry after the replica returns is the original sequence flaggedDuplicate, with one copy stored.replicaQuorumis a majority of the configured set for 1..5 nodes, and zero with no mesh.TestMeshLeaderServesLocally— whose set contains a hostless peer, so its publish is now unackable — asserts that documented outcome, in a synctest bubble so the deadline stays off the wall clock.Verification
go vet ./...andgo vet -tags rapid ./...— clean.golangci-lint runand--build-tags rapid— 0 issues.go test -race ./...— 22-25 s, plus-tags rapid31 s. 5 shuffled full runs and 6 x-count=20runs of the mesh/replication/quorum tests, all clean (that hammer is how the two races above were found).go test -run='^$' -fuzz='^FuzzDecodeCommand$' -fuzztime=15s— ok (no new parser in this change).gremlins unleash --dry-run— 1168 runnable, 91 not covered, 92.77 % mutator coverage (baseline 1098/93/92.19 %, previous 1137/92/92.51 %): one fewer uncovered mutant than the baseline and none of them in the new code; the one mutant this change added (dropSealingLocked's nil-file guard, unreachable by construction) was deleted rather than documented.coverage.logrecords it, and the earlier "rewind branches are 0-hit" known gap is retired — closing the segment file reaches the fsync failure, which two tests now do.Plan text
The implementation disagreed with
plan.mdin four places, so the plan was corrected rather than the code: the group-commit bullet now says what a batch actually promises (the writes before its syscall, not the ones landing during it — the latter is not something a real filesystem guarantees), the quorum is stated as being over the configured set with itsN = 2consequence, the relayed-publish connection is added as a decision, and theErrNoQuorumtest bullet records that a duplicate retry waits on the quorum too.Not done here (deliberately)
No promotion, no liveness, no holder sets, no batched pulls — those are phases 3, 4 and 10. Nothing is pushed to
main; this is for review.AI review of feat: quorum-durable publishes (plan.md Phase 2) — quorum-acked writes, mandatory group commit,
ErrNoQuorum, relayed-publish connections, and two pre-existing race fixes. Reviewed with the samber/cc-skills-golang skill set against the headfcb19f3.Verification performed (this session, on a worktree at the PR head)
go build ./...,go vet ./...,go vet -tags rapid ./...— clean.golangci-lint run—0 issues.go test -race ./...— ok in 23.8s (matches the PR's claim); targeted quorum/group-commit/mesh tests pass, including the synctest-bubble deadline tests.go.mod/go.sumare unchanged vs the base (8f4f6ad) — no new dependencies; the only govulncheck "affected" finding is GO-2026-6604, a Windows-onlyos.Rootjunction issue in the container's stdlib (go1.27.1 → fixed 1.27.2) reachable via pre-existingstore.listStreams/streamExistspaths — a toolchain matter, not this PR.Concurrency deep-dive (the group commit)
Traced
appendPublishLocked→syncSegmentLocked→startSegmentSyncLockedunder every interleaving I could construct (joiner whose record postdates the batch start; failed batch rolling back a joiner's record; roll of a segment with an fsync in flight; seal-vs-pending ordering;resetToLocked/removeawaiting in-flight batches). The wait/re-check loop and thebatch.syncedcapture are sound: a joiner whose record is not covered by the batch it waited on re-checks and starts its own batch, a failed fsync rolls back to the previous durable end and fails every writer in it (none is ever acked on a failed fsync), and sealing waits forpending == 0andbatch == nil. The write-lock is released beforeawaitQuorum, so the ack path that produces confirmations is never blocked by a waiting publish.quorumTracker's close-and-replace wake channel cannot lose a confirmation (check+channel under one lock) and cannot be double-closed (close undermu). Both race fixes (storeRef,trimToFloorreadingmemlog.baseunder the stream lock) are correct — all remaining directs.storereads are unders.lock, andmemlog.basewriters are stream-lock holders.Findings — 2 suggestions, none blocking
awaitQuorumrewrites client cancellations/deadlines asErrNoQuorum(replicate.go:360).See the inline comments for conditions and fixes. Everything else — wire mapping, dedup-retry semantics, benchmark methodology, tests, docs — verified consistent with the design. This is a genuinely solid PR; the group-commit state machine is the hardest code in this repo and it is carefully argued and well-tested.
@ -203,0 +357,4 @@return nil}err = fmt.Errorf("%w: stream %q record %d: confirmed by %d of %d replicas: %w",🟡 SUGGESTION —
awaitQuorumcollapses a client cancellation/deadline intoErrNoQuorum.quorumTracker.awaitreturnsctx.Err()oncase <-ctx.Done()(replicate.go:305-306, verified by reading the head file), and thisfmt.Errorf("%w: ...: %w", ErrNoQuorum, ..., err)wrap turns every await error intoErrNoQuorum. The server applies the caller's near deadline to the request context (README 'Write bounds' row: "the caller's deadline is used when it is near"), so a client whose deadline is shorter thanquorumTimeout(30s) gets wire codecodeNoQuorumand a Warn "a publish reached no quorum" log entry for what was actually a deadline — a misclassified failure in logs and in any client logic that keys offErrNoQuorum. The record's durability state is identical either way, so this is classification, not data loss. Fix: preserve the context error, e.g.if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { return err }before the wrap.Review follow-up (review #65, APPROVED): finding #2, which was listed in the review body but should also be inline here.
🟡 SUGGESTION — a quorum vote is granted without verifying the confirming node is durable (replicate.go:373,
Server.confirmQuorum). The confirmation is derived purely from the ack on the command log, andreplicateLoopsends that ack afterapplyRecordLocked— which skips the journal entirely when the replica has no store (applyRecordLocked,if s.journal == nil, stream_state.go:378-379). Nothing ties mesh membership (WithMeshPeers) toWithStore, so a mesh mixing durable and in-memory nodes can ack a publish as "durable on a quorum of holders" when a majority of the confirming copies are RAM-only — and phase 3's promotion read-quorum would then find the acked record missing. The single-node in-memory server is exempt (replicaQuorum() == 0with no mesh), and an all-durable mesh is fine today; the gap is that the assumption is unenforced and unstated. Fix: gate the vote/wait on durability (skipconfirmQuorum/awaitQuorumwhen the local store is absent) or state the all-holders-durable deployment requirement where mesh membership is configured. Not blocking: verified by readingconfirmQuorum(replicate.go:373-411) andapplyRecordLocked(stream_state.go:377-405) at headfcb19f3; no test exercises a mixed-durability mesh.Review resolved in
d4d55e2(pushed; head offeat/phase-2-quorum-dirable-publishes— correction, head offeat/phase-2-quorum-durable-publishes). All three points addressed, one of them with a different fix than suggested, with the reason below. Thanks for the trace of the group-commit state machine — that is the part I most wanted a second pair of eyes on.1.
awaitQuorumrewrites cancellations asErrNoQuorum(inline 1194) — resolved, classification kept deliberatelyFixed in the parts that were actually wrong, kept in the part where I think
ErrNoQuorumis the right code, and documented so the behaviour is a contract rather than an accident:Debug("a publish stopped waiting for a quorum"), not aWarnabout a quorum that never had the chance to form. The deadline case staysWarn."the wait ended with N of M replicas confirmed"for the context case versus"confirmed by N of M replicas"for the deadline.fmt.Errorfwith two%wverbs produces an error matching both, soerrors.Is(err, ErrNoQuorum)anderrors.Is(err, context.Canceled)(orcontext.DeadlineExceeded) are true today. There is a test for exactly that now.codeForhas no mapping for context errors, so it would land oncodeInternal: the wire would say "internal server error" andServer.errorResponsewould log a client's departure or a shutdown at ERROR level. A publish that stopped waiting is not a server fault, and its outcome is the same as a quorum failure: nothing was acked, andErrNoQuorum's contract ("may still be stored, may still be readable, may still replicate later") is the actionable statement for the caller — retry idempotently with the sameWithDeduplicationID. Inventing acodeCanceledfor a case no client can act on differently seemed worse than one code with the cause in the message and the inner error.ErrNoQuorum's doc, README's Write-quorum row,doc.goandplan.mdall now say this explicitly.rpcStream.writeRequestpushingSetWriteDeadline). On the server the context ends when the inbound application stream does: a relaying node's relay deadline expiring and dropping its connection, the client dropping it, orClosecancellingstopServing. Those are exactly the cases the Debug log now covers.2. Quorum votes granted without verifying the confirming node (replicate.go:373) — resolved, and you were right that this is the sharp edge
This was the more serious of the two and I had rationalised it as "phase 6 authentication": the vote's identity was the consumer name the ack carried (
rep-<peer ID>), and nothing bound that name to the node behind the connection. Any mesh peer could confirm a record for a node that never applied it, and a single node could reach a quorum by acking under several names — a quorum with no second copy behind it, which is exactly the promise P1 exists to make. Fixed:dispatchFromnow owns theframeForwarddispatch and passes the sending peer intopeerDispatch(it is the only place that knows who sent the frame; the branch moved out ofclientDispatch, which documented that a relay never reaches it).peerDispatchcalls the newcheckRelayedAckVote, which refuses a relayed ack on a reserved stream unless its consumer is the sending node's ownreplicaConsumerName(from). A client cannot reach this path at all (reserved names are rejected for every client-facing kind), and a client's ack on a user stream is untouched.TestRelayedAckVotesOnlyForTheSendingNoderelays an ack from node 1 naming node 0's consumer: refused, confirmation count stays at 1, and the comment records that the replica's own ack for the same position is the one that was accepted (that is what the test'swaitForRecordswaited for).3. 13 lines of doc comment on a 27-line helper — resolved
Fair. The new comments are trimmed to the invariants a reader cannot infer from the code: what a batch covers, why a segment keeps its file while a writer is pending, what the vote check does and does not promise. The rationale now lives on the code that depends on it (
confirmQuorum,syncSegmentLocked,awaitQuorum) rather than being restated on each helper. The pre-existing long comments elsewhere instore.gowere not touched — they describe the on-disk format and the rewind invariants, but if you want that pass too, say so and I will do it as its own commit.Verification on this head
go vet ./.../go vet -tags rapid ./...— clean;golangci-lint runand--build-tags rapid—0 issues.go test -race ./...24.9 s, plus three-shuffle=onruns (clean; that hammer is what found the two races the first commit fixed).go test -race -tags rapid ./...41.5 s.gremlins unleash --dry-run— 1171 runnable, 90 not covered, 92.86% (from 1168/91/92.77%): no new uncovered class, and coverage 88.6% (from 88.5%).coverage.loghas the line.bench.logunchanged: neither finding touches a benchmarked path, and the two benchmarks published by this PR (BenchmarkPublishGroupCommit,BenchmarkQuorumPublish) are unaffected by a name check on the relayed ack path and a log-level branch.