feat: quorum-durable publishes (plan.md Phase 2) #16

Merged
sam merged 2 commits from feat/phase-2-quorum-durable-publishes into main 2026-10-19 19:55:27 +00:00
Member

Phase 2 of plan.md: a publish is acked only once its record is durable on a quorum of the stream's holders, so an ack is a promise a later promotion can check instead of a fact about one node. Until now the ack landed after one local fsync of the Append command, and no election could repair that — a promoted leader could mint sequence numbers below records clients already hold a PublishAck{Seq} for. The write path had to change first.

The write quorum (promise P1)

  • quorum(R) = R/2 + 1 over the stream's holders (every mesh node today, so R = N). The ack waits for the replica confirmations beyond this node's own durable append — needed = R/2 distinct replicas, so 1/2/3/5 holders need 0/1/1/2. Neither number is configurable: a promotion later takes a read quorum of the same set and the two majorities must intersect.
  • The confirmation channel is the replica consumer's existing ack, exactly as the plan asks: Server.ack records that replica's confirmation in the stream's quorumTracker once the command has been applied durably, and Server.publish waits on it (Server.awaitQuorum) before replying. quorumTracker holds one watermark per replica consumer, never a per-sequence tally: a replica applies and acks the log in order, so a confirmation of a later record covers every earlier one. That is what makes waiting for one sequence a comparison, and what lets the retry of an already-confirmed publish check the same condition — the state is bounded by the mesh, not by the number of publishes.
  • A publish that cannot reach the quorum within quorumTimeout fails ErrNoQuorum (new sentinel, new wire code codeNoQuorum) instead of hanging. Nothing is undone: the record's Append command is durable on the leader, so the record may still be readable there and may still replicate. A retry that must not duplicate it carries the same WithDeduplicationID and reads the original sequence number back. A duplicate reply is itself an ack for that sequence, so it waits on the same quorum — a dedup hit is never granted on one copy.
  • Read paths are untouched: pulls stay local and never wait on a quorum.
  • A relayed publish now travels on a connection of its own (meshPeers.forwardOwnConn). A node serves one application stream's frames one at a time, so a publish waiting out its quorum on the shared per-peer connection would stall that node's own reconciliation and consumer registration — exactly the work that produces the confirmation it waits for. I hit that cycle while writing the tests: the shared connection means node1 -> node0 publish relays and node1's reconciliation with node0 are the same serialized stream, so the replica could never be asked for the confirmation.
  • quorum(R) is over the configured set and this phase has no liveness rule, so quorum(R) of the configured members must be up for a stream to be writable. With two nodes, either one being down stops writes (durability and reads survive). This is the trade-off phase 4 states, arriving early, and phase 3's live view is what restores write availability. Recorded in plan.md, README.md and doc.go.

Group commit, mandatory and automatic

  • The segment fsync is the group commit. syncSegmentLocked joins the fsync already in flight on the record's segment and returns when that segment's durable end has passed the record — often by an fsync a later writer started, so a covered writer pays no fsync of its own. startSegmentSyncLocked runs the fsync with the journal lock released; a failed fsync fails every writer in the batch and rolls the segment back to its durable end, so no writer is ever told a record is durable after a failed fsync.
  • Durability state moved onto the segment (end, synced, batch, pending) because the writer lock can no longer be held across the fsync. On-disk order must stay sequence order, so a roll cannot wait for the fsync in flight on the segment it moves past: it queues the segment for finalization, which happens at a batch boundary once no fsync and no writer still need it (pending). A per-append seek is gone too — the segment tracks its own end.
  • WithFlushInterval is deleted, as the plan says: once the ack waits for the covering fsync, its documented tradeoff (an early ack against a crash window) no longer exists, and an idle server pays exactly the one fsync it always has.
  • Measured on this machine (arm64, 3 CPUs): BenchmarkPublishGroupCommit reports 0.5 fsyncs/op against BenchmarkPublishDurable's 1, and BenchmarkQuorumPublish reports a publish on a one-node mesh at 61 µs against 182 µs on three nodes — the quorum round trip, which is the number the plan asks to be published. Both are appended to bench.log by make bench-log now (the Makefile's target gained them).
  • The rapid-tagged journal differential now pays one real fsync per append (it used to run in batched mode); it measured 16.5 s here, so it stays behind the tag and TESTING_PLAN.md/README.md say so without a fixed number.

Two pre-existing data races, fixed here

Both were reproduced on a copy of the unmodified base tree (git archive HEAD) before being touched, and both become easy to hit in make test — which runs -race — because this phase adds longer-lived concurrent traffic:

  1. Server.listStreamNames, listStorePage, enforceStorageBudget, enforceDurableStorageBudget, evictColdDurableToBudget and deleteStreamLocal read s.store outside the lock Close nils it under (the existing comment claimed the field was only read under the lock — acquire does, those did not). They now go through one storeRef accessor that takes the lock the write uses.
  2. streamState.trimToFloor read s.log.base after unlocking, racing the ack path's trim. The base is now read under the lock.

Tests

New: quorum_test.go, quorum_property_test.go, store_group_commit_test.go.

  • An acked sequence is present on a surviving holder after the leader's node is closed uncleanly (no decommission, no flush) — asserted directly, since phase 3's promotion rests on it.
  • ErrNoQuorum when no replica can confirm: the unacked record stays readable on the leader, and a same-ID retry after the replica returns is the original sequence flagged Duplicate, with one copy stored.
  • One live replica is not a quorum of three.
  • The confirmation count is per replica: a repeat and a later record's confirmation each count once — a rapid shadow model, plus a unit test that a missing replica is not a confirmation of record 0 (the map's zero value is not a vote), and that the wait returns on a confirmation, on expiry, or immediately for a quorum of zero.
  • replicaQuorum is a majority of the configured set for 1..5 nodes, and zero with no mesh.
  • Group commit: an idle publish costs exactly one fsync; one fsync covers every writer in a batch (none of them pays one); a failed batch fsync fails every writer rather than acking one and leaves the durable end unmoved; and concurrent durable publishes keep on-disk order equal to sequence order (that test caught a real bug in my first draft: releasing the lock before the write let a later sequence number land first).
  • Mesh/replication tests that took one node of two down and kept writing move to three nodes (the quorum of three outlives one node), and TestMeshLeaderServesLocally — whose set contains a hostless peer, so its publish is now unackable — asserts that documented outcome, in a synctest bubble so the deadline stays off the wall clock.

Verification

  • go vet ./... and go vet -tags rapid ./... — clean.
  • golangci-lint run and --build-tags rapid — 0 issues.
  • go test -race ./... — 22-25 s, plus -tags rapid 31 s. 5 shuffled full runs and 6 x -count=20 runs of the mesh/replication/quorum tests, all clean (that hammer is how the two races above were found).
  • go test -run='^$' -fuzz='^FuzzDecodeCommand$' -fuzztime=15s — ok (no new parser in this change).
  • gremlins unleash --dry-run — 1168 runnable, 91 not covered, 92.77 % mutator coverage (baseline 1098/93/92.19 %, previous 1137/92/92.51 %): one fewer uncovered mutant than the baseline and none of them in the new code; the one mutant this change added (dropSealingLocked's nil-file guard, unreachable by construction) was deleted rather than documented. coverage.log records it, and the earlier "rewind branches are 0-hit" known gap is retired — closing the segment file reaches the fsync failure, which two tests now do.
  • Coverage 88.5 % statements (was 88.0 %).

Plan text

The implementation disagreed with plan.md in four places, so the plan was corrected rather than the code: the group-commit bullet now says what a batch actually promises (the writes before its syscall, not the ones landing during it — the latter is not something a real filesystem guarantees), the quorum is stated as being over the configured set with its N = 2 consequence, the relayed-publish connection is added as a decision, and the ErrNoQuorum test bullet records that a duplicate retry waits on the quorum too.

Not done here (deliberately)

No promotion, no liveness, no holder sets, no batched pulls — those are phases 3, 4 and 10. Nothing is pushed to main; this is for review.

Phase 2 of `plan.md`: a publish is acked only once its record is durable on a **quorum** of the stream's holders, so an ack is a promise a later promotion can check instead of a fact about one node. Until now the ack landed after one local fsync of the `Append` command, and no election could repair that — a promoted leader could mint sequence numbers below records clients already hold a `PublishAck{Seq}` for. The write path had to change first. ## The write quorum (promise P1) - `quorum(R) = R/2 + 1` over the stream's holders (every mesh node today, so `R = N`). The ack waits for the replica confirmations beyond this node's own durable append — `needed = R/2` distinct replicas, so 1/2/3/5 holders need 0/1/1/2. Neither number is configurable: a promotion later takes a read quorum of the same set and the two majorities must intersect. - The confirmation channel is the replica consumer's existing ack, exactly as the plan asks: `Server.ack` records that replica's confirmation in the stream's `quorumTracker` once the command has been applied durably, and `Server.publish` waits on it (`Server.awaitQuorum`) before replying. `quorumTracker` holds **one watermark per replica consumer**, never a per-sequence tally: a replica applies and acks the log in order, so a confirmation of a later record covers every earlier one. That is what makes waiting for one sequence a comparison, and what lets the retry of an already-confirmed publish check the same condition — the state is bounded by the mesh, not by the number of publishes. - A publish that cannot reach the quorum within `quorumTimeout` fails `ErrNoQuorum` (new sentinel, new wire code `codeNoQuorum`) instead of hanging. Nothing is undone: the record's `Append` command is durable on the leader, so the record may still be readable there and may still replicate. A retry that must not duplicate it carries the same `WithDeduplicationID` and reads the original sequence number back. A duplicate reply is itself an ack for that sequence, so it waits on the same quorum — a dedup hit is never granted on one copy. - Read paths are untouched: pulls stay local and never wait on a quorum. - A relayed publish now travels on a connection of its own (`meshPeers.forwardOwnConn`). A node serves one application stream's frames one at a time, so a publish waiting out its quorum on the shared per-peer connection would stall that node's own reconciliation and consumer registration — exactly the work that produces the confirmation it waits for. I hit that cycle while writing the tests: the shared connection means `node1 -> node0` publish relays and `node1`'s reconciliation with `node0` are the same serialized stream, so the replica could never be asked for the confirmation. - **`quorum(R)` is over the *configured* set and this phase has no liveness rule**, so `quorum(R)` of the configured members must be up for a stream to be writable. With two nodes, either one being down stops writes (durability and reads survive). This is the trade-off phase 4 states, arriving early, and phase 3's live view is what restores write availability. Recorded in `plan.md`, `README.md` and `doc.go`. ## Group commit, mandatory and automatic - The segment fsync is the group commit. `syncSegmentLocked` joins the fsync already in flight on the record's segment and returns when that segment's durable end has passed the record — often by an fsync a *later* writer started, so a covered writer pays no fsync of its own. `startSegmentSyncLocked` runs the fsync with the journal lock released; a failed fsync fails **every** writer in the batch and rolls the segment back to its durable end, so no writer is ever told a record is durable after a failed fsync. - Durability state moved onto the segment (`end`, `synced`, `batch`, `pending`) because the writer lock can no longer be held across the fsync. On-disk order must stay sequence order, so a roll cannot wait for the fsync in flight on the segment it moves past: it queues the segment for finalization, which happens at a batch boundary once no fsync and no writer still need it (`pending`). A per-append seek is gone too — the segment tracks its own end. - `WithFlushInterval` is deleted, as the plan says: once the ack waits for the covering fsync, its documented tradeoff (an early ack against a crash window) no longer exists, and an idle server pays exactly the one fsync it always has. - Measured on this machine (arm64, 3 CPUs): `BenchmarkPublishGroupCommit` reports **0.5 fsyncs/op** against `BenchmarkPublishDurable`'s 1, and `BenchmarkQuorumPublish` reports a publish on a one-node mesh at **61 µs** against **182 µs** on three nodes — the quorum round trip, which is the number the plan asks to be published. Both are appended to `bench.log` by `make bench-log` now (the Makefile's target gained them). - The `rapid`-tagged journal differential now pays one real fsync per append (it used to run in batched mode); it measured **16.5 s** here, so it stays behind the tag and `TESTING_PLAN.md`/`README.md` say so without a fixed number. ## Two pre-existing data races, fixed here Both were reproduced on a copy of the unmodified base tree (`git archive HEAD`) before being touched, and both become easy to hit in `make test` — which runs `-race` — because this phase adds longer-lived concurrent traffic: 1. `Server.listStreamNames`, `listStorePage`, `enforceStorageBudget`, `enforceDurableStorageBudget`, `evictColdDurableToBudget` and `deleteStreamLocal` read `s.store` outside the lock `Close` nils it under (the existing comment claimed the field was only read under the lock — `acquire` does, those did not). They now go through one `storeRef` accessor that takes the lock the write uses. 2. `streamState.trimToFloor` read `s.log.base` *after* unlocking, racing the ack path's trim. The base is now read under the lock. ## Tests New: `quorum_test.go`, `quorum_property_test.go`, `store_group_commit_test.go`. - An acked sequence is present on a surviving holder after the leader's node is closed **uncleanly** (no decommission, no flush) — asserted directly, since phase 3's promotion rests on it. - `ErrNoQuorum` when no replica can confirm: the unacked record stays readable on the leader, and a same-ID retry after the replica returns is the original sequence flagged `Duplicate`, with one copy stored. - One live replica is not a quorum of three. - The confirmation count is per replica: a repeat and a later record's confirmation each count once — a rapid shadow model, plus a unit test that a missing replica is not a confirmation of record 0 (the map's zero value is not a vote), and that the wait returns on a confirmation, on expiry, or immediately for a quorum of zero. - `replicaQuorum` is a majority of the configured set for 1..5 nodes, and zero with no mesh. - Group commit: an idle publish costs exactly one fsync; one fsync covers every writer in a batch (none of them pays one); a failed batch fsync fails every writer rather than acking one and leaves the durable end unmoved; and concurrent durable publishes keep on-disk order equal to sequence order (that test caught a real bug in my first draft: releasing the lock before the write let a later sequence number land first). - Mesh/replication tests that took one node of two down and kept writing move to three nodes (the quorum of three outlives one node), and `TestMeshLeaderServesLocally` — whose set contains a hostless peer, so its publish is now unackable — asserts that documented outcome, in a synctest bubble so the deadline stays off the wall clock. ## Verification - `go vet ./...` and `go vet -tags rapid ./...` — clean. - `golangci-lint run` and `--build-tags rapid` — 0 issues. - `go test -race ./...` — 22-25 s, plus `-tags rapid` 31 s. 5 shuffled full runs and 6 x `-count=20` runs of the mesh/replication/quorum tests, all clean (that hammer is how the two races above were found). - `go test -run='^$' -fuzz='^FuzzDecodeCommand$' -fuzztime=15s` — ok (no new parser in this change). - `gremlins unleash --dry-run` — 1168 runnable, 91 not covered, 92.77 % mutator coverage (baseline 1098/93/92.19 %, previous 1137/92/92.51 %): **one fewer** uncovered mutant than the baseline and none of them in the new code; the one mutant this change added (`dropSealingLocked`'s nil-file guard, unreachable by construction) was deleted rather than documented. `coverage.log` records it, and the earlier "rewind branches are 0-hit" known gap is retired — closing the segment file reaches the fsync failure, which two tests now do. - Coverage 88.5 % statements (was 88.0 %). ## Plan text The implementation disagreed with `plan.md` in four places, so the plan was corrected rather than the code: the group-commit bullet now says what a batch actually promises (the writes before its syscall, not the ones landing during it — the latter is not something a real filesystem guarantees), the quorum is stated as being over the configured set with its `N = 2` consequence, the relayed-publish connection is added as a decision, and the `ErrNoQuorum` test bullet records that a duplicate retry waits on the quorum too. ## Not done here (deliberately) No promotion, no liveness, no holder sets, no batched pulls — those are phases 3, 4 and 10. Nothing is pushed to `main`; this is for review.
feat: quorum-durable publishes (plan.md Phase 2)
All checks were successful
code-review / Agent review (pull_request) Successful in 9s
fcb19f3263
Phase 2 of plan.md: a publish is acked only once its record is durable on a
quorum of the stream's holders, so an ack is a promise a later promotion can
check instead of a fact about one node. The write path had to change first,
because no election can repair "the ack landed on the leader alone".

The write quorum
- `quorum(R) = R/2 + 1` over the stream's holders (every mesh node today, so
  `R = N`), and the ack waits for the replica confirmations beyond this node's
  own durable append: `needed = R/2` distinct replicas, so 1/2/3/5 holders
  need 0/1/1/2.
- The confirmation is the replica consumer's existing ack: `Server.ack` on a
  command log records that replica's watermark in the stream's `quorumTracker`
  after the command's durable apply, and `Server.publish` waits on it
  (`awaitQuorum`) before replying. `quorumTracker` holds one watermark per
  replica consumer, never a per-sequence tally: a replica applies and acks the
  log in order, so a confirmation of a later record covers every earlier one,
  which is what makes waiting for one sequence a comparison and lets the retry
  of an already-confirmed publish check the same condition.
- A publish that cannot reach the quorum within `quorumTimeout` fails
  `ErrNoQuorum` (new sentinel, new `codeNoQuorum`, doc'd as "may still be
  stored, may still be readable, may still replicate later"). Nothing is
  undone: the record's command is durable here, so a retry carrying the same
  `WithDeduplicationID` reads the original sequence number back. A duplicate is
  itself an ack for that sequence, so it waits on the same quorum — a dedup hit
  is never granted on one copy.
- The quorum is over the *configured* set, and this phase has no liveness rule:
  `quorum(R)` of the configured members must be up for a stream to be writable.
  A two-node deployment stops accepting writes when either node is down
  (durability and reads survive). Phase 3's live view is what restores write
  availability, which is why this phase can be this small.
- A relayed publish travels on a connection of its own (`forwardOwnConn`). A
  node serves one application stream's requests one at a time, so a relayed
  publish waiting out its quorum on the shared per-peer connection would stall
  that node's own reconciliation and replication registration — exactly the
  work that produces the confirmation it waits for, a cycle that only the
  publish's deadline could break.

Group commit, mandatory and automatic
- The segment fsync is now the group commit: `syncSegmentLocked` joins the
  fsync in flight on the record's segment and returns when that segment's
  durable end has passed the record — often by an fsync a *later* writer
  started, so a covered writer pays no fsync of its own. `startSegmentSyncLocked`
  runs the fsync with the journal lock released, and a failed fsync fails every
  writer in the batch and rolls the segment back to its durable end.
- Durability state moved onto the segment (`end`, `synced`, `batch`,
  `pending`), because the journal writer lock may no longer be held across the
  fsync: on-disk order must stay sequence order, so a roll cannot wait for the
  fsync in flight on the segment it moves past. A rolled-past segment is
  finalized (header count and timestamp, file closed) at a batch boundary once
  no fsync and no writer still need it. This is also why the per-append seek is
  gone: the segment tracks its own end.
- `WithFlushInterval` is deleted. Once the ack waits for the covering fsync,
  its tradeoff (an early ack against a crash window) no longer exists, and an
  idle server pays exactly the one fsync it always has.
- Measured: `BenchmarkPublishGroupCommit` reports `fsyncs/op` (0.5 on this
  machine against `BenchmarkPublishDurable`'s 1), and `BenchmarkQuorumPublish`
  reports a publish on a one-node mesh at 61µs against 182µs on three nodes —
  the quorum round trip. `make bench-log` gains both.

Two pre-existing data races, fixed here because this phase's longer-lived
concurrent traffic makes them easy to hit in `make test` (which runs -race).
Both were reproduced on the unmodified base tree first:
- `Server.listStreamNames`, `listStorePage`, `enforceStorageBudget`,
  `enforceDurableStorageBudget`, `evictColdDurableToBudget` and
  `deleteStreamLocal` read `s.store` outside the lock `Close` nils it under.
  They now go through `storeRef`, which takes the lock the write uses.
- `streamState.trimToFloor` read `s.log.base` after unlocking, racing the ack
  path's trim; the base is now read under the lock.

Tests: an acked record is present on a surviving holder after the leader's node
is closed uncleanly; a publish with no reachable replica fails `ErrNoQuorum`
and the unacked record stays readable on the leader, then a same-ID retry after
the replica returns is the original sequence flagged duplicate, with one copy
stored; one live replica is not a quorum of three; the confirmation count is
per replica (repeats and a later record's confirmation both counted once) with
a rapid shadow model, and the wait returns on a confirmation or on bubble-time
expiry; `replicaQuorum` is a majority of the configured set for 1..5 nodes;
group commit pays one fsync for an idle publish and covers every writer in a
batch; a failed fsync fails every writer rather than acking one, and rolls the
batch back; concurrent durable publishes keep on-disk order equal to sequence
order. Mesh and replication tests that took one node of two down and kept
writing move to three nodes (the cases that depend on a node being down), and
`TestMeshLeaderServesLocally` — whose set contains a hostless peer, so its
publish is now unackable — asserts that documented outcome, inside a bubble so
the deadline stays off the wall clock.

Verification: `go vet ./...` and `go vet -tags rapid ./...`,
`golangci-lint run` and `--build-tags rapid` (0 issues), `go test -race ./...`
(~22s, 5 shuffled runs and 6 x count=20 runs of the mesh/replication/quorum
tests clean), `go test -race -tags rapid ./...` (31s), a 15s
`FuzzDecodeCommand` run, `gremlins unleash --dry-run` (1168 runnable, 91 not
covered, 92.77% — one fewer uncovered than the baseline, none in the new code),
coverage 88.5% (was 88.0%), and the machine-stable benchmarks plus the two new
ones appended to bench.log.
pi-reviewer left a comment

AI review of feat: quorum-durable publishes (plan.md Phase 2) — quorum-acked writes, mandatory group commit, ErrNoQuorum, relayed-publish connections, and two pre-existing race fixes. Reviewed with the samber/cc-skills-golang skill set against the head fcb19f3.

Verification performed (this session, on a worktree at the PR head)

  • go build ./..., go vet ./..., go vet -tags rapid ./... — clean.
  • golangci-lint run — 0 issues.
  • go test -race ./... — ok in 23.8s (matches the PR's claim); targeted quorum/group-commit/mesh tests pass, including the synctest-bubble deadline tests.
  • go.mod/go.sum are unchanged vs the base (8f4f6ad) — no new dependencies; the only govulncheck "affected" finding is GO-2026-6604, a Windows-only os.Root junction issue in the container's stdlib (go1.27.1 → fixed 1.27.2) reachable via pre-existing store.listStreams/streamExists paths — a toolchain matter, not this PR.

Concurrency deep-dive (the group commit)

Traced appendPublishLocked → syncSegmentLocked → startSegmentSyncLocked under every interleaving I could construct (joiner whose record postdates the batch start; failed batch rolling back a joiner's record; roll of a segment with an fsync in flight; seal-vs-pending ordering; resetToLocked/remove awaiting in-flight batches). The wait/re-check loop and the batch.synced capture are sound: a joiner whose record is not covered by the batch it waited on re-checks and starts its own batch, a failed fsync rolls back to the previous durable end and fails every writer in it (none is ever acked on a failed fsync), and sealing waits for pending == 0 and batch == nil. The write-lock is released before awaitQuorum, so the ack path that produces confirmations is never blocked by a waiting publish. quorumTracker's close-and-replace wake channel cannot lose a confirmation (check+channel under one lock) and cannot be double-closed (close under mu). Both race fixes (storeRef, trimToFloor reading memlog.base under the stream lock) are correct — all remaining direct s.store reads are under s.lock, and memlog.base writers are stream-lock holders.

Findings — 2 suggestions, none blocking

  1. 🟡 awaitQuorum rewrites client cancellations/deadlines as ErrNoQuorum (replicate.go:360).
  2. 🟡 Quorum votes are granted without verifying the confirming node is durable (replicate.go:373).

See the inline comments for conditions and fixes. Everything else — wire mapping, dedup-retry semantics, benchmark methodology, tests, docs — verified consistent with the design. This is a genuinely solid PR; the group-commit state machine is the hardest code in this repo and it is carefully argued and well-tested.

AI review of **feat: quorum-durable publishes (plan.md Phase 2)** — quorum-acked writes, mandatory group commit, `ErrNoQuorum`, relayed-publish connections, and two pre-existing race fixes. Reviewed with the samber/cc-skills-golang skill set against the head `fcb19f3`. ## Verification performed (this session, on a worktree at the PR head) - `go build ./...`, `go vet ./...`, `go vet -tags rapid ./...` — clean. - `golangci-lint run` — `0 issues.` - `go test -race ./...` — ok in 23.8s (matches the PR's claim); targeted quorum/group-commit/mesh tests pass, including the synctest-bubble deadline tests. - `go.mod`/`go.sum` are unchanged vs the base (`8f4f6ad`) — no new dependencies; the only govulncheck "affected" finding is GO-2026-6604, a **Windows-only** `os.Root` junction issue in the container's stdlib (go1.27.1 → fixed 1.27.2) reachable via pre-existing `store.listStreams`/`streamExists` paths — a toolchain matter, not this PR. ## Concurrency deep-dive (the group commit) Traced `appendPublishLocked` → `syncSegmentLocked` → `startSegmentSyncLocked` under every interleaving I could construct (joiner whose record postdates the batch start; failed batch rolling back a joiner's record; roll of a segment with an fsync in flight; seal-vs-pending ordering; `resetToLocked`/`remove` awaiting in-flight batches). The wait/re-check loop and the `batch.synced` capture are sound: a joiner whose record is not covered by the batch it waited on re-checks and starts its own batch, a failed fsync rolls back to the previous durable end and fails every writer in it (none is ever acked on a failed fsync), and sealing waits for `pending == 0` and `batch == nil`. The write-lock is released before `awaitQuorum`, so the ack path that produces confirmations is never blocked by a waiting publish. `quorumTracker`'s close-and-replace wake channel cannot lose a confirmation (check+channel under one lock) and cannot be double-closed (close under `mu`). Both race fixes (`storeRef`, `trimToFloor` reading `memlog.base` under the stream lock) are correct — all remaining direct `s.store` reads are under `s.lock`, and `memlog.base` writers are stream-lock holders. ## Findings — 2 suggestions, none blocking 1. 🟡 `awaitQuorum` rewrites client cancellations/deadlines as `ErrNoQuorum` (replicate.go:360). 2. 🟡 Quorum votes are granted without verifying the confirming node is durable (replicate.go:373). See the inline comments for conditions and fixes. Everything else — wire mapping, dedup-retry semantics, benchmark methodology, tests, docs — verified consistent with the design. This is a genuinely solid PR; the group-commit state machine is the hardest code in this repo and it is carefully argued and well-tested.
@ -203,0 +357,4 @@
return nil
}
err = fmt.Errorf("%w: stream %q record %d: confirmed by %d of %d replicas: %w",
Member

🟡 SUGGESTION — awaitQuorum collapses a client cancellation/deadline into ErrNoQuorum. quorumTracker.await returns ctx.Err() on case <-ctx.Done() (replicate.go:305-306, verified by reading the head file), and this fmt.Errorf("%w: ...: %w", ErrNoQuorum, ..., err) wrap turns every await error into ErrNoQuorum. The server applies the caller's near deadline to the request context (README 'Write bounds' row: "the caller's deadline is used when it is near"), so a client whose deadline is shorter than quorumTimeout (30s) gets wire code codeNoQuorum and a Warn "a publish reached no quorum" log entry for what was actually a deadline — a misclassified failure in logs and in any client logic that keys off ErrNoQuorum. The record's durability state is identical either way, so this is classification, not data loss. Fix: preserve the context error, e.g. if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { return err } before the wrap.

🟡 SUGGESTION — `awaitQuorum` collapses a client cancellation/deadline into `ErrNoQuorum`. `quorumTracker.await` returns `ctx.Err()` on `case <-ctx.Done()` (replicate.go:305-306, verified by reading the head file), and this `fmt.Errorf("%w: ...: %w", ErrNoQuorum, ..., err)` wrap turns **every** await error into `ErrNoQuorum`. The server applies the caller's near deadline to the request context (README 'Write bounds' row: "the caller's deadline is used when it is near"), so a client whose deadline is shorter than `quorumTimeout` (30s) gets wire code `codeNoQuorum` and a Warn "a publish reached no quorum" log entry for what was actually a deadline — a misclassified failure in logs and in any client logic that keys off `ErrNoQuorum`. The record's durability state is identical either way, so this is classification, not data loss. Fix: preserve the context error, e.g. `if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) { return err }` before the wrap.
Member

Review follow-up (review #65, APPROVED): finding #2, which was listed in the review body but should also be inline here.

🟡 SUGGESTION — a quorum vote is granted without verifying the confirming node is durable (replicate.go:373, Server.confirmQuorum). The confirmation is derived purely from the ack on the command log, and replicateLoop sends that ack after applyRecordLocked — which skips the journal entirely when the replica has no store (applyRecordLocked, if s.journal == nil, stream_state.go:378-379). Nothing ties mesh membership (WithMeshPeers) to WithStore, so a mesh mixing durable and in-memory nodes can ack a publish as "durable on a quorum of holders" when a majority of the confirming copies are RAM-only — and phase 3's promotion read-quorum would then find the acked record missing. The single-node in-memory server is exempt (replicaQuorum() == 0 with no mesh), and an all-durable mesh is fine today; the gap is that the assumption is unenforced and unstated. Fix: gate the vote/wait on durability (skip confirmQuorum/awaitQuorum when the local store is absent) or state the all-holders-durable deployment requirement where mesh membership is configured. Not blocking: verified by reading confirmQuorum (replicate.go:373-411) and applyRecordLocked (stream_state.go:377-405) at head fcb19f3; no test exercises a mixed-durability mesh.

Review follow-up (review #65, APPROVED): finding #2, which was listed in the review body but should also be inline here. 🟡 SUGGESTION — a quorum vote is granted without verifying the confirming node is durable (replicate.go:373, `Server.confirmQuorum`). The confirmation is derived purely from the ack on the command log, and `replicateLoop` sends that ack after `applyRecordLocked` — which skips the journal entirely when the replica has no store (`applyRecordLocked`, `if s.journal == nil`, stream_state.go:378-379). Nothing ties mesh membership (`WithMeshPeers`) to `WithStore`, so a mesh mixing durable and in-memory nodes can ack a publish as "durable on a quorum of holders" when a majority of the confirming copies are RAM-only — and phase 3's promotion read-quorum would then find the acked record missing. The single-node in-memory server is exempt (`replicaQuorum() == 0` with no mesh), and an all-durable mesh is fine today; the gap is that the assumption is unenforced and unstated. Fix: gate the vote/wait on durability (skip `confirmQuorum`/`awaitQuorum` when the local store is absent) or state the all-holders-durable deployment requirement where mesh membership is configured. Not blocking: verified by reading `confirmQuorum` (replicate.go:373-411) and `applyRecordLocked` (stream_state.go:377-405) at head `fcb19f3`; no test exercises a mixed-durability mesh.
Resolves the review on PR #16 (pi-reviewer, review id 65, APPROVED with two
non-blocking suggestions) and the style finding on its own doc comments.

1. A quorum vote is now attributable (server.go, replicate.go). An ack on a
   stream's command log is a vote — the leader counts one per replica and acks
   the client once a quorum has the record — and the vote's identity was the
   consumer name the ack carried, which nothing bound to the node behind the
   connection. Any mesh peer could therefore confirm a record for a node that
   never applied it, and one node could reach a quorum by claiming to be several
   — a quorum holding no second copy. `peerDispatch` now takes the sending peer
   (frameForward is dispatched from `dispatchFrom`, which is the only place that
   knows it, instead of from `clientDispatch`) and refuses a relayed ack on a
   reserved stream whose consumer is not the sending node's own
   `rep-<peer ID>`, via `checkRelayedAckVote`. That is attribution, not truth: a
   peer still vouches for its own disk, and authenticating peers is plan.md
   phase 6. `confirmQuorum` records the precondition and the boundary.

2. An abandoned wait is classified, not renamed (replicate.go, protocol.go). The
   finding was that `awaitQuorum` turns every await error into `ErrNoQuorum`, so
   a wait cut short reads as a quorum failure in the logs and in
   `errors.Is(err, ErrNoQuorum)`. It keeps reporting `ErrNoQuorum` — nothing was
   acked either way, and that sentinel's contract ("may still be stored, may still
   be readable, may still replicate later") covers both, while the alternative
   would report a caller's departure or a shutdown as the server's internal error
   — but it now logs at debug instead of warning about a quorum that never had the
   chance to form, and wraps context.Canceled/context.DeadlineExceeded as its
   inner error, so a Go caller can tell the two apart. `ErrNoQuorum`'s doc,
   README's Write-quorum row, doc.go and plan.md say so.

3. Comment proportionality (server.go, store.go, replicate.go). The review's
   style finding was a 13-line doc comment on a 27-line helper; the new comments
   are trimmed to the invariants they carry (what a batch covers, why a segment
   keeps its file while a writer is pending, what the vote's identity check does
   and does not promise), and the rationale that a caller cannot infer stays on
   the code that depends on it.

Tests: TestRelayedAckVotesOnlyForTheSendingNode (a replica's forged vote for the
leader is refused and the confirmation count stays 1, while the replica's own ack
is the one that was accepted);
TestAbandonedQuorumWaitIsNoQuorumWithItsCause (an abandoned wait matches both
ErrNoQuorum and context.Canceled, and carries codeNoQuorum);
TestAwaitQuorumReturnsWhenConfirmed became TestAwaitQuorumReportsWhyItStopped
with the cancelled-caller case.

Verification: `go vet` and `golangci-lint run` with and without `-tags rapid`
(0 issues), `go test -race ./...` 24.9s plus three shuffled runs (no races),
`go test -race -tags rapid ./...` 41.5s, gremlins 1171 runnable / 90 not covered
/ 92.86% (from 1168/91/92.77%), coverage 88.6% (from 88.5%), coverage.log
updated. bench.log is unchanged: neither finding touches a benchmarked path.
Author
Member

Review resolved in d4d55e2 (pushed; head of feat/phase-2-quorum-dirable-publishes — correction, head of feat/phase-2-quorum-durable-publishes). All three points addressed, one of them with a different fix than suggested, with the reason below. Thanks for the trace of the group-commit state machine — that is the part I most wanted a second pair of eyes on.

1. awaitQuorum rewrites cancellations as ErrNoQuorum (inline 1194) — resolved, classification kept deliberately

Fixed in the parts that were actually wrong, kept in the part where I think ErrNoQuorum is the right code, and documented so the behaviour is a contract rather than an accident:

  • The log level was wrong: a wait the context ended is now Debug ("a publish stopped waiting for a quorum"), not a Warn about a quorum that never had the chance to form. The deadline case stays Warn.
  • The error text now says which one it was: "the wait ended with N of M replicas confirmed" for the context case versus "confirmed by N of M replicas" for the deadline.
  • The inner error is preserved, and I want to flag that it already was: fmt.Errorf with two %w verbs produces an error matching both, so errors.Is(err, ErrNoQuorum) and errors.Is(err, context.Canceled) (or context.DeadlineExceeded) are true today. There is a test for exactly that now.
  • I did not switch to returning the context error unwrapped. codeFor has no mapping for context errors, so it would land on codeInternal: the wire would say "internal server error" and Server.errorResponse would log a client's departure or a shutdown at ERROR level. A publish that stopped waiting is not a server fault, and its outcome is the same as a quorum failure: nothing was acked, and ErrNoQuorum's contract ("may still be stored, may still be readable, may still replicate later") is the actionable statement for the caller — retry idempotently with the same WithDeduplicationID. Inventing a codeCanceled for a case no client can act on differently seemed worse than one code with the cause in the message and the inner error. ErrNoQuorum's doc, README's Write-quorum row, doc.go and plan.md all now say this explicitly.
  • One correction to the premise, which does not change the outcome but is worth recording: the server-side handler runs under the stream context, not the request context, so a client's near deadline is not applied to it — the README "Write bounds" row you cited is about the client's own frame write (rpcStream.writeRequest pushing SetWriteDeadline). On the server the context ends when the inbound application stream does: a relaying node's relay deadline expiring and dropping its connection, the client dropping it, or Close cancelling stopServing. Those are exactly the cases the Debug log now covers.

2. Quorum votes granted without verifying the confirming node (replicate.go:373) — resolved, and you were right that this is the sharp edge

This was the more serious of the two and I had rationalised it as "phase 6 authentication": the vote's identity was the consumer name the ack carried (rep-<peer ID>), and nothing bound that name to the node behind the connection. Any mesh peer could confirm a record for a node that never applied it, and a single node could reach a quorum by acking under several names — a quorum with no second copy behind it, which is exactly the promise P1 exists to make. Fixed:

  • dispatchFrom now owns the frameForward dispatch and passes the sending peer into peerDispatch (it is the only place that knows who sent the frame; the branch moved out of clientDispatch, which documented that a relay never reaches it).
  • peerDispatch calls the new checkRelayedAckVote, which refuses a relayed ack on a reserved stream unless its consumer is the sending node's own replicaConsumerName(from). A client cannot reach this path at all (reserved names are rejected for every client-facing kind), and a client's ack on a user stream is untouched.
  • The limit is documented rather than overclaimed: this makes a vote attributable, not truthful. A peer can still ack a command it did not apply durably — a mesh member is trusted with its own copy, the same trust that lets it hold one — and authenticating peers is phase 6. What is now impossible is a quorum assembled from names that do not correspond to the nodes that sent them.
  • TestRelayedAckVotesOnlyForTheSendingNode relays an ack from node 1 naming node 0's consumer: refused, confirmation count stays at 1, and the comment records that the replica's own ack for the same position is the one that was accepted (that is what the test's waitForRecords waited for).

3. 13 lines of doc comment on a 27-line helper — resolved

Fair. The new comments are trimmed to the invariants a reader cannot infer from the code: what a batch covers, why a segment keeps its file while a writer is pending, what the vote check does and does not promise. The rationale now lives on the code that depends on it (confirmQuorum, syncSegmentLocked, awaitQuorum) rather than being restated on each helper. The pre-existing long comments elsewhere in store.go were not touched — they describe the on-disk format and the rewind invariants, but if you want that pass too, say so and I will do it as its own commit.

Verification on this head

  • go vet ./... / go vet -tags rapid ./... — clean; golangci-lint run and --build-tags rapid — 0 issues.
  • go test -race ./... 24.9 s, plus three -shuffle=on runs (clean; that hammer is what found the two races the first commit fixed).
  • go test -race -tags rapid ./... 41.5 s.
  • gremlins unleash --dry-run — 1171 runnable, 90 not covered, 92.86% (from 1168/91/92.77%): no new uncovered class, and coverage 88.6% (from 88.5%). coverage.log has the line.
  • bench.log unchanged: neither finding touches a benchmarked path, and the two benchmarks published by this PR (BenchmarkPublishGroupCommit, BenchmarkQuorumPublish) are unaffected by a name check on the relayed ack path and a log-level branch.
Review resolved in `d4d55e2` (pushed; head of `feat/phase-2-quorum-dirable-publishes` — correction, head of `feat/phase-2-quorum-durable-publishes`). All three points addressed, one of them with a different fix than suggested, with the reason below. Thanks for the trace of the group-commit state machine — that is the part I most wanted a second pair of eyes on. ## 1. `awaitQuorum` rewrites cancellations as `ErrNoQuorum` (inline 1194) — resolved, classification kept deliberately Fixed in the parts that were actually wrong, kept in the part where I think `ErrNoQuorum` is the right code, and documented so the behaviour is a contract rather than an accident: - The **log level** was wrong: a wait the context ended is now `Debug` (`"a publish stopped waiting for a quorum"`), not a `Warn` about a quorum that never had the chance to form. The deadline case stays `Warn`. - The **error text** now says which one it was: `"the wait ended with N of M replicas confirmed"` for the context case versus `"confirmed by N of M replicas"` for the deadline. - The **inner error is preserved**, and I want to flag that it already was: `fmt.Errorf` with two `%w` verbs produces an error matching both, so `errors.Is(err, ErrNoQuorum)` *and* `errors.Is(err, context.Canceled)` (or `context.DeadlineExceeded`) are true today. There is a test for exactly that now. - I did **not** switch to returning the context error unwrapped. `codeFor` has no mapping for context errors, so it would land on `codeInternal`: the wire would say "internal server error" and `Server.errorResponse` would log a client's departure or a shutdown at ERROR level. A publish that stopped waiting is not a server fault, and its outcome is the same as a quorum failure: nothing was acked, and `ErrNoQuorum`'s contract ("may still be stored, may still be readable, may still replicate later") is the actionable statement for the caller — retry idempotently with the same `WithDeduplicationID`. Inventing a `codeCanceled` for a case no client can act on differently seemed worse than one code with the cause in the message and the inner error. `ErrNoQuorum`'s doc, README's Write-quorum row, `doc.go` and `plan.md` all now say this explicitly. - One correction to the premise, which does not change the outcome but is worth recording: the server-side handler runs under the *stream* context, not the request context, so a client's near deadline is not applied to it — the README "Write bounds" row you cited is about the client's own frame write (`rpcStream.writeRequest` pushing `SetWriteDeadline`). On the server the context ends when the inbound application stream does: a relaying node's relay deadline expiring and dropping its connection, the client dropping it, or `Close` cancelling `stopServing`. Those are exactly the cases the Debug log now covers. ## 2. Quorum votes granted without verifying the confirming node (replicate.go:373) — resolved, and you were right that this is the sharp edge This was the more serious of the two and I had rationalised it as "phase 6 authentication": the vote's identity was the consumer name the ack carried (`rep-<peer ID>`), and nothing bound that name to the node behind the connection. Any mesh peer could confirm a record for a node that never applied it, and a single node could reach a quorum by acking under several names — a quorum with no second copy behind it, which is exactly the promise P1 exists to make. Fixed: - `dispatchFrom` now owns the `frameForward` dispatch and passes the sending peer into `peerDispatch` (it is the only place that knows who sent the frame; the branch moved out of `clientDispatch`, which documented that a relay never reaches it). - `peerDispatch` calls the new `checkRelayedAckVote`, which refuses a relayed ack on a reserved stream unless its consumer is the sending node's own `replicaConsumerName(from)`. A client cannot reach this path at all (reserved names are rejected for every client-facing kind), and a client's ack on a user stream is untouched. - The limit is documented rather than overclaimed: this makes a vote **attributable, not truthful**. A peer can still ack a command it did not apply durably — a mesh member is trusted with its own copy, the same trust that lets it hold one — and authenticating peers is phase 6. What is now impossible is a quorum assembled from names that do not correspond to the nodes that sent them. - `TestRelayedAckVotesOnlyForTheSendingNode` relays an ack from node 1 naming node 0's consumer: refused, confirmation count stays at 1, and the comment records that the replica's own ack for the same position is the one that was accepted (that is what the test's `waitForRecords` waited for). ## 3. 13 lines of doc comment on a 27-line helper — resolved Fair. The new comments are trimmed to the invariants a reader cannot infer from the code: what a batch covers, why a segment keeps its file while a writer is pending, what the vote check does and does not promise. The rationale now lives on the code that depends on it (`confirmQuorum`, `syncSegmentLocked`, `awaitQuorum`) rather than being restated on each helper. The pre-existing long comments elsewhere in `store.go` were not touched — they describe the on-disk format and the rewind invariants, but if you want that pass too, say so and I will do it as its own commit. ## Verification on this head - `go vet ./...` / `go vet -tags rapid ./...` — clean; `golangci-lint run` and `--build-tags rapid` — `0 issues.` - `go test -race ./...` 24.9 s, plus three `-shuffle=on` runs (clean; that hammer is what found the two races the first commit fixed). - `go test -race -tags rapid ./...` 41.5 s. - `gremlins unleash --dry-run` — 1171 runnable, 90 not covered, 92.86% (from 1168/91/92.77%): no new uncovered class, and coverage 88.6% (from 88.5%). `coverage.log` has the line. - `bench.log` unchanged: neither finding touches a benchmarked path, and the two benchmarks published by this PR (`BenchmarkPublishGroupCommit`, `BenchmarkQuorumPublish`) are unaffected by a name check on the relayed ack path and a log-level branch.
sam merged commit 1cf83ccf40 into main 2026-10-19 19:55:27 +00:00
sam deleted branch feat/phase-2-quorum-durable-publishes 2026-10-19 19:55:27 +00:00
Sign in to join this conversation.
No reviewers
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
norobo/webstream!16
No description provided.