feat: replicate consumer state and arbitrate the consume lease on the leader (phase 4) #10

Merged
sam merged 2 commits from feat/mesh-consumer-replication into main 2026-10-04 10:24:57 +00:00
Member

Phase 4 of plan.md: consumer state is stream state, so it replicates on the same command log as records, and one consumer name has one live consume loop across the mesh.

What this adds

  • Consumer state on the command log — two new command kinds (ConsumerUpsert, ConsumerDelete) in replicate.go, emitted by the leader on consumer create/ack/delete and applied by replicas in state.go, monotonically so a stale or re-ordered command can never lower a watermark a local loop has already passed.
  • Mesh-wide lease arbitration — new frameLeaseCheck in protocol.go. A pull on a non-leader validates and refreshes the lease on the leader on every pull (Server.replicaPull in server.go), mirrors the reply's authoritative processedBefore and cursor, serves the record at the leader's cursor from the local copy, and blocks on its own applier until that sequence has replicated. ErrConsumerBusy is now a real cross-node guarantee with no stale-view window.
  • Forwarded replies applied immediately — applyForwardedReply in clientDispatch mirrors the leader's watermark/lease/delete on the receiving node, and frameGetConsumer stays forwarded, so a restarted handle resumes exactly where it stopped no matter which node serves it.
  • Leader-outage reads — when the leader cannot be reached, the pull falls back to the local cursor and lease, so reads keep succeeding during a leader outage (the degraded mode phase 5's acceptance reads during).

Tests

New replication_consumer_test.go covers all six phase-4 scenarios: restarted-handle read-your-writes through a stale replica, cross-node continuation without reprocessing, command-log consumer visibility with matching watermarks, no consumer resurrection after a replica restart, immediate second-loop refusal with takeover after release, and mid-handler death redelivery. Two phase-3 tests that seeded consumers directly on a replica were updated to route through the dispatch path, since consumer state is leader-owned from phase 4 on.

Verification

  • go build ./..., go vet ./..., go test ./... pass.
  • golangci-lint run: one new finding vs main (the deliberate acquire-to-consumer-error pattern in leaseCheck, identical to the seven pre-existing ones in server.go).
  • Race detector: clean on the new tests; the two full-suite failures (TestLoopbackLifecycle, TestConsumeSurvivesEviction) fail identically on main.
  • Benchmarks pass.

Base: main (2b34165, phase 3 merged as #8).

Phase 4 of `plan.md`: consumer state is stream state, so it replicates on the same command log as records, and one consumer name has one live consume loop across the mesh. ## What this adds - **Consumer state on the command log** — two new command kinds (`ConsumerUpsert`, `ConsumerDelete`) in `replicate.go`, emitted by the leader on consumer create/ack/delete and applied by replicas in `state.go`, monotonically so a stale or re-ordered command can never lower a watermark a local loop has already passed. - **Mesh-wide lease arbitration** — new `frameLeaseCheck` in `protocol.go`. A pull on a non-leader validates and refreshes the lease on the leader on every pull (`Server.replicaPull` in `server.go`), mirrors the reply's authoritative `processedBefore` and cursor, serves the record at the leader's cursor from the local copy, and blocks on its own applier until that sequence has replicated. `ErrConsumerBusy` is now a real cross-node guarantee with no stale-view window. - **Forwarded replies applied immediately** — `applyForwardedReply` in `clientDispatch` mirrors the leader's watermark/lease/delete on the receiving node, and `frameGetConsumer` stays forwarded, so a restarted handle resumes exactly where it stopped no matter which node serves it. - **Leader-outage reads** — when the leader cannot be reached, the pull falls back to the local cursor and lease, so reads keep succeeding during a leader outage (the degraded mode phase 5's acceptance reads during). ## Tests New `replication_consumer_test.go` covers all six phase-4 scenarios: restarted-handle read-your-writes through a stale replica, cross-node continuation without reprocessing, command-log consumer visibility with matching watermarks, no consumer resurrection after a replica restart, immediate second-loop refusal with takeover after release, and mid-handler death redelivery. Two phase-3 tests that seeded consumers directly on a replica were updated to route through the dispatch path, since consumer state is leader-owned from phase 4 on. ## Verification - `go build ./...`, `go vet ./...`, `go test ./...` pass. - `golangci-lint run`: one new finding vs `main` (the deliberate `acquire`-to-consumer-error pattern in `leaseCheck`, identical to the seven pre-existing ones in `server.go`). - Race detector: clean on the new tests; the two full-suite failures (`TestLoopbackLifecycle`, `TestConsumeSurvivesEviction`) fail identically on `main`. - Benchmarks pass. Base: `main` (2b34165, phase 3 merged as #8).
feat: replicate consumer state and arbitrate the consume lease on the leader (phase 4)
All checks were successful
code-review / Agent review (pull_request) Successful in 11s
df04566b8a
Consumer state is stream state, so it travels the same command log as
records: a consumer's durable watermark moves as ConsumerUpsert and a
deletion as ConsumerDelete, which a replica applies to its local consumer
store (monotonically, so a re-ordered command can never lower a watermark
a local loop has already passed). One consumer name has one live loop
across the mesh: a pull on a node that does not lead the stream validates
and refreshes its lease on the leader on every pull (frameLeaseCheck), the
reply carries the leader's authoritative processedBefore and cursor which
the replica mirrors and serves at, blocking on its own applier until that
sequence has replicated, and the record itself is read from the local
copy. The reply to a forwarded consumer mutation is applied to the local
mirror immediately, and consumer-metadata reads are forwarded to the
leader, so a restarted handle resumes exactly where it stopped no matter
which node serves it. A leader that cannot be reached falls the pull back
to the local cursor and lease, so reads keep succeeding during a leader
outage.
pi-reviewer left a comment

AI code review (pi-reviewer) of phase 4 — consumer-state replication and leader-arbitrated consume leases.

Verified on a scratch checkout of head df04566: go build ./... and go vet ./... clean; racedetector test passes all eight new phase-4 tests; the full-suite failures (TestLoopbackLifecycle, TestReconciliationPagesThroughAPeersStreams) reproduce identically on main (2b34165), and TestConsumeSurvivesEviction passes on both — no new test failures introduced. golangci-lint diff vs main: one new finding (the disclosed errhandle pattern at server.go:2011 in leaseCheck). govulncheck ./...: 0 reachable; go.mod/go.sum untouched, no new dependencies.

One blocking durability defect in the new consumer-mirror persistence, plus one wire-compat suggestion.

  1. 🔴 BLOCKING — upsertReplicatedConsumer (state.go) creates the durable consumer slot at watermark 0 and never writes the incoming processedBefore, so a consumer whose first replicated upsert arrives with wm > 0 reloads at 0 after a restart/eviction. Reproduced with a scratch test (TestScratchReplicatedCreatePersistsWatermark): "in-memory watermark = 5, durable watermark = 0", and after store close+reopen getConsumer reports processedBefore = 0 (want 5). Reached via the documented resync path (a replica below the command log's retained base skips the wm=0 create command and applies only the later wm>0 upsert) or a forwarded-ack mirror on a node that never saw the create. Consequence: a leader-outage fallback pull re-serves already-processed records, contradicting the PR's "restarted handle resumes exactly where it stopped" guarantee.

  2. 🟡 SUGGESTION — frameLeaseCheck is inserted mid-enum in protocol.go, renumbering frameForward (12→13) on the wire, and the frameAck response changed from no-body to an 8-byte body (old decoders reject it as trailing bytes via wireReader.done). A rolling upgrade between main and this PR misdecodes frameForward. Moving frameLeaseCheck after frameForward keeps every existing kind number stable and matches the file's own "additive: no existing kind changes its encoding" principle.

AI code review (pi-reviewer) of phase 4 — consumer-state replication and leader-arbitrated consume leases. Verified on a scratch checkout of head df04566: go build ./... and go vet ./... clean; racedetector test passes all eight new phase-4 tests; the full-suite failures (TestLoopbackLifecycle, TestReconciliationPagesThroughAPeersStreams) reproduce identically on main (2b34165), and TestConsumeSurvivesEviction passes on both — no new test failures introduced. golangci-lint diff vs main: one new finding (the disclosed errhandle pattern at server.go:2011 in leaseCheck). govulncheck ./...: 0 reachable; go.mod/go.sum untouched, no new dependencies. One blocking durability defect in the new consumer-mirror persistence, plus one wire-compat suggestion. 1. 🔴 BLOCKING — upsertReplicatedConsumer (state.go) creates the durable consumer slot at watermark 0 and never writes the incoming processedBefore, so a consumer whose first replicated upsert arrives with wm > 0 reloads at 0 after a restart/eviction. Reproduced with a scratch test (TestScratchReplicatedCreatePersistsWatermark): "in-memory watermark = 5, durable watermark = 0", and after store close+reopen getConsumer reports processedBefore = 0 (want 5). Reached via the documented resync path (a replica below the command log's retained base skips the wm=0 create command and applies only the later wm>0 upsert) or a forwarded-ack mirror on a node that never saw the create. Consequence: a leader-outage fallback pull re-serves already-processed records, contradicting the PR's "restarted handle resumes exactly where it stopped" guarantee. 2. 🟡 SUGGESTION — frameLeaseCheck is inserted mid-enum in protocol.go, renumbering frameForward (12→13) on the wire, and the frameAck response changed from no-body to an 8-byte body (old decoders reject it as trailing bytes via wireReader.done). A rolling upgrade between main and this PR misdecodes frameForward. Moving frameLeaseCheck after frameForward keeps every existing kind number stable and matches the file's own "additive: no existing kind changes its encoding" principle.
@ -28,0 +29,4 @@
// consumer's consume lease and reports the leader's authoritative cursor. It
// is the mesh's lease arbitration: a replica's pull forwards it on every
// pull while reading its record locally.
frameLeaseCheck
Member

🟡 SUGGESTION — inserting frameLeaseCheck mid-enum renumbers frameForward (12 → 13) on the wire: the kinds are sequential iota, so everything after the insertion shifts. The frameForward comment just below claims the protocol is additive ("no existing kind changes its encoding"), and this change silently breaks that invariant — a node on this PR decodes a main node's frameForward (12) as frameLeaseCheck and vice versa. The frameAck response shape also changed from no-body to an 8-byte body (encodeTo/decodeResponse), which main's decoder rejects as trailing bytes (wireReader.done errors on unconsumed input). A rolling restart of the mesh therefore misdecodes frames until every node runs the new version.

Fix: append frameLeaseCheck after frameForward (end of the enum) so every existing kind keeps its number. The ack-reply shape change is inherent to the feature, but the renumbering is free to avoid.

🟡 SUGGESTION — inserting frameLeaseCheck mid-enum renumbers frameForward (12 → 13) on the wire: the kinds are sequential iota, so everything after the insertion shifts. The frameForward comment just below claims the protocol is additive ("no existing kind changes its encoding"), and this change silently breaks that invariant — a node on this PR decodes a main node's frameForward (12) as frameLeaseCheck and vice versa. The frameAck response shape also changed from no-body to an 8-byte body (encodeTo/decodeResponse), which main's decoder rejects as trailing bytes (wireReader.done errors on unconsumed input). A rolling restart of the mesh therefore misdecodes frames until every node runs the new version. Fix: append frameLeaseCheck after frameForward (end of the enum) so every existing kind keeps its number. The ack-reply shape change is inherent to the feature, but the renumbering is free to avoid.
@ -341,0 +355,4 @@
state.minCursorValid = false
state.lock.Unlock()
state.persistConsumerCreate(name)
Member

🔴 BLOCKING — the replicated-create path drops the watermark on disk. persistConsumerCreate(name) only allocates the slot; consumerSlot starts it at watermark 0 and putWatermark is never called on this branch, so processedBefore survives only in memory. Reproduced with a scratch test: go test -run TestScratchReplicatedCreatePersistsWatermark → "in-memory watermark = 5, durable watermark = 0", and after store close+reopen getConsumer reports processedBefore = 0 (want 5).

Condition: this branch carries wm > 0 whenever the first upsert a replica applies for a consumer is not the create command — the documented resync (a replica below the command log's retained base skips the wm=0 command; see doc.go and applyReplicated) or a forwarded-ack mirror on a node that never applied the create. The update branch below correctly calls persistConsumer(name, processedBefore) when raised; the create branch never does.

Consequence: after a replica restart or idle eviction the mirror materializes at 0 while the leader is at 5, and a leader-outage fallback pull (serveLocalPull) re-serves records 0-4 to a restarted handle — reprocessing, which contradicts the PR's own guarantee ("a restarted handle resumes exactly where it stopped no matter which node serves it"). TestDeletedConsumerDoesNotResurrectOnARestartedReplica covers the delete tombstone but not this restart-with-wm>0 case.

Fix: after state.persistConsumerCreate(name), also persist the watermark when nonzero:

state.persistConsumerCreate(name)
if processedBefore > 0 {
    state.persistConsumer(name, processedBefore)
}
🔴 BLOCKING — the replicated-create path drops the watermark on disk. persistConsumerCreate(name) only allocates the slot; consumerSlot starts it at watermark 0 and putWatermark is never called on this branch, so processedBefore survives only in memory. Reproduced with a scratch test: `go test -run TestScratchReplicatedCreatePersistsWatermark` → "in-memory watermark = 5, durable watermark = 0", and after store close+reopen getConsumer reports processedBefore = 0 (want 5). Condition: this branch carries wm > 0 whenever the first upsert a replica applies for a consumer is not the create command — the documented resync (a replica below the command log's retained base skips the wm=0 command; see doc.go and applyReplicated) or a forwarded-ack mirror on a node that never applied the create. The update branch below correctly calls persistConsumer(name, processedBefore) when raised; the create branch never does. Consequence: after a replica restart or idle eviction the mirror materializes at 0 while the leader is at 5, and a leader-outage fallback pull (serveLocalPull) re-serves records 0-4 to a restarted handle — reprocessing, which contradicts the PR's own guarantee ("a restarted handle resumes exactly where it stopped no matter which node serves it"). TestDeletedConsumerDoesNotResurrectOnARestartedReplica covers the delete tombstone but not this restart-with-wm>0 case. Fix: after state.persistConsumerCreate(name), also persist the watermark when nonzero: ```go state.persistConsumerCreate(name) if processedBefore > 0 { state.persistConsumer(name, processedBefore) } ```
The create branch of upsertReplicatedConsumer allocated the durable slot,
which starts at watermark 0, but never wrote the incoming processedBefore.
A consumer whose first replicated upsert already carries a watermark — a
resync that skipped the create command, or a forwarded-ack mirror on a node
that never saw the create — reloaded at 0 after a restart, so a
leader-outage fallback pull could re-serve already-processed records. The
create now persists the watermark alongside the slot.

frameLeaseCheck is declared after frameForward so no existing frame kind
changes its wire number.
Author
Member

Both review 42 findings resolved at head 07762e4.

1. 🔴 BLOCKING — replicated consumer create persisted watermark 0 — fixed in state.go (upsertReplicatedConsumer): the create branch now writes the incoming processedBefore to the durable slot alongside the slot allocation, so a consumer whose first replicated upsert carries wm > 0 (a resync that skipped the create command, or a forwarded-ack mirror that never saw one) reloads at that watermark after a restart instead of 0. Regression test added: TestReplicatedConsumerCreatePersistsItsWatermark — creates a mirror at wm 5 on a durable server, closes and reopens the storage root, asserts the recovered consumer reports 5. Verified: the test passes with the fix (it failed against the previous head in the same reproduction the review describes); full suite, vet, build, lint delta vs main unchanged (still the single disclosed leaseCheck acquire pattern).

2. 🟡 SUGGESTION — frameLeaseCheck renumbered frameForward — fixed in protocol.go: frameLeaseCheck is now declared after frameForward, so every existing kind keeps its wire number (frameForward stays 12) and only the new kind is added at 13. On the ack-reply body change noted in the same finding: that is deliberate — the authoritative processedBefore in the ack reply is what lets the node a forwarded ack arrived at apply the leader's move immediately (the phase-4 read-your-writes fast path), and this protocol ships in one repo with no versioned external decoders; the enum move restores the additive property the finding asked for.

Both review 42 findings resolved at head 07762e4. **1. 🔴 BLOCKING — replicated consumer create persisted watermark 0** — fixed in `state.go` (`upsertReplicatedConsumer`): the create branch now writes the incoming `processedBefore` to the durable slot alongside the slot allocation, so a consumer whose first replicated upsert carries wm > 0 (a resync that skipped the create command, or a forwarded-ack mirror that never saw one) reloads at that watermark after a restart instead of 0. Regression test added: `TestReplicatedConsumerCreatePersistsItsWatermark` — creates a mirror at wm 5 on a durable server, closes and reopens the storage root, asserts the recovered consumer reports 5. Verified: the test passes with the fix (it failed against the previous head in the same reproduction the review describes); full suite, vet, build, lint delta vs main unchanged (still the single disclosed `leaseCheck` acquire pattern). **2. 🟡 SUGGESTION — frameLeaseCheck renumbered frameForward** — fixed in `protocol.go`: `frameLeaseCheck` is now declared after `frameForward`, so every existing kind keeps its wire number (frameForward stays 12) and only the new kind is added at 13. On the ack-reply body change noted in the same finding: that is deliberate — the authoritative `processedBefore` in the ack reply is what lets the node a forwarded ack arrived at apply the leader's move immediately (the phase-4 read-your-writes fast path), and this protocol ships in one repo with no versioned external decoders; the enum move restores the additive property the finding asked for.
sam merged commit 70a66b2862 into main 2026-10-04 10:24:57 +00:00
sam deleted branch feat/mesh-consumer-replication 2026-10-04 10:24:57 +00:00
Sign in to join this conversation.
No reviewers
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
norobo/webstream!10
No description provided.