feat: replicate consumer state and arbitrate the consume lease on the leader (phase 4) #10
No reviewers
Labels
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
norobo/webstream!10
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/mesh-consumer-replication"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Phase 4 of
plan.md: consumer state is stream state, so it replicates on the same command log as records, and one consumer name has one live consume loop across the mesh.What this adds
ConsumerUpsert,ConsumerDelete) inreplicate.go, emitted by the leader on consumer create/ack/delete and applied by replicas instate.go, monotonically so a stale or re-ordered command can never lower a watermark a local loop has already passed.frameLeaseCheckinprotocol.go. A pull on a non-leader validates and refreshes the lease on the leader on every pull (Server.replicaPullinserver.go), mirrors the reply's authoritativeprocessedBeforeand cursor, serves the record at the leader's cursor from the local copy, and blocks on its own applier until that sequence has replicated.ErrConsumerBusyis now a real cross-node guarantee with no stale-view window.applyForwardedReplyinclientDispatchmirrors the leader's watermark/lease/delete on the receiving node, andframeGetConsumerstays forwarded, so a restarted handle resumes exactly where it stopped no matter which node serves it.Tests
New
replication_consumer_test.gocovers all six phase-4 scenarios: restarted-handle read-your-writes through a stale replica, cross-node continuation without reprocessing, command-log consumer visibility with matching watermarks, no consumer resurrection after a replica restart, immediate second-loop refusal with takeover after release, and mid-handler death redelivery. Two phase-3 tests that seeded consumers directly on a replica were updated to route through the dispatch path, since consumer state is leader-owned from phase 4 on.Verification
go build ./...,go vet ./...,go test ./...pass.golangci-lint run: one new finding vsmain(the deliberateacquire-to-consumer-error pattern inleaseCheck, identical to the seven pre-existing ones inserver.go).TestLoopbackLifecycle,TestConsumeSurvivesEviction) fail identically onmain.Base:
main(2b34165, phase 3 merged as #8).AI code review (pi-reviewer) of phase 4 — consumer-state replication and leader-arbitrated consume leases.
Verified on a scratch checkout of head
df04566: go build ./... and go vet ./... clean; racedetector test passes all eight new phase-4 tests; the full-suite failures (TestLoopbackLifecycle, TestReconciliationPagesThroughAPeersStreams) reproduce identically on main (2b34165), and TestConsumeSurvivesEviction passes on both — no new test failures introduced. golangci-lint diff vs main: one new finding (the disclosed errhandle pattern at server.go:2011 in leaseCheck). govulncheck ./...: 0 reachable; go.mod/go.sum untouched, no new dependencies.One blocking durability defect in the new consumer-mirror persistence, plus one wire-compat suggestion.
🔴 BLOCKING — upsertReplicatedConsumer (state.go) creates the durable consumer slot at watermark 0 and never writes the incoming processedBefore, so a consumer whose first replicated upsert arrives with wm > 0 reloads at 0 after a restart/eviction. Reproduced with a scratch test (TestScratchReplicatedCreatePersistsWatermark): "in-memory watermark = 5, durable watermark = 0", and after store close+reopen getConsumer reports processedBefore = 0 (want 5). Reached via the documented resync path (a replica below the command log's retained base skips the wm=0 create command and applies only the later wm>0 upsert) or a forwarded-ack mirror on a node that never saw the create. Consequence: a leader-outage fallback pull re-serves already-processed records, contradicting the PR's "restarted handle resumes exactly where it stopped" guarantee.
🟡 SUGGESTION — frameLeaseCheck is inserted mid-enum in protocol.go, renumbering frameForward (12→13) on the wire, and the frameAck response changed from no-body to an 8-byte body (old decoders reject it as trailing bytes via wireReader.done). A rolling upgrade between main and this PR misdecodes frameForward. Moving frameLeaseCheck after frameForward keeps every existing kind number stable and matches the file's own "additive: no existing kind changes its encoding" principle.
@ -28,0 +29,4 @@// consumer's consume lease and reports the leader's authoritative cursor. It// is the mesh's lease arbitration: a replica's pull forwards it on every// pull while reading its record locally.frameLeaseCheck🟡 SUGGESTION — inserting frameLeaseCheck mid-enum renumbers frameForward (12 → 13) on the wire: the kinds are sequential iota, so everything after the insertion shifts. The frameForward comment just below claims the protocol is additive ("no existing kind changes its encoding"), and this change silently breaks that invariant — a node on this PR decodes a main node's frameForward (12) as frameLeaseCheck and vice versa. The frameAck response shape also changed from no-body to an 8-byte body (encodeTo/decodeResponse), which main's decoder rejects as trailing bytes (wireReader.done errors on unconsumed input). A rolling restart of the mesh therefore misdecodes frames until every node runs the new version.
Fix: append frameLeaseCheck after frameForward (end of the enum) so every existing kind keeps its number. The ack-reply shape change is inherent to the feature, but the renumbering is free to avoid.
@ -341,0 +355,4 @@state.minCursorValid = falsestate.lock.Unlock()state.persistConsumerCreate(name)🔴 BLOCKING — the replicated-create path drops the watermark on disk. persistConsumerCreate(name) only allocates the slot; consumerSlot starts it at watermark 0 and putWatermark is never called on this branch, so processedBefore survives only in memory. Reproduced with a scratch test:
go test -run TestScratchReplicatedCreatePersistsWatermark→ "in-memory watermark = 5, durable watermark = 0", and after store close+reopen getConsumer reports processedBefore = 0 (want 5).Condition: this branch carries wm > 0 whenever the first upsert a replica applies for a consumer is not the create command — the documented resync (a replica below the command log's retained base skips the wm=0 command; see doc.go and applyReplicated) or a forwarded-ack mirror on a node that never applied the create. The update branch below correctly calls persistConsumer(name, processedBefore) when raised; the create branch never does.
Consequence: after a replica restart or idle eviction the mirror materializes at 0 while the leader is at 5, and a leader-outage fallback pull (serveLocalPull) re-serves records 0-4 to a restarted handle — reprocessing, which contradicts the PR's own guarantee ("a restarted handle resumes exactly where it stopped no matter which node serves it"). TestDeletedConsumerDoesNotResurrectOnARestartedReplica covers the delete tombstone but not this restart-with-wm>0 case.
Fix: after state.persistConsumerCreate(name), also persist the watermark when nonzero:
Both review 42 findings resolved at head
07762e4.1. 🔴 BLOCKING — replicated consumer create persisted watermark 0 — fixed in
state.go(upsertReplicatedConsumer): the create branch now writes the incomingprocessedBeforeto the durable slot alongside the slot allocation, so a consumer whose first replicated upsert carries wm > 0 (a resync that skipped the create command, or a forwarded-ack mirror that never saw one) reloads at that watermark after a restart instead of 0. Regression test added:TestReplicatedConsumerCreatePersistsItsWatermark— creates a mirror at wm 5 on a durable server, closes and reopens the storage root, asserts the recovered consumer reports 5. Verified: the test passes with the fix (it failed against the previous head in the same reproduction the review describes); full suite, vet, build, lint delta vs main unchanged (still the single disclosedleaseCheckacquire pattern).2. 🟡 SUGGESTION — frameLeaseCheck renumbered frameForward — fixed in
protocol.go:frameLeaseCheckis now declared afterframeForward, so every existing kind keeps its wire number (frameForward stays 12) and only the new kind is added at 13. On the ack-reply body change noted in the same finding: that is deliberate — the authoritativeprocessedBeforein the ack reply is what lets the node a forwarded ack arrived at apply the leader's move immediately (the phase-4 read-your-writes fast path), and this protocol ships in one repo with no versioned external decoders; the enum move restores the additive property the finding asked for.