feat: stream terms, fencing and the stale-term refusal (Phase 1) #15
No reviewers
Labels
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
norobo/webstream!15
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/phase-1-terms-fencing"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What this is
Phase 1 of
plan.md— stream terms, fencing and the safety net — plus theplan-resolution commits it depends on.
f0d796ce84b19973dbaa4d9f8c1c5c04cbb76d4d8aBranch name follows Conventional Branch (
feat/); every commit followsConventional Commits. The review's findings and their resolutions are answered in
the comment below (
76d4d8a).Phase 1, as implemented
term uint64is monotone, 1-based, stream-scoped, and stops one belowMaxUint64(reserved as the invalid value). It lives instream.meta(format2: magic, version, term, maxAge) and on every command of the stream's
internal-<name>command log, including the newTermcommand that announces araise.
Server.persistTermwrites the meta durably — temp file, fsync, rename, and anfsync of the stream's own directory, which the store did not do for a renamed
stream.meta— before any command carries the term.Server.bumpTermraises,records and announces a term. Startup resumes at the meta's term.
appendCommand, under the stream'swriteLock, so no caller can supply a stale one and a raise cannot interleavebetween reading a term and writing the command that carries it. Callers that
already hold that lock (the publish pipeline) use
appendCommandLocked.Server.admitTermruns before every applied command: below this node's termthe command is refused (
errStaleTermCommand, and the replica stops applyingthat log), at it is applied, above it is recorded durably first. Admitting a
Term command is applying it.
superseded: publish, ack, createConsumer and deleteConsumer fail with the new
ErrStaleTerm(codeStaleTerm), and the append refuses to write at a staleterm, which also covers the trim notices the byte budget publishes on its own.
Refusals change no local state and consume no sequence number.
PublishAck.Term.unchanged.
Two corrections to the plan text
Both were applied to
plan.mdrather than worked around in code:the publish reply carries it: publish is the mutation whose reply a caller
correlates with a sequence number, and a refused mutation reports the term
through
ErrStaleTerminstead. Six reply shapes with no reader would have beenwire ceremony.
stream.metaand the command log.Writing the meta first makes the log unable to be ahead of it, so recovery
reads the meta and there is no maximum to take. The plan now states the
ordering invariant instead of the reconciliation.
One obligation is recorded for Phase 3, not silently absorbed: a loaded stream
resumes serving at the term it recovered, which is safe only while leadership
cannot move. Phase 3 owes the other half — a node serves a stream only at a term
it has itself announced, after the read quorum.
Tests
ErrStaleTermand changes no localstate (sequence number, command log, retained bytes, consumer watermarks);
ack, createConsumer and deleteConsumer refuse the same way.
its own term — exactly the boundary, not one either side.
with no bump keeps its term.
refusal of a stale write) against a shadow model.
FuzzDecodeCommandcarries the term, seeds the two invalid values, andasserts no accepted command carries a term outside the valid range.
and the stream keeps serving with one leader, with no
ErrForwardLoopon therelay path.
refusing to bump, an invalid term on disk being refused, a cold stream's trim
notice carrying its meta term, a payload truncated inside the term field
reporting the truncation, and
term.go'sTerm/ErrTermwrapper tests thatan earlier commit of this branch deleted by filename collision.
Verification
Coverage is 88.0% statements, up from 87.8%. The NOT COVERED count is below the
recorded baseline (1098/93/92.19%); the mutants this change adds fall into the two
documented false-positive classes (package-level const arithmetic and Go's
counter-less case guards),
coverage.logrecords the detail andTESTING_PLAN.mdthe baseline.
bench.loggained the command-codec numbers, which move asexpected: an append command carries 8 more bytes (368 ns/encode, 440 B/op vs
289 ns/344 B/op) and decoding reads one more field (237 ns vs 217 ns).
Not in this PR
Phases 2–12 of
plan.md: quorum-durable publishes, promotion, holder sets,dynamic membership, the cluster metadata log, security, the porcupine harness, the
ops surface, snapshots and batched pulls, the browser client, release readiness.
Nothing here is merged, and
mainis untouched.Every question in the plan's "Risks / open questions" section is replaced by a decision, and the defaults that can only be justified by a measurement are moved behind a named gate with the number that releases them. Decided by the operator: - `WithWriteQuorum(1)` stays the default everywhere; a majority is required by a runtime guard in `Server.Promote`, `Server.TransferLeadership` and the automatic path, so the combination that loses an acked sequence cannot be configured. If group commit cannot cover the added round trip the knob is the holder count `R`, never a quorum below majority. - Batched pulls batch the fetch, not the acknowledgement: one ack per record, written without waiting for each confirmation, so `processedBefore` stays the scalar watermark and the exactly-once boundary stays the record. A page-level ack is rejected because it would move that boundary to the page. - `WithMeshPeers` becomes config epoch 0 and `AddPeer`/`RemovePeer` publish `Config{epoch >= 1}` on `internal-cluster` from then on. The initial set stays local because the set names the nodes that tail the log that would carry it. - `R < N` becomes the recommended default, gated on Phase 4's three measurements (forwarded read latency, holder occupancy under foreign blocked pulls, re-placement time under load) landing in `bench.log` in the same phase; the default flips to `min(3, N)` and `WithReplicationFactor(0)` is the documented `R = N` opt-out. Decided from what is already in the tree: - Term scope stays stream-scoped; the config epoch carries the total order over membership and placement, so no cluster-wide term is needed. - The majority assumption is stated in `doc.go` and tested by a symmetric-partition case: the minority fails closed on both promotion and writes while still answering reads. - Race detection is a fact of the image, not an open question. - Porcupine cost is bounded by shortening histories, never by weakening models. - A promoter keeps the tail above the read-quorum prefix, and the tail's length becomes a metric so the decision is revisited with data. - The term's durable home is `stream.meta` version 2, written (temp, fsync, rename, directory fsync) before the first write under that term; a snapshot carries `{base, next, term}`, so a `Term{t}` trimmed below a snapshot loses nothing. Also resolves two in-phase hesitations: the metrics dependency is `github.com/prometheus/client_golang`, already required by libp2p, rather than a hand-rolled exporter, and the P4 and consumer-dedup rows in the promise and resolved-decision tables are updated to match.`WithWriteQuorum` and `WithReplicationFactor` are gone from the plan. Both are policy that follows from the topology, and offering them as options let an operator configure the one combination that loses an acked record: a write quorum that does not intersect the read quorum a promotion takes. The rules, now the whole contract: R = min(3, N) # holders = the R lowest-rank live peers quorum = R/2 + 1 # so R = 1 -> 1, 2 -> 2, 3 -> 2, 5 -> 3 `N` is the applied member count (config epoch, Phase 5), so placement recomputes itself when membership changes and nothing stores it: no option, no per-stream placement record, no startup guard, no way to drift. `R = min(3, N)` equals `R = N` for three nodes or fewer, so no shipped test changes behavior and only a mesh of four or more nodes sees placement at all. A single node (N = 1, R = 1, quorum 1) keeps the shipped per-publish fsync cost byte for byte. Consequences recorded in the plan: - The promotion guard becomes a startup assertion: the derivation cannot produce a quorum below a majority of holders, so there is nothing for an operator to misconfigure. - The availability table is written from the derived rule. `N = 2` keeps durability and loses write availability (`quorum(2) = 2`): two holders are enough for durability, not for failover, which is what the plan now says explicitly. `N >= 3` tolerates one loss for writes, two for reads. - The default flip and its gate are gone, because there is no default to release; Phase 4's measurements (forwarded read latency, holder occupancy under foreign blocked pulls, re-placement time) become published numbers. The "Gates (measurements that release defaults)" section becomes "Measurements and the claims they back", and still picks the two tuned constants the design has left: the batched page size and the porcupine caps. - Phase 5 states that `R` follows the applied member count, so a membership change needs no placement record, and Phase 4 states that `Holders{epoch, ids}` is the leader's derived statement rather than each node's own computation, since live views legitimately differ while a failure is detected. Also fixes two stale references: the P6 row and Phase 1's fencing note no longer describe forwarding or failover as gated on a measurement.Two changes to Phase 2 and Phase 3. `WithFlushInterval` is deleted rather than defaulted. With quorum writes the ack waits for the fsync that covers it, so the option's documented tradeoff -- an early ack against a crash window -- stops existing, and the only choice left is latency against throughput, which the write path can make per load. Group commit becomes a coalescing loop rather than a timer: the first publish with no fsync outstanding becomes the syncer, later arrivals join its batch and wait on a generation counter, and the next publish after it completes starts the next batch. An idle server pays the one fsync it pays today, a busy one pays about one fsync per batch, and no hidden interval shows up in the published latency. A failed batch fsync fails every waiter in it. README's durable-publish row, the Phase 9 API-change list and the Phase 9 metrics all follow. The promotion read quorum is taken over the recorded `Holders{epoch, ids}` -- the set that still contains the failed leader -- not over a recomputation from the live view. Recomputing first would let the read quorum quietly stop intersecting the write quorum, and at R = 1 it would let a node with no copy at all promote an empty log. A dead holder is asked and does not answer; that is a failed read quorum, which is the correct outcome, and only a serving leader records the next epoch.feat: stream terms and fencing (Phase 1), with the plan's open questions resolvedto feat: stream terms, fencing and the stale-term refusal (Phase 1)AI code review — Phase 1 (stream terms, fencing, stale-term refusal)
Reviewed the full PR diff (5 commits, f0d796c..5c04cbb) via forgejo-mcp, with the diff read against surrounding context at the head ref. This is a high-quality change: the term machine is small, the durable-first ordering (meta before command) is coherent, the boundary tests are exact (at/below/above the term), and the plan text documents the Phase-3 obligations it deliberately leaves open.
Verification reproduced in this session
go vet ./...— clean (exit 0)golangci-lint run— 0 issuesgo test -race ./...— okgo test -race -tags rapid -run TestTermProperties— okgo test -run '^$' -fuzz=FuzzDecodeCommand -fuzztime=8s— 35k execs, PASSreplacedirectives. Module isgo 1.27.1, somax(Go 1.21+),slog.Uint64anderrors.Isusage is valid.Findings
One 🟠 IMPORTANT (latent, not reachable in Phase-1 production paths) and three 🟡 suggestions, inline. No 🔴 blocking findings: every claim in the PR body reproduced, and I found no defect that fails under Phase-1's shipped code paths.
Summary of the IMPORTANT finding:
appendCommandvalidates only supersession (term > served), not that the command's term still equals the stream's current term at append time. The three appenders that read the term outside the stream'swriteLock(publishTrim,emitConsumerUpsert,emitConsumerDelete) can therefore append a stale-term command (e.g.Trimat term 1) after aTerm{2}command once a raise interleaves. Every replica that applied the raise refuses the stale command witherrStaleTermCommand, andreplicateLooptreats that as permanent divergence and returns; reconciliation (reconcileInterval= 2s) restarts the loop, which re-pulls the same never-acked command and dies again — a permanent restart cycle per replica until the stale entry is trimmed. TodaybumpTermhas no production caller (verified:grep -rn "bumpTerm" --include="*.go" | grep -v _test.gofinds only replicate.go's definition), so the race is not reachable in Phase 1 — but Phase 3's promotion is exactly the code that callsbumpTerm, andpublishTrimruns every reconcile pass, so this is a landmine the plan's own Phase 3 will step on. The fix is one line: stamp the current term insideappendCommandat append time (all callers pass the same value today) instead of trusting the caller, which also serializes the term read against a concurrent raise; the alternative is holding the stream'swriteLockacross read+append in the emit/trim paths, aspublishReplicatedalready does.Also noted (body, no inline anchor): the old
term_test.gounit tests for the pre-existing terminal-handler wrapperTerm()/ErrTerm(TestTermWrapsError,TestTermNilYieldsErrTerm) were deleted when the file was repurposed, with no replacement.term.gois still shipped andconsumer.go's terminal-error branch (errors.Is(err, ErrTerm)) still depends on its semantics; end-to-end coverage survives via the "poison" handler tests (consumer_test.go,integration_test.go), so this is a focused-unit-test loss, not a coverage hole — restoring the two small tests is cheap.No code was modified, no commits pushed, no labels touched. Posting as COMMENT because no verified blocking issue requires REQUEST_CHANGES.
@ -96,2 +105,3 @@cmd := command{kind: kind}term := reader.readUint64()if !validTerm(term) {🟡 SUGGESTION — a payload truncated between the kind byte and the term field is misreported.
wireReader.readUint64(protocol.go:1138) returns 0 withreader.err = io.ErrUnexpectedEOFon a short payload;validTerm(0)is false, so this returns "command term 0" and masks the real truncation error — the exact case the comment on the kind-byte read (line 99) warns about. Fix: checkreader.errafterreadUint64and before thevalidTermgate, e.g.if reader.err != nil { return command{}, fmt.Errorf("decoding command: %w", reader.err) }. Verified:go test -run '^$' -fuzz=FuzzDecodeCommand -fuzztime=8spasses, but the error attribution on torn reads is still wrong for debugging.@ -699,1 +715,4 @@func (s *Server) appendCommand(ctx context.Context, stream string, cmd command) error {// A superseded node must not write at a stale term: every replica refuses such a command, so the// append is dropped here. bumpTerm raises the term before it appends its own Term command.if s.staleResident(stream) {🟠 IMPORTANT (latent) — this guard validates only supersession (
term > served), not thatcmd.termstill equals the stream's current term at append time.publishTrim(line 1424) andemitConsumerUpsert/emitConsumerDelete(1057/1077) read the term before callingappendCommandand outside the stream'swriteLock, so a raise interleaving between the read and this append lands a stale-term command (e.g.Trimat term 1) in the log after theTerm{2}command. Replicas that applied the raise then refuse it witherrStaleTermCommand, and thereplicateLooperror handling (lines 1236–1241) treats that as permanent divergence and returns;forgetLoopdrops the entry and reconciliation (reconcileInterval= 2s, line 198) restarts the loop, which re-pulls the same never-acked command and dies again — a permanent 2s restart cycle per replica until the leader's log trims the entry. Verified today:grep -rn "bumpTerm" --include="*.go" . | grep -v _test.gofinds no production caller, so Phase 1 cannot trigger it; Phase 3's promotion is exactly what callsbumpTermnext topublishResidentTrims' per-passpublishTrim, so the landmine is real. Fix: derivecmd.terminsideappendCommandfrom the resident stream at append time (every caller passes the same value today), or hold the stream'swriteLockacross read+append in the emit/trim paths aspublishReplicatedalready does — serializing term adoption underwriteLockalso protects the sharedstream.meta.tmpfrom concurrentpersistTermwriters once promotion lands.@ -819,0 +869,4 @@}defer s.release(name)if state.termValue() < term {🟡 SUGGESTION — dead branch: in the replication path
admitTerm(line 881) runs first and already persists any term above the node's own (term > current→persistTerm), so whenapplyTermrunsstate.termValue() < termis never true and this persist is unreachable; the function reduces to a second acquire+release perTermcommand. The only caller that exercises the persist branch is the test at term_test.go:131. Either fold the term-command handling intoadmitTermand dropapplyTerm, or document why the double-check exists. Verified by reading replicate.go 881–905 (admitTerm's default branch) and the call sequence at 1213–1215.Resolved in
76d4d8a(pushed). One piece of bookkeeping first: you were right thatterm_test.go's twoTerm/ErrTermtests vanished, and it was worse than a repurpose — I wrote the new stream-term tests overterm_test.goby filename collision, soTestTermWrapsErrorandTestTermNilYieldsErrTermwere deleted outright.term_test.gois restored verbatim from the pre-change tree and the stream-term tests live instream_term_test.go/stream_term_property_test.gonow. Thank you for catching that; a coverage log line records it.🟠 IMPORTANT — stale-term append races a raise
Fixed as you suggested (stamp at append time), with the lock discipline made explicit rather than left to callers:
appendCommand(ctx, state, cmd)takes the stream'swriteLock, checks supersession, stampscmd.term = state.termValue(), then appends.publishTrim,emitConsumerUpsertandemitConsumerDeleteno longer read or pass a term at all — the bug class is unrepresentable rather than merely fixed, since nothing outsideappendCommandcan supply a term.appendCommandLockedis that same step forpublishReplicated, which already holdswriteLockacross reserving the sequence number, writing the command and applying the record.appendToCommandLogappends an already-stamped command and has exactly one caller: the cold-stream trim path, where the term can only come from the meta because nothing has the stream loaded — and a term cannot be raised on a stream that is not loaded, so there is nothing to race.persistTermraises the term before the announcement is appended, so the stamp is the new term.TestTrimNoticeCarriesTheTermInForcepins that.stream.meta.tmp:persistTermnow takes the samewriteLock(with apersistTermLockedvariant forbumpTerm, which holds it), so two writers of onestream.metacannot interleave once promotion lands.🟡 SUGGESTION —
applyTerm's unreachable persistDropped, as you suggested.
admitTermpersists any term above the node's own before the apply switch runs, so admitting a Term command is applying it; the switch case now says exactly that and the extra acquire/release per Term command is gone.TestSupersededLeaderRefusesMutationsdroveapplyTermand now drivesadmitTerm, which is the call the apply loop actually makes.🟡 SUGGESTION — truncated payload misreported as an invalid term
Fixed:
reader.erris checked after the term read and beforevalidTerm, so a payload cut inside the term field reports the truncation instead of "command term 0".TestDecodeCommandReportsATruncatedTermcovers both cut points (at the term's first byte and one byte short of it).Verification after the fix
Coverage is 88.0% statements. Relevant to your finding's blast radius, the mutation numbers moved the right way: the recorded baseline was 1098/93/92.19%, and folding
appendToCommandLog's retried-acquire error check into one shared branch removed an uncovered mutant and made that check killable, so this is now 92 not covered and 92.51%.TESTING_PLAN.mdandcoverage.logcarry the new figures.One thing I deliberately did not change, for the record: the replica still treats a command below its term as permanent divergence and stops (
errStaleTermCommand), rather than skipping it or resyncing. With the producer gone that state is unreachable by construction, and skipping a command would diverge a copy silently; the loud stop plus the comment on the sentinel is the outcome I want if it ever does appear.