feat: stream terms, fencing and the stale-term refusal (Phase 1) #15

Merged
sam merged 6 commits from feat/phase-1-terms-fencing into main 2026-10-18 16:35:18 +00:00
Member

What this is

Phase 1 of plan.md — stream terms, fencing and the safety net — plus the
plan-resolution commits it depends on.

Commit Subject
f0d796c answer the plan's open questions before implementation starts
e84b199 derive replication placement and write quorum instead of configuring them
73dbaa4 automatic group commit, and pin the promotion read quorum to the recorded epoch
d9f8c1c drop the version-negotiation and upgrade ceremony
5c04cbb carry a stream term and fence a superseded leader (Phase 1)
76d4d8a stamp a command's term at append time, not in its caller (review findings)

Branch name follows Conventional Branch (feat/); every commit follows
Conventional Commits. The review's findings and their resolutions are answered in
the comment below (76d4d8a).

Phase 1, as implemented

  • term uint64 is monotone, 1-based, stream-scoped, and stops one below
    MaxUint64 (reserved as the invalid value). It lives in stream.meta (format
    2: magic, version, term, maxAge) and on every command of the stream's
    internal-<name> command log, including the new Term command that announces a
    raise.
  • Server.persistTerm writes the meta durably — temp file, fsync, rename, and an
    fsync of the stream's own directory, which the store did not do for a renamed
    stream.meta — before any command carries the term. Server.bumpTerm raises,
    records and announces a term. Startup resumes at the meta's term.
  • A command's term is stamped inside appendCommand, under the stream's
    writeLock, so no caller can supply a stale one and a raise cannot interleave
    between reading a term and writing the command that carries it. Callers that
    already hold that lock (the publish pipeline) use appendCommandLocked.
  • Server.admitTerm runs before every applied command: below this node's term
    the command is refused (errStaleTermCommand, and the replica stops applying
    that log), at it is applied, above it is recorded durably first. Admitting a
    Term command is applying it.
  • A node that observed a term above the one it serves a stream under is
    superseded: publish, ack, createConsumer and deleteConsumer fail with the new
    ErrStaleTerm (codeStaleTerm), and the append refuses to write at a stale
    term, which also covers the trim notices the byte budget publishes on its own.
    Refusals change no local state and consume no sequence number.
  • A publish reply carries the term it was served under, as PublishAck.Term.
  • Additive: with no failover every command carries term 1 and behavior is
    unchanged.

Two corrections to the plan text

Both were applied to plan.md rather than worked around in code:

  1. The plan said the term is added to "the response of a forwarded mutation". Only
    the publish reply carries it: publish is the mutation whose reply a caller
    correlates with a sequence number, and a refused mutation reports the term
    through ErrStaleTerm instead. Six reply shapes with no reader would have been
    wire ceremony.
  2. The plan said recovery takes the maximum of stream.meta and the command log.
    Writing the meta first makes the log unable to be ahead of it, so recovery
    reads the meta and there is no maximum to take. The plan now states the
    ordering invariant instead of the reconciliation.

One obligation is recorded for Phase 3, not silently absorbed: a loaded stream
resumes serving at the term it recovered, which is safe only while leadership
cannot move. Phase 3 owes the other half — a node serves a stream only at a term
it has itself announced, after the read quorum.

Tests

  • A superseded leader's publish fails ErrStaleTerm and changes no local
    state (sequence number, command log, retained bytes, consumer watermarks);
    ack, createConsumer and deleteConsumer refuse the same way.
  • A replica rejects a command carrying a term below its own and accepts one at
    its own term — exactly the boundary, not one either side.
  • A term bump survives a restart on both the leader and the replica; a stream
    with no bump keeps its term.
  • A rapid property model of the term machine (monotone term, supersession,
    refusal of a stale write) against a shadow model.
  • FuzzDecodeCommand carries the term, seeds the two invalid values, and
    asserts no accepted command carries a term outside the valid range.
  • A commit-log term disagreement across a restart resolves to the higher term
    and the stream keeps serving with one leader, with no ErrForwardLoop on the
    relay path.
  • Also covered: a trim notice stamped with the term in force, the ceiling
    refusing to bump, an invalid term on disk being refused, a cold stream's trim
    notice carrying its meta term, a payload truncated inside the term field
    reporting the truncation, and term.go's Term/ErrTerm wrapper tests that
    an earlier commit of this branch deleted by filename collision.

Verification

go vet ./...                                     clean
golangci-lint run                               0 issues (also -tags rapid)
go test -race ./...                             ok
go test -race -tags rapid ./...                  ok
go test -fuzz=FuzzDecodeCommand -fuzztime=15s    66k execs, no failure
gremlins unleash --dry-run                      1137 runnable, 92 not covered, 92.51%

Coverage is 88.0% statements, up from 87.8%. The NOT COVERED count is below the
recorded baseline (1098/93/92.19%); the mutants this change adds fall into the two
documented false-positive classes (package-level const arithmetic and Go's
counter-less case guards), coverage.log records the detail and TESTING_PLAN.md
the baseline. bench.log gained the command-codec numbers, which move as
expected: an append command carries 8 more bytes (368 ns/encode, 440 B/op vs
289 ns/344 B/op) and decoding reads one more field (237 ns vs 217 ns).

Not in this PR

Phases 2–12 of plan.md: quorum-durable publishes, promotion, holder sets,
dynamic membership, the cluster metadata log, security, the porcupine harness, the
ops surface, snapshots and batched pulls, the browser client, release readiness.

Nothing here is merged, and main is untouched.

## What this is Phase 1 of `plan.md` — **stream terms, fencing and the safety net** — plus the plan-resolution commits it depends on. | Commit | Subject | | --- | --- | | `f0d796c` | answer the plan's open questions before implementation starts | | `e84b199` | derive replication placement and write quorum instead of configuring them | | `73dbaa4` | automatic group commit, and pin the promotion read quorum to the recorded epoch | | `d9f8c1c` | drop the version-negotiation and upgrade ceremony | | `5c04cbb` | carry a stream term and fence a superseded leader (Phase 1) | | `76d4d8a` | stamp a command's term at append time, not in its caller (review findings) | Branch name follows Conventional Branch (`feat/`); every commit follows Conventional Commits. The review's findings and their resolutions are answered in the comment below (`76d4d8a`). ## Phase 1, as implemented - `term uint64` is monotone, 1-based, stream-scoped, and stops one below `MaxUint64` (reserved as the invalid value). It lives in `stream.meta` (format 2: magic, version, term, maxAge) and on every command of the stream's `internal-<name>` command log, including the new `Term` command that announces a raise. - `Server.persistTerm` writes the meta durably — temp file, fsync, rename, and an fsync of the stream's own directory, which the store did not do for a renamed `stream.meta` — before any command carries the term. `Server.bumpTerm` raises, records and announces a term. Startup resumes at the meta's term. - A command's term is stamped **inside** `appendCommand`, under the stream's `writeLock`, so no caller can supply a stale one and a raise cannot interleave between reading a term and writing the command that carries it. Callers that already hold that lock (the publish pipeline) use `appendCommandLocked`. - `Server.admitTerm` runs before every applied command: below this node's term the command is refused (`errStaleTermCommand`, and the replica stops applying that log), at it is applied, above it is recorded durably first. Admitting a Term command is applying it. - A node that observed a term above the one it serves a stream under is superseded: publish, ack, createConsumer and deleteConsumer fail with the new `ErrStaleTerm` (`codeStaleTerm`), and the append refuses to write at a stale term, which also covers the trim notices the byte budget publishes on its own. Refusals change no local state and consume no sequence number. - A publish reply carries the term it was served under, as `PublishAck.Term`. - Additive: with no failover every command carries term 1 and behavior is unchanged. ## Two corrections to the plan text Both were applied to `plan.md` rather than worked around in code: 1. The plan said the term is added to "the response of a forwarded mutation". Only the publish reply carries it: publish is the mutation whose reply a caller correlates with a sequence number, and a refused mutation reports the term through `ErrStaleTerm` instead. Six reply shapes with no reader would have been wire ceremony. 2. The plan said recovery takes the maximum of `stream.meta` and the command log. Writing the meta first makes the log unable to be ahead of it, so recovery reads the meta and there is no maximum to take. The plan now states the ordering invariant instead of the reconciliation. One obligation is recorded for Phase 3, not silently absorbed: a loaded stream resumes *serving* at the term it recovered, which is safe only while leadership cannot move. Phase 3 owes the other half — a node serves a stream only at a term it has itself announced, after the read quorum. ## Tests - [x] A superseded leader's publish fails `ErrStaleTerm` and changes no local state (sequence number, command log, retained bytes, consumer watermarks); ack, createConsumer and deleteConsumer refuse the same way. - [x] A replica rejects a command carrying a term below its own and accepts one at its own term — exactly the boundary, not one either side. - [x] A term bump survives a restart on both the leader and the replica; a stream with no bump keeps its term. - [x] A rapid property model of the term machine (monotone term, supersession, refusal of a stale write) against a shadow model. - [x] `FuzzDecodeCommand` carries the term, seeds the two invalid values, and asserts no accepted command carries a term outside the valid range. - [x] A commit-log term disagreement across a restart resolves to the higher term and the stream keeps serving with one leader, with no `ErrForwardLoop` on the relay path. - [x] Also covered: a trim notice stamped with the term in force, the ceiling refusing to bump, an invalid term on disk being refused, a cold stream's trim notice carrying its meta term, a payload truncated inside the term field reporting the truncation, and `term.go`'s `Term`/`ErrTerm` wrapper tests that an earlier commit of this branch deleted by filename collision. ## Verification ``` go vet ./... clean golangci-lint run 0 issues (also -tags rapid) go test -race ./... ok go test -race -tags rapid ./... ok go test -fuzz=FuzzDecodeCommand -fuzztime=15s 66k execs, no failure gremlins unleash --dry-run 1137 runnable, 92 not covered, 92.51% ``` Coverage is 88.0% statements, up from 87.8%. The NOT COVERED count is below the recorded baseline (1098/93/92.19%); the mutants this change adds fall into the two documented false-positive classes (package-level const arithmetic and Go's counter-less case guards), `coverage.log` records the detail and `TESTING_PLAN.md` the baseline. `bench.log` gained the command-codec numbers, which move as expected: an append command carries 8 more bytes (368 ns/encode, 440 B/op vs 289 ns/344 B/op) and decoding reads one more field (237 ns vs 217 ns). ## Not in this PR Phases 2–12 of `plan.md`: quorum-durable publishes, promotion, holder sets, dynamic membership, the cluster metadata log, security, the porcupine harness, the ops surface, snapshots and batched pulls, the browser client, release readiness. Nothing here is merged, and `main` is untouched.
Every question in the plan's "Risks / open questions" section is replaced by a
decision, and the defaults that can only be justified by a measurement are
moved behind a named gate with the number that releases them.

Decided by the operator:

- `WithWriteQuorum(1)` stays the default everywhere; a majority is required by
  a runtime guard in `Server.Promote`, `Server.TransferLeadership` and the
  automatic path, so the combination that loses an acked sequence cannot be
  configured. If group commit cannot cover the added round trip the knob is the
  holder count `R`, never a quorum below majority.
- Batched pulls batch the fetch, not the acknowledgement: one ack per record,
  written without waiting for each confirmation, so `processedBefore` stays the
  scalar watermark and the exactly-once boundary stays the record. A
  page-level ack is rejected because it would move that boundary to the page.
- `WithMeshPeers` becomes config epoch 0 and `AddPeer`/`RemovePeer` publish
  `Config{epoch >= 1}` on `internal-cluster` from then on. The initial set stays
  local because the set names the nodes that tail the log that would carry it.
- `R < N` becomes the recommended default, gated on Phase 4's three
  measurements (forwarded read latency, holder occupancy under foreign blocked
  pulls, re-placement time under load) landing in `bench.log` in the same phase;
  the default flips to `min(3, N)` and `WithReplicationFactor(0)` is the
  documented `R = N` opt-out.

Decided from what is already in the tree:

- Term scope stays stream-scoped; the config epoch carries the total order over
  membership and placement, so no cluster-wide term is needed.
- The majority assumption is stated in `doc.go` and tested by a
  symmetric-partition case: the minority fails closed on both promotion and
  writes while still answering reads.
- Race detection is a fact of the image, not an open question.
- Porcupine cost is bounded by shortening histories, never by weakening models.
- A promoter keeps the tail above the read-quorum prefix, and the tail's length
  becomes a metric so the decision is revisited with data.
- The term's durable home is `stream.meta` version 2, written (temp, fsync,
  rename, directory fsync) before the first write under that term; a snapshot
  carries `{base, next, term}`, so a `Term{t}` trimmed below a snapshot loses
  nothing.

Also resolves two in-phase hesitations: the metrics dependency is
`github.com/prometheus/client_golang`, already required by libp2p, rather than a
hand-rolled exporter, and the P4 and consumer-dedup rows in the promise and
resolved-decision tables are updated to match.
`WithWriteQuorum` and `WithReplicationFactor` are gone from the plan. Both are
policy that follows from the topology, and offering them as options let an
operator configure the one combination that loses an acked record: a write
quorum that does not intersect the read quorum a promotion takes.

The rules, now the whole contract:

  R       = min(3, N)              # holders = the R lowest-rank live peers
  quorum  = R/2 + 1                # so R = 1 -> 1, 2 -> 2, 3 -> 2, 5 -> 3

`N` is the applied member count (config epoch, Phase 5), so placement
recomputes itself when membership changes and nothing stores it: no option, no
per-stream placement record, no startup guard, no way to drift. `R = min(3, N)`
equals `R = N` for three nodes or fewer, so no shipped test changes behavior and
only a mesh of four or more nodes sees placement at all. A single node
(N = 1, R = 1, quorum 1) keeps the shipped per-publish fsync cost byte for byte.

Consequences recorded in the plan:

- The promotion guard becomes a startup assertion: the derivation cannot produce
  a quorum below a majority of holders, so there is nothing for an operator to
  misconfigure.
- The availability table is written from the derived rule. `N = 2` keeps
  durability and loses write availability (`quorum(2) = 2`): two holders are
  enough for durability, not for failover, which is what the plan now says
  explicitly. `N >= 3` tolerates one loss for writes, two for reads.
- The default flip and its gate are gone, because there is no default to
  release; Phase 4's measurements (forwarded read latency, holder occupancy
  under foreign blocked pulls, re-placement time) become published numbers.
  The "Gates (measurements that release defaults)" section becomes
  "Measurements and the claims they back", and still picks the two tuned
  constants the design has left: the batched page size and the porcupine caps.
- Phase 5 states that `R` follows the applied member count, so a membership
  change needs no placement record, and Phase 4 states that `Holders{epoch,
  ids}` is the leader's derived statement rather than each node's own
  computation, since live views legitimately differ while a failure is detected.

Also fixes two stale references: the P6 row and Phase 1's fencing note no longer
describe forwarding or failover as gated on a measurement.
Two changes to Phase 2 and Phase 3.

`WithFlushInterval` is deleted rather than defaulted. With quorum writes the ack
waits for the fsync that covers it, so the option's documented tradeoff -- an
early ack against a crash window -- stops existing, and the only choice left is
latency against throughput, which the write path can make per load. Group commit
becomes a coalescing loop rather than a timer: the first publish with no fsync
outstanding becomes the syncer, later arrivals join its batch and wait on a
generation counter, and the next publish after it completes starts the next
batch. An idle server pays the one fsync it pays today, a busy one pays about one
fsync per batch, and no hidden interval shows up in the published latency. A
failed batch fsync fails every waiter in it. README's durable-publish row, the
Phase 9 API-change list and the Phase 9 metrics all follow.

The promotion read quorum is taken over the recorded `Holders{epoch, ids}` -- the
set that still contains the failed leader -- not over a recomputation from the
live view. Recomputing first would let the read quorum quietly stop intersecting
the write quorum, and at R = 1 it would let a node with no copy at all promote an
empty log. A dead holder is asked and does not answer; that is a failed read
quorum, which is the correct outcome, and only a serving leader records the next
epoch.
There are no deployments, no deployed stores and no older peers: the server and
its clients are built from this repository together. A protocol version bump, a
version matrix, a rolling-upgrade procedure, a downgrade path and protocol
negotiation all exist to serve a mixed-version fleet that will never exist here,
so they go.

- `streamProtocol` stays `/norobo/webstream/1.0.0`; adding a field to a command is
  just a change. Phase 11 loses its negotiation bullet and its mismatch test and
  is now the browser/WebTransport client only.
- The `stream.meta` format byte moves to 2 as a validity guard so a reader refuses
  a file it cannot parse, not as a compatibility promise, and there is no
  migration: an unknown format is refused because there is nothing to upgrade.
- Phase 9's upgrade/rollback bullet becomes "compatibility is not a phase's
  problem": a breaking change is a changelog line, and the rolling-upgrade test
  and the version-upgrade runbook are deleted.
- Phase 7 no longer asks whether the `(account, stream)` on-disk mapping is a
  migration, because there is nothing to migrate.
feat: carry a stream term and fence a superseded leader (plan.md Phase 1)
All checks were successful
code-review / Agent review (pull_request) Successful in 9s
5c04cbb9cc
Phase 1 of plan.md: a stream-scoped term, a durable term bump, and the refusal
that makes a superseded leader's write loud instead of silent.

The term
- `term uint64` is monotone, 1-based, stream-scoped, and stops one below
  MaxUint64 (the reserved invalid value), so no term field's zero value and no
  overflow can name a term a node holds.
- It lives in `stream.meta` (format 2: magic, version, term, maxAge) and on every
  command of the stream's `internal-<name>` command log, including the new `Term`
  command that announces a raise.
- `Server.persistTerm` writes the meta durably (temp, fsync, rename and an fsync
  of the stream's own directory, which the store did not do for a renamed meta)
  before any command carries the term, so the log is never ahead of the meta and
  recovery resumes at the meta's term with nothing to reconcile. `bumpTerm`
  raises the term, records it and announces it, then serves the stream under it.
- A publish reply carries the term it was served under, as `PublishAck.Term`.

Fencing
- Every incoming command passes `Server.admitTerm` first: below this node's term
  it is refused (`errStaleTermCommand`, and the replica stops applying that log,
  because the log went backwards), at it is applied, above it is recorded durably
  and then applied.
- A node that has observed a term above the one it serves the stream under is
  superseded: publish, ack, createConsumer and deleteConsumer fail with the new
  `ErrStaleTerm` (`codeStaleTerm`), and `appendCommand` refuses to write a command
  at a stale term, which covers the trim notices the byte budget publishes on its
  own. Refusals change no local state, and a publish's sequence number is not
  consumed.
- A cold stream's trim notice reads its term from the stream's meta, since the
  only path that publishes a trim for a stream nothing has loaded has no state to
  ask.
- Additive: a deployment with no failover carries term 1 everywhere and behaves
  exactly as before.

Two corrections to the plan text, applied here rather than to the code:
- The plan said the term is added to "the response of a forwarded mutation"; only
  the publish reply carries it, because that is the mutation whose reply a caller
  correlates with a sequence number. A refused mutation reports the term through
  ErrStaleTerm instead.
- The plan said recovery takes the maximum of `stream.meta` and the command log.
  Ordering the two writes (meta first) makes the log unable to be ahead, so
  recovery reads the meta and there is no maximum to take.

Tests: the term codec's round trip and its two invalid values (fuzz seeds and an
`f.Add` assertion that no accepted command carries an invalid term), the
superseded leader refusing every mutation without touching state, the exact
boundary of the apply rule, a bump surviving a restart on both nodes, a stream
without a bump keeping term 1, a term disagreement resolving to the higher term
with no forwarding loop, the ceiling refusing to wrap, an invalid term on disk
being refused, the cold-stream trim, and a rapid model of the term machine
(monotone term, supersession, refusal) against a shadow model.

Verification: `go vet ./...`, `golangci-lint run` (also with `-tags rapid`),
`go test -race ./...`, `go test -race -tags rapid ./...`, a 20s FuzzDecodeCommand
run, `gremlins unleash --dry-run` (1140 runnable, 93 not covered — unchanged in
count, all new uncovered mutants in the documented const-arithmetic and case-guard
classes), coverage 88.0% (was 87.8%), and the machine-stable benchmarks appended
to bench.log.
pi-agent changed title from feat: stream terms and fencing (Phase 1), with the plan's open questions resolved to feat: stream terms, fencing and the stale-term refusal (Phase 1) 2026-10-18 15:57:21 +00:00
pi-reviewer left a comment

AI code review — Phase 1 (stream terms, fencing, stale-term refusal)

Reviewed the full PR diff (5 commits, f0d796c..5c04cbb) via forgejo-mcp, with the diff read against surrounding context at the head ref. This is a high-quality change: the term machine is small, the durable-first ordering (meta before command) is coherent, the boundary tests are exact (at/below/above the term), and the plan text documents the Phase-3 obligations it deliberately leaves open.

Verification reproduced in this session

  • go vet ./... — clean (exit 0)
  • golangci-lint run — 0 issues
  • go test -race ./... — ok
  • go test -race -tags rapid -run TestTermProperties — ok
  • go test -run '^$' -fuzz=FuzzDecodeCommand -fuzztime=8s — 35k execs, PASS
  • go.mod/go.sum are untouched by this PR: no new dependencies, no replace directives. Module is go 1.27.1, so max (Go 1.21+), slog.Uint64 and errors.Is usage is valid.

Findings

One 🟠 IMPORTANT (latent, not reachable in Phase-1 production paths) and three 🟡 suggestions, inline. No 🔴 blocking findings: every claim in the PR body reproduced, and I found no defect that fails under Phase-1's shipped code paths.

Summary of the IMPORTANT finding: appendCommand validates only supersession (term > served), not that the command's term still equals the stream's current term at append time. The three appenders that read the term outside the stream's writeLock (publishTrim, emitConsumerUpsert, emitConsumerDelete) can therefore append a stale-term command (e.g. Trim at term 1) after a Term{2} command once a raise interleaves. Every replica that applied the raise refuses the stale command with errStaleTermCommand, and replicateLoop treats that as permanent divergence and returns; reconciliation (reconcileInterval = 2s) restarts the loop, which re-pulls the same never-acked command and dies again — a permanent restart cycle per replica until the stale entry is trimmed. Today bumpTerm has no production caller (verified: grep -rn "bumpTerm" --include="*.go" | grep -v _test.go finds only replicate.go's definition), so the race is not reachable in Phase 1 — but Phase 3's promotion is exactly the code that calls bumpTerm, and publishTrim runs every reconcile pass, so this is a landmine the plan's own Phase 3 will step on. The fix is one line: stamp the current term inside appendCommand at append time (all callers pass the same value today) instead of trusting the caller, which also serializes the term read against a concurrent raise; the alternative is holding the stream's writeLock across read+append in the emit/trim paths, as publishReplicated already does.

Also noted (body, no inline anchor): the old term_test.go unit tests for the pre-existing terminal-handler wrapper Term()/ErrTerm (TestTermWrapsError, TestTermNilYieldsErrTerm) were deleted when the file was repurposed, with no replacement. term.go is still shipped and consumer.go's terminal-error branch (errors.Is(err, ErrTerm)) still depends on its semantics; end-to-end coverage survives via the "poison" handler tests (consumer_test.go, integration_test.go), so this is a focused-unit-test loss, not a coverage hole — restoring the two small tests is cheap.

No code was modified, no commits pushed, no labels touched. Posting as COMMENT because no verified blocking issue requires REQUEST_CHANGES.

## AI code review — Phase 1 (stream terms, fencing, stale-term refusal) Reviewed the full PR diff (5 commits, f0d796c..5c04cbb) via forgejo-mcp, with the diff read against surrounding context at the head ref. This is a high-quality change: the term machine is small, the durable-first ordering (meta before command) is coherent, the boundary tests are exact (at/below/above the term), and the plan text documents the Phase-3 obligations it deliberately leaves open. ### Verification reproduced in this session - `go vet ./...` — clean (exit 0) - `golangci-lint run` — 0 issues - `go test -race ./...` — ok - `go test -race -tags rapid -run TestTermProperties` — ok - `go test -run '^$' -fuzz=FuzzDecodeCommand -fuzztime=8s` — 35k execs, PASS - go.mod/go.sum are untouched by this PR: no new dependencies, no `replace` directives. Module is `go 1.27.1`, so `max` (Go 1.21+), `slog.Uint64` and `errors.Is` usage is valid. ### Findings One 🟠 IMPORTANT (latent, not reachable in Phase-1 production paths) and three 🟡 suggestions, inline. No 🔴 blocking findings: every claim in the PR body reproduced, and I found no defect that fails under Phase-1's shipped code paths. **Summary of the IMPORTANT finding:** `appendCommand` validates only *supersession* (`term > served`), not that the command's term still equals the stream's current term at append time. The three appenders that read the term outside the stream's `writeLock` (`publishTrim`, `emitConsumerUpsert`, `emitConsumerDelete`) can therefore append a stale-term command (e.g. `Trim` at term 1) *after* a `Term{2}` command once a raise interleaves. Every replica that applied the raise refuses the stale command with `errStaleTermCommand`, and `replicateLoop` treats that as permanent divergence and returns; reconciliation (`reconcileInterval` = 2s) restarts the loop, which re-pulls the same never-acked command and dies again — a permanent restart cycle per replica until the stale entry is trimmed. Today `bumpTerm` has no production caller (verified: `grep -rn "bumpTerm" --include="*.go" | grep -v _test.go` finds only replicate.go's definition), so the race is not reachable in Phase 1 — but Phase 3's promotion is exactly the code that calls `bumpTerm`, and `publishTrim` runs every reconcile pass, so this is a landmine the plan's own Phase 3 will step on. The fix is one line: stamp the current term inside `appendCommand` at append time (all callers pass the same value today) instead of trusting the caller, which also serializes the term read against a concurrent raise; the alternative is holding the stream's `writeLock` across read+append in the emit/trim paths, as `publishReplicated` already does. **Also noted (body, no inline anchor):** the old `term_test.go` unit tests for the pre-existing terminal-handler wrapper `Term()`/`ErrTerm` (`TestTermWrapsError`, `TestTermNilYieldsErrTerm`) were deleted when the file was repurposed, with no replacement. `term.go` is still shipped and `consumer.go`'s terminal-error branch (`errors.Is(err, ErrTerm)`) still depends on its semantics; end-to-end coverage survives via the "poison" handler tests (`consumer_test.go`, `integration_test.go`), so this is a focused-unit-test loss, not a coverage hole — restoring the two small tests is cheap. No code was modified, no commits pushed, no labels touched. Posting as COMMENT because no verified blocking issue requires REQUEST_CHANGES.
@ -96,2 +105,3 @@
cmd := command{kind: kind}
term := reader.readUint64()
if !validTerm(term) {
Member

🟡 SUGGESTION — a payload truncated between the kind byte and the term field is misreported. wireReader.readUint64 (protocol.go:1138) returns 0 with reader.err = io.ErrUnexpectedEOF on a short payload; validTerm(0) is false, so this returns "command term 0" and masks the real truncation error — the exact case the comment on the kind-byte read (line 99) warns about. Fix: check reader.err after readUint64 and before the validTerm gate, e.g. if reader.err != nil { return command{}, fmt.Errorf("decoding command: %w", reader.err) }. Verified: go test -run '^$' -fuzz=FuzzDecodeCommand -fuzztime=8s passes, but the error attribution on torn reads is still wrong for debugging.

🟡 SUGGESTION — a payload truncated between the kind byte and the term field is misreported. `wireReader.readUint64` (protocol.go:1138) returns 0 with `reader.err = io.ErrUnexpectedEOF` on a short payload; `validTerm(0)` is false, so this returns "command term 0" and masks the real truncation error — the exact case the comment on the kind-byte read (line 99) warns about. Fix: check `reader.err` after `readUint64` and before the `validTerm` gate, e.g. `if reader.err != nil { return command{}, fmt.Errorf("decoding command: %w", reader.err) }`. Verified: `go test -run '^$' -fuzz=FuzzDecodeCommand -fuzztime=8s` passes, but the error attribution on torn reads is still wrong for debugging.
replicate.go Outdated
@ -699,1 +715,4 @@
func (s *Server) appendCommand(ctx context.Context, stream string, cmd command) error {
// A superseded node must not write at a stale term: every replica refuses such a command, so the
// append is dropped here. bumpTerm raises the term before it appends its own Term command.
if s.staleResident(stream) {
Member

🟠 IMPORTANT (latent) — this guard validates only supersession (term > served), not that cmd.term still equals the stream's current term at append time. publishTrim (line 1424) and emitConsumerUpsert/emitConsumerDelete (1057/1077) read the term before calling appendCommand and outside the stream's writeLock, so a raise interleaving between the read and this append lands a stale-term command (e.g. Trim at term 1) in the log after the Term{2} command. Replicas that applied the raise then refuse it with errStaleTermCommand, and the replicateLoop error handling (lines 1236–1241) treats that as permanent divergence and returns; forgetLoop drops the entry and reconciliation (reconcileInterval = 2s, line 198) restarts the loop, which re-pulls the same never-acked command and dies again — a permanent 2s restart cycle per replica until the leader's log trims the entry. Verified today: grep -rn "bumpTerm" --include="*.go" . | grep -v _test.go finds no production caller, so Phase 1 cannot trigger it; Phase 3's promotion is exactly what calls bumpTerm next to publishResidentTrims' per-pass publishTrim, so the landmine is real. Fix: derive cmd.term inside appendCommand from the resident stream at append time (every caller passes the same value today), or hold the stream's writeLock across read+append in the emit/trim paths as publishReplicated already does — serializing term adoption under writeLock also protects the shared stream.meta.tmp from concurrent persistTerm writers once promotion lands.

🟠 IMPORTANT (latent) — this guard validates only *supersession* (`term > served`), not that `cmd.term` still equals the stream's current term at append time. `publishTrim` (line 1424) and `emitConsumerUpsert`/`emitConsumerDelete` (1057/1077) read the term *before* calling `appendCommand` and outside the stream's `writeLock`, so a raise interleaving between the read and this append lands a stale-term command (e.g. `Trim` at term 1) in the log *after* the `Term{2}` command. Replicas that applied the raise then refuse it with `errStaleTermCommand`, and the `replicateLoop` error handling (lines 1236–1241) treats that as permanent divergence and returns; `forgetLoop` drops the entry and reconciliation (`reconcileInterval` = 2s, line 198) restarts the loop, which re-pulls the same never-acked command and dies again — a permanent 2s restart cycle per replica until the leader's log trims the entry. Verified today: `grep -rn "bumpTerm" --include="*.go" . | grep -v _test.go` finds no production caller, so Phase 1 cannot trigger it; Phase 3's promotion is exactly what calls `bumpTerm` next to `publishResidentTrims`' per-pass `publishTrim`, so the landmine is real. Fix: derive `cmd.term` inside `appendCommand` from the resident stream at append time (every caller passes the same value today), or hold the stream's `writeLock` across read+append in the emit/trim paths as `publishReplicated` already does — serializing term adoption under `writeLock` also protects the shared `stream.meta.tmp` from concurrent `persistTerm` writers once promotion lands.
replicate.go Outdated
@ -819,0 +869,4 @@
}
defer s.release(name)
if state.termValue() < term {
Member

🟡 SUGGESTION — dead branch: in the replication path admitTerm (line 881) runs first and already persists any term above the node's own (term > current → persistTerm), so when applyTerm runs state.termValue() < term is never true and this persist is unreachable; the function reduces to a second acquire+release per Term command. The only caller that exercises the persist branch is the test at term_test.go:131. Either fold the term-command handling into admitTerm and drop applyTerm, or document why the double-check exists. Verified by reading replicate.go 881–905 (admitTerm's default branch) and the call sequence at 1213–1215.

🟡 SUGGESTION — dead branch: in the replication path `admitTerm` (line 881) runs first and already persists any term above the node's own (`term > current` → `persistTerm`), so when `applyTerm` runs `state.termValue() < term` is never true and this persist is unreachable; the function reduces to a second acquire+release per `Term` command. The only caller that exercises the persist branch is the test at term_test.go:131. Either fold the term-command handling into `admitTerm` and drop `applyTerm`, or document why the double-check exists. Verified by reading replicate.go 881–905 (admitTerm's default branch) and the call sequence at 1213–1215.
Resolves the review on PR #15.

IMPORTANT — the term is now stamped inside the append, under the stream's
writeLock, instead of being passed by the caller. `publishTrim`,
`emitConsumerUpsert` and `emitConsumerDelete` read `state.termValue()` before
calling `appendCommand` and outside any lock, so a raise interleaving between the
read and the append landed a stale-term command *after* the `Term{t}` command. A
replica that had applied the raise refused it with `errStaleTermCommand`, which
`replicateLoop` treats as permanent divergence, so it stopped and reconciliation
restarted it every `reconcileInterval` — a permanent 2s cycle per replica until
the entry was trimmed. Unreachable in Phase 1 (`bumpTerm` has no production
caller yet), but Phase 3's promotion is what calls it next to a trim published
every pass.

- `appendCommand(ctx, state, cmd)` takes the stream's writeLock, checks
  supersession, stamps `cmd.term = state.termValue()` and appends;
  `appendCommandLocked` is the same step for the publish pipeline, which already
  holds that lock across reserving the sequence number, writing the command and
  applying the record. `appendToCommandLog` appends an already-stamped command and
  is used only by the cold-stream trim path, whose term can only come from its
  meta (nothing has the stream loaded, and a term cannot be raised on a stream
  that is not loaded).
- `persistTerm` takes the same writeLock (with a `...Locked` variant for
  `bumpTerm`), so two writers of one `stream.meta` cannot interleave — the
  reviewer's second point about the shared `stream.meta.tmp` once promotion lands.
- The Term command is stamped like any other command: `persistTerm` raises the
  stream's term before the announcement is appended, so the stamp is the new term.

SUGGESTION — `applyTerm` dropped. `admitTerm` runs first for every applied command
and already persists any term above the node's own, so `applyTerm`'s
`state.termValue() < term` was unreachable and the function was a second
acquire/release per Term command. Admitting a Term command *is* applying it, and
the apply switch now says so.

SUGGESTION — `decodeCommand` checks `reader.err` after the term field and before
the term validation: a payload cut inside the term reads as zero, and reporting
"command term 0" hid the truncation behind an invalid-term refusal. Covered by
TestDecodeCommandReportsATruncatedTerm.

Also, from the review body: `term_test.go` was overwritten by this change's first
commit — a filename collision, not an intentional deletion — which lost
TestTermWrapsError and TestTermNilYieldsErrTerm for `term.go`'s `Term`/`ErrTerm`.
The file is restored verbatim and the stream-term tests moved to
stream_term_test.go / stream_term_property_test.go. Added
TestTrimNoticeCarriesTheTermInForce for the stamping rule, and folded the
command-log retry's error check into one shared branch, which removes an
uncovered mutant and makes it killable.

Verification: go vet clean, golangci-lint 0 issues (also -tags rapid), go test
-race ok, -tags rapid ok, FuzzDecodeCommand 15s ok, coverage 88.0% statements,
gremlins dry run 1137 runnable / 92 not covered / 92.51% (the recorded baseline
was 1098/93/92.19%).
Author
Member

Resolved in 76d4d8a (pushed). One piece of bookkeeping first: you were right that term_test.go's two Term/ErrTerm tests vanished, and it was worse than a repurpose — I wrote the new stream-term tests over term_test.go by filename collision, so TestTermWrapsError and TestTermNilYieldsErrTerm were deleted outright. term_test.go is restored verbatim from the pre-change tree and the stream-term tests live in stream_term_test.go / stream_term_property_test.go now. Thank you for catching that; a coverage log line records it.

🟠 IMPORTANT — stale-term append races a raise

Fixed as you suggested (stamp at append time), with the lock discipline made explicit rather than left to callers:

  • appendCommand(ctx, state, cmd) takes the stream's writeLock, checks supersession, stamps cmd.term = state.termValue(), then appends. publishTrim, emitConsumerUpsert and emitConsumerDelete no longer read or pass a term at all — the bug class is unrepresentable rather than merely fixed, since nothing outside appendCommand can supply a term.
  • appendCommandLocked is that same step for publishReplicated, which already holds writeLock across reserving the sequence number, writing the command and applying the record. appendToCommandLog appends an already-stamped command and has exactly one caller: the cold-stream trim path, where the term can only come from the meta because nothing has the stream loaded — and a term cannot be raised on a stream that is not loaded, so there is nothing to race.
  • The Term command is stamped like any other: persistTerm raises the term before the announcement is appended, so the stamp is the new term. TestTrimNoticeCarriesTheTermInForce pins that.
  • Your second point about stream.meta.tmp: persistTerm now takes the same writeLock (with a persistTermLocked variant for bumpTerm, which holds it), so two writers of one stream.meta cannot interleave once promotion lands.

🟡 SUGGESTION — applyTerm's unreachable persist

Dropped, as you suggested. admitTerm persists any term above the node's own before the apply switch runs, so admitting a Term command is applying it; the switch case now says exactly that and the extra acquire/release per Term command is gone. TestSupersededLeaderRefusesMutations drove applyTerm and now drives admitTerm, which is the call the apply loop actually makes.

🟡 SUGGESTION — truncated payload misreported as an invalid term

Fixed: reader.err is checked after the term read and before validTerm, so a payload cut inside the term field reports the truncation instead of "command term 0". TestDecodeCommandReportsATruncatedTerm covers both cut points (at the term's first byte and one byte short of it).

Verification after the fix

go vet ./...                                     clean
golangci-lint run                               0 issues (also -tags rapid)
go test -race ./...                             ok
go test -race -tags rapid ./...                  ok
go test -fuzz=FuzzDecodeCommand -fuzztime=15s    66k execs, no failure
gremlins unleash --dry-run                      1137 runnable, 92 not covered, 92.51%

Coverage is 88.0% statements. Relevant to your finding's blast radius, the mutation numbers moved the right way: the recorded baseline was 1098/93/92.19%, and folding appendToCommandLog's retried-acquire error check into one shared branch removed an uncovered mutant and made that check killable, so this is now 92 not covered and 92.51%. TESTING_PLAN.md and coverage.log carry the new figures.

One thing I deliberately did not change, for the record: the replica still treats a command below its term as permanent divergence and stops (errStaleTermCommand), rather than skipping it or resyncing. With the producer gone that state is unreachable by construction, and skipping a command would diverge a copy silently; the loud stop plus the comment on the sentinel is the outcome I want if it ever does appear.

Resolved in `76d4d8a` (pushed). One piece of bookkeeping first: you were right that `term_test.go`'s two `Term`/`ErrTerm` tests vanished, and it was worse than a repurpose — I wrote the new stream-term tests **over** `term_test.go` by filename collision, so `TestTermWrapsError` and `TestTermNilYieldsErrTerm` were deleted outright. `term_test.go` is restored verbatim from the pre-change tree and the stream-term tests live in `stream_term_test.go` / `stream_term_property_test.go` now. Thank you for catching that; a coverage log line records it. ### 🟠 IMPORTANT — stale-term append races a raise Fixed as you suggested (stamp at append time), with the lock discipline made explicit rather than left to callers: - `appendCommand(ctx, state, cmd)` takes the stream's `writeLock`, checks supersession, stamps `cmd.term = state.termValue()`, then appends. `publishTrim`, `emitConsumerUpsert` and `emitConsumerDelete` no longer read or pass a term at all — the bug class is unrepresentable rather than merely fixed, since nothing outside `appendCommand` can supply a term. - `appendCommandLocked` is that same step for `publishReplicated`, which already holds `writeLock` across reserving the sequence number, writing the command and applying the record. `appendToCommandLog` appends an already-stamped command and has exactly one caller: the cold-stream trim path, where the term can only come from the meta because nothing has the stream loaded — and a term cannot be raised on a stream that is not loaded, so there is nothing to race. - The Term command is stamped like any other: `persistTerm` raises the term *before* the announcement is appended, so the stamp is the new term. `TestTrimNoticeCarriesTheTermInForce` pins that. - Your second point about `stream.meta.tmp`: `persistTerm` now takes the same `writeLock` (with a `persistTermLocked` variant for `bumpTerm`, which holds it), so two writers of one `stream.meta` cannot interleave once promotion lands. ### 🟡 SUGGESTION — `applyTerm`'s unreachable persist Dropped, as you suggested. `admitTerm` persists any term above the node's own before the apply switch runs, so admitting a Term command *is* applying it; the switch case now says exactly that and the extra acquire/release per Term command is gone. `TestSupersededLeaderRefusesMutations` drove `applyTerm` and now drives `admitTerm`, which is the call the apply loop actually makes. ### 🟡 SUGGESTION — truncated payload misreported as an invalid term Fixed: `reader.err` is checked after the term read and before `validTerm`, so a payload cut inside the term field reports the truncation instead of "command term 0". `TestDecodeCommandReportsATruncatedTerm` covers both cut points (at the term's first byte and one byte short of it). ### Verification after the fix ``` go vet ./... clean golangci-lint run 0 issues (also -tags rapid) go test -race ./... ok go test -race -tags rapid ./... ok go test -fuzz=FuzzDecodeCommand -fuzztime=15s 66k execs, no failure gremlins unleash --dry-run 1137 runnable, 92 not covered, 92.51% ``` Coverage is 88.0% statements. Relevant to your finding's blast radius, the mutation numbers moved the right way: the recorded baseline was 1098/93/92.19%, and folding `appendToCommandLog`'s retried-acquire error check into one shared branch removed an uncovered mutant and made that check killable, so this is now 92 not covered and 92.51%. `TESTING_PLAN.md` and `coverage.log` carry the new figures. One thing I deliberately did **not** change, for the record: the replica still treats a command below its term as permanent divergence and stops (`errStaleTermCommand`), rather than skipping it or resyncing. With the producer gone that state is unreachable by construction, and skipping a command would diverge a copy silently; the loud stop plus the comment on the sentinel is the outcome I want if it ever does appear.
sam merged commit 8f4f6ad6cd into main 2026-10-18 16:35:18 +00:00
sam deleted branch feat/phase-1-terms-fencing 2026-10-18 16:35:18 +00:00
Sign in to join this conversation.
No reviewers
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
norobo/webstream!15
No description provided.