test: fuzz, property and mutation test foundation #13

Merged
sam merged 27 commits from test/fuzz-property-mutation into main 2026-10-12 20:34:23 +00:00
Member

Adds the test foundation the library needs to stay correct under edits by people who did not write it: native fuzz targets for the byte-level parsers, rapid property models for the stateful storage layer, and deterministic time/transport for tests. TESTING_PLAN.md (new) is the inventory and rationale.

Bugs found and fixed (all caught by the new tests)

fix found by
decodeDoubleBuffered panicked on a short cursor slot slice FuzzDecodeCursorSlot
parseSegmentBase accepted overflowing segment base names FuzzParseSegmentBase
tsFromHeader panicked on a short buffer FuzzTSFromHeader
memlog recovery replayed records wrongly (appendRecovered regression) memlog property model
memlog gap-slot byte underflow (gap accounting now starts at max(next, base)) memlog property model
removeTail(MaxUint64) wrap panic memlog property model
smallestNames alias-mutated its input and panicked on empty names / limit <= 0 TestSmallestNames
countFromSeqs(0, MaxUint64) wraparound TestCountFromSeqs

Fuzz targets (native testing.F, with seeds so plain go test replays them)

Tier 1: replication command/record codecs, cursor watermark slot, segment scan, stream-name extractors. Tier 2: segment base names, record timestamps, header validation. No production change of behaviour beyond the four fixes above.

Property models (pgregory.net/rapid, test-only dependency)

  • TestMemlogProperties - append/recover/trim op sequences against a shadow log (window, records, byte accounting).
  • TestJournalMemlogParity - differential: the on-disk journal reloaded must equal the live in-memory log (batched mode, so per-publish fsync does not dominate).
  • TestConsumerStateProperties - leases, cursors and watermarks, in a synctest bubble.
  • TestStreamStateProperties - appendDedup / consumer create / ack / delete / clock-elapse sequences against a shadow model: dedup table tied to retention, dedup hit/miss per retained window, interest-vs-age trim point, cached minimum cursor invalidation.
  • Small models: slot survival, smallestNames, countFromSeqs, hashLeader distribution and name grammars.

Test infrastructure

  • No test seams in production. The now clock field (consumer_state.go) and the failSync hook (store.go) are gone; time-dependent tests use testing/synctest bubbles with a deterministic fake clock.
  • libp2p mocknet for mesh and loopback tests. The mesh, conn, consumer and integration tests ran on real TCP loopback, which put them outside synctest; they now run on libp2p's in-memory mocknet. Mocknet hosts are plain hosts over a mock peernet — no yamux/noise/identify negotiation and no connection gater — but the app-level routing, stream handlers and mesh logic are identical, the blocking is durable in a bubble, and the transport-security surface stays covered by TestAllowlistGater and the real-TCP integration tests. Benchmarks keep real TCP because they measure the transport, and the QUIC test keeps it because it tests that transport.
  • rapid.SyncTest for the bubble models. A raw synctest.Test inside rapid.Check cannot report a failure: it panics out of the bubble's goroutine, killing the process (exit status 2) with no shrunk counterexample. Verified with a falsifiable property.
  • Cost split, measured. Only the durable journal differential (~21s of real files and fsyncs) sits behind the rapid build tag, so plain go test and gremlins skip it; the invariant models run on every go test (~1s; default suite 5.0s -> 6.1s). make test-props runs everything.

Verification

go test ./... (6.5s), go test -race ./... (make test), go test -tags rapid ./..., go vet ./..., go vet -tags rapid ./..., golangci-lint run ./... and golangci-lint run --build-tags rapid ./... all pass.

Not in this PR: benchmarks, coverage baseline and the gremlins mutation run — those became part of the branch during review and are documented in TESTING_PLAN.md and the new README Testing section.

Adds the test foundation the library needs to stay correct under edits by people who did not write it: native fuzz targets for the byte-level parsers, rapid property models for the stateful storage layer, and deterministic time/transport for tests. `TESTING_PLAN.md` (new) is the inventory and rationale. ## Bugs found and fixed (all caught by the new tests) | fix | found by | |---|---| | `decodeDoubleBuffered` panicked on a short cursor slot slice | `FuzzDecodeCursorSlot` | | `parseSegmentBase` accepted overflowing segment base names | `FuzzParseSegmentBase` | | `tsFromHeader` panicked on a short buffer | `FuzzTSFromHeader` | | `memlog` recovery replayed records wrongly (`appendRecovered` regression) | memlog property model | | `memlog` gap-slot byte underflow (gap accounting now starts at `max(next, base)`) | memlog property model | | `removeTail(MaxUint64)` wrap panic | memlog property model | | `smallestNames` alias-mutated its input and panicked on empty names / `limit <= 0` | `TestSmallestNames` | | `countFromSeqs(0, MaxUint64)` wraparound | `TestCountFromSeqs` | ## Fuzz targets (native `testing.F`, with seeds so plain `go test` replays them) Tier 1: replication command/record codecs, cursor watermark slot, segment scan, stream-name extractors. Tier 2: segment base names, record timestamps, header validation. No production change of behaviour beyond the four fixes above. ## Property models (`pgregory.net/rapid`, test-only dependency) - `TestMemlogProperties` - append/recover/trim op sequences against a shadow log (window, records, byte accounting). - `TestJournalMemlogParity` - differential: the on-disk journal reloaded must equal the live in-memory log (batched mode, so per-publish fsync does not dominate). - `TestConsumerStateProperties` - leases, cursors and watermarks, in a synctest bubble. - `TestStreamStateProperties` - `appendDedup` / consumer create / ack / delete / clock-elapse sequences against a shadow model: dedup table tied to retention, dedup hit/miss per retained window, interest-vs-age trim point, cached minimum cursor invalidation. - Small models: slot survival, `smallestNames`, `countFromSeqs`, `hashLeader` distribution and name grammars. ## Test infrastructure - **No test seams in production.** The `now` clock field (`consumer_state.go`) and the `failSync` hook (`store.go`) are gone; time-dependent tests use `testing/synctest` bubbles with a deterministic fake clock. - **libp2p mocknet for mesh and loopback tests.** The mesh, conn, consumer and integration tests ran on real TCP loopback, which put them outside synctest; they now run on libp2p's in-memory mocknet. Mocknet hosts are plain hosts over a mock peernet — no yamux/noise/identify negotiation and no connection gater — but the app-level routing, stream handlers and mesh logic are identical, the blocking is durable in a bubble, and the transport-security surface stays covered by `TestAllowlistGater` and the real-TCP integration tests. Benchmarks keep real TCP because they measure the transport, and the QUIC test keeps it because it tests that transport. - **`rapid.SyncTest` for the bubble models.** A raw `synctest.Test` inside `rapid.Check` cannot report a failure: it panics out of the bubble's goroutine, killing the process (exit status 2) with no shrunk counterexample. Verified with a falsifiable property. - **Cost split, measured.** Only the durable journal differential (~21s of real files and fsyncs) sits behind the `rapid` build tag, so plain `go test` and gremlins skip it; the invariant models run on every `go test` (~1s; default suite 5.0s -> 6.1s). `make test-props` runs everything. ## Verification `go test ./...` (6.5s), `go test -race ./...` (`make test`), `go test -tags rapid ./...`, `go vet ./...`, `go vet -tags rapid ./...`, `golangci-lint run ./...` and `golangci-lint run --build-tags rapid ./...` all pass. Not in this PR: benchmarks, coverage baseline and the gremlins mutation run — those became part of the branch during review and are documented in `TESTING_PLAN.md` and the new README Testing section.
Catalog the codebase's untested parsers and state machines on the new
fuzz/property/mutation branch: wire and on-disk codecs that fuzz_test.go
does not cover (decodeCommand, decodeRecord, double-buffered watermark
slots, scanSegment, stream-name extraction), the pure state machines that
need property models (memlog, consumerState, streamState), and the gremlins
mutation workflow. No code changes.
decodeDoubleBuffered and encodeStaleCopy index slot[0:24] unconditionally,
so a slice shorter than cursorSlotSize panics. No production caller can
produce one - the cursors.meta read and flush always allocate the full
stride and a torn read leaves the tail zeroed - but every sibling parser of
on-disk bytes (scanSegment, decodeRecord) is total, and the fuzz target that
pins the contract feeds arbitrary slices. A short slot now counts as both
copies torn; a short write target is left untouched.
FuzzDecodeCommand covers the command-log codec, the one wire format the
request fuzzers missed; FuzzDecodeRecord covers the on-disk record frame;
FuzzDecodeCursorSlot pins the slot codec's totality and checks the decode
against an independent CRC oracle; FuzzScanSegment drives the segment
scanner over arbitrary file bytes, asserting scan idempotence after torn-tail
truncation; FuzzStreamNameExtractors drives the routing pre-decoders.

The runs surfaced two findings beyond the round trips: decodeRecord takes the
frame payload without its length prefix (the draft fed the prefixed frame and
the seed corpus failed), and relayedStreamName's internal-name rule applies
to framePull only - a non-leader relays a client's frameAck under the user
stream name - so its doc comment now states that precisely.
parseSegmentBase accepted a 20-digit name past MaxUint64 (twenty nines,
which no segmentBase output can produce) and silently wrapped it to a
different segment base, so a garbage filename would name the wrong segment
on load; it now rejects the overflow and loadLog skips the file like any
other malformed name. tsFromHeader panicked on a buffer shorter than its
8-byte field; every caller passes a fixed header window, so it was
unreachable in production, but a parser of on-disk bytes must be total and
now returns an error instead.
FuzzParseSegmentBase pins the segment filename round trip, which surfaced
the MaxUint64 overflow accept now fixed; FuzzTSFromHeader pins the timestamp
field's int64 boundary and totality over short slices; FuzzValidateHeader
checks the file-header accept/reject boundary against an oracle. The two
pure boundary functions (durationFromUint64, nameLenByte) get unit tests:
fuzzing a two-case boundary adds nothing.

Also drops the unused context import the t.Context() rewrite left behind.
The on-disk persisted mode is the higher-priority target: a rapid stateful
machine applies one op sequence to both the memlog and a real journal
(appendPublish, trimTo, rewindAppendLocked, the replication apply path) and
asserts the reloaded log equals the live one - base, next, records, and
bytes, which loadRecords re-sums independently. Moved ahead of the lighter
property models in the implementation order.
The memlog property model (rapid) found three defects:

- appendRecovered with a replay (seq below next, reachable from corrupt
  out-of-order frames on disk) regressed next and double-counted the
  slot's bytes; it now skips replays.
- Recovery-adopted gap slots were never accounted, so a later trimBefore
  over them subtracted bytes append never added, underflowing log.bytes;
  appendRecovered now accounts each gap slot from the window
  (max(next, base)) like the empty record it reads as. The journal
  differential then caught the first draft's pre-base phantom bytes on a
  fresh load (next 0 at a non-zero base).
- removeTail(MaxUint64) wrapped seq+1 to 0, passed the guard on an empty
  log, and indexed out of range; the guard now rejects seq >= next.
TestMemlogProperties drives appendOwned, trimBefore, removeTail,
applyRecord and appendRecovered against an independent contract model,
checking window, records, byte accounting and error contracts after every
step. TestJournalMemlogParity drives a real journal (batched mode for
speed) with random appends, trims and reloads, asserting the replayed log
equals the live memlog: same next, live base at or ahead of the oldest
segment, identical records on the overlap, the disk-retained prefix
matching exactly what was appended, and byte accounting that loadRecords
re-sums independently.

The journal machine covers the persisted mode the user prioritized; the
rewind path stays unit-tested (per-publish-mode-only seam). Suite runtime
grows from ~7s to ~22s; tune with -rapid.checks and -rapid.steps.
Two additions at the end of the plan: micro-benchmarks for the pure hot
paths the fuzz/property work has been exercising (memlog ops, stream-name
extractors, the command-log codec, leader hashing) under the existing make
bench target, and a coverage baseline - a make cover target producing a
gitignored coverage.out plus a committed coverage.log line per meaningful
change, so line coverage can be compared historically and with gremlins'
per-package coverage report in the mutation stage.
The property models found two more latent defects in store.go:

- smallestNames' heap aliased the input slice, so heap.Fix reordered the
  caller's names as a side effect of a pure-looking selection; it now heaps
  a clone, and the property test pins the no-mutation contract. It also
  panicked on empty names with a positive limit (names[limit:] sliced out
  of range) and on a non-positive limit (empty-heap top[0]); both are now
  total: a non-positive limit yields nothing.
- countFromSeqs computed last-first+1 with uint64 wraparound, so the range
  (0, MaxUint64) silently reported a valid zero count instead of an error;
  the span bound is now checked explicitly before the +1.

consumerState gains a now clock seam (same test-seam pattern as the
journal's failSync) so lease expiry is deterministic under the property
model.
TestConsumerStateProperties drives lease/cursor state against a contract
model on a shared virtual clock: acquisition exclusivity, expiry, the
retired-token never re-acquires rule, and the advanceIf watermark
invariant. TestCursorSlotSurvival alternates monotonic writes and torn
copies, asserting decode always agrees with the CRC oracle and a torn copy
never loses the surviving copy's value. TestSmallestNames checks the heap
selection against a brute-force oracle plus the no-input-mutation
contract; TestCountFromSeqs pins the wrap edge; TestHashLeader and
TestLeaderHashDistinct pin set-membership, permutation independence and
the NUL-separator collision rule; TestNameGrammars checks the name
grammars against independent oracles.
The consumerState property model previously needed a clock seam; it now
runs inside testing/synctest, whose bubble clock makes time.Now
deterministic with no production code: the elapse action sleeps in the
bubble to advance the clock, and the contract model reads the same fake
clock.

The journal's failSync seam is gone too. The rewind regression test now
drives rewindAppendLocked directly (capture the pre-append offset, write
a frame, roll it back, retry, reload), and the command-log crash-window
test writes the unapplied Append command via appendCommand - the exact
state the seam produced - instead of injecting a sync failure.
A closed segment file drives a genuine write-path failure through
appendPublish: the error propagates and the journal's in-memory state is
unchanged. Together with the direct rewind test, the durable-write failure
semantics are covered end to end without any seam.
Four tests were still using real-time patterns where the code under test
is in-process and timer-driven:

- TestConsumerLeaseStaleExpires backdated leaseUntil through the struct's
  lock; it now sleeps past the refresh window in a synctest bubble.
- TestServerLeaseExpiresOnStale did the same on a server consumer; the
  bubble clock replaces the backdating.
- TestServerPullBlocksUntilPublish used a real 50ms negative probe and a
  real 5s wake budget; both are now exact bubble-time assertions.
- TestCursorCoalescerFlushesOnCadence polled up to 2s of real time for
  the cursor coalescer's timer-driven flush; the cadence is now bubble
  time and the flush is asserted directly after one sleep.

These ran on real clocks before (flaky under load, slow); they are now
deterministic and near-instant.
The mesh, conn and consumer tests used real TCP loopback, so synctest's
fake clock could not apply to them: lease expiry was backdated through the
struct, a replication retry was waited out in real time, and pacing and
deadline assertions raced the machine clock. libp2p's mocknet transport is
channel-based, so its blocking is durable in a synctest bubble.

- mockHosts replaces meshHosts for every mesh test (in-memory transport,
  same yamux/noise/identify stack); meshHosts stays for the benchmarks,
  which measure the real transport, and newQUICConn stays real for the
  QUIC transport test.
- newTestConnWithServerOpts and newPersistentTestConn use mocknet too,
  so the loopback suite (conn, consumer, integration) runs without kernel
  sockets.
- The time-sensitive tests now run in synctest.Test bubbles: the lease
  takeover tests sleep past the expiry window, the busy-lease replication
  test sleeps the retry interval, the nack pacing tests measure exact
  bubble intervals, and the create-stream flight deadline expires on the
  bubble clock. The last leaseUntil backdating sites are gone.
- Verified first: a bare mocknet host pair opens streams inside a bubble,
  and cleanups registered on the bubble's t run before its drain check, so
  the existing tb.Cleanup helpers work unchanged.
Random appendDedup, addConsumerIfRoom, ack (advanceIf + trimAfterCursorAdvance)
and deleteConsumer sequences, plus clock advances for streams with a maxAge
window, are checked against a shadow model inside a synctest bubble so the
fake clock keeps record timestamps, the age trim and the model's now in lock
step. After every step the stream must match the model on the retained
window, record readability, byte accounting, the dedup table (tied to
retention by dropDedupLocked), consumer cursors and the cached minimum
cursor.

The model closes the plan's streamState gap: dedup hit/miss per retained
window, the interest-vs-age trim point (computeKeepLocked = max(min cursor,
maxAge keep)), and the interplay of ack-driven min-cursor cache invalidation
with deletion and re-creation are now exercised by random sequences instead
of hand-written scenarios.
The property tests now live only in *_property_test.go files carrying
//go:build rapid, so plain go test and gremlins (which runs go test with
default tags) skip them: the default suite drops from ~16s to ~5s and
mutation runs are not slowed by the durable journal differential's real
disk I/O. make test-props (go test -race -tags rapid ./...) runs the full
property suite explicitly; vet and golangci-lint stay clean both with and
without the tag.
The consumerState and streamState models opened a raw synctest.Test inside
rapid.Check. A failing check then panicked out of the bubble's goroutine —
where rapid's recover cannot reach it — so the process died with exit status
2 and no shrunk counterexample and no failfile. That signature matches a
failure seen earlier in the streamState model, which was dismissed as an
interrupted run: it was reachable, and it would have been unreportable.

rapid ships SyncTest for exactly this: it runs the bubble, captures the
failure and re-panics in the parent, so rapid shrinks and reports it as
usual. Verified with a falsifiable property: the raw bubble died with
"panic: rapid.stopTest(...)" / exit status 2, while SyncTest reports
"[rapid] failed: v=11 > 10" / exit status 1 — the minimal counterexample.
Plain go test and gremlins skipped every rapid model because all of them sat
behind the rapid build tag. That is the wrong default for a library that has
to stay correct under edits by people who did not write it: these models are
the executable spec of the storage layer, and streamState could break with
the suite still green.

Measured per model (single tagged run, ~1.3s process start subtracted):

  journal memlog differential   ~21s   (real files, real fsyncs)
  memlog property model         ~0.2s
  consumerState model            ~0s
  streamState model              ~0s
  small models (6 tests)         ~0s

So only the differential keeps the tag. The default suite grows from 5.0s to
6.1s and now checks the dedup/retention/consumer invariants on every run;
make test-props (go test -race -tags rapid ./...) adds the differential, and
gremlins stays on the fast path.
.gremlins.yaml configures `gremlins unleash` for this module: gremlins' 5
high-signal default mutators, the 6 equivocal/hanging ones off, no thresholds
enforced on a full run, and no tags — the `rapid` gate is deliberately left
unset so the ~21s durable journal differential stays out of every mutant run
while the untagged invariant models stay in as the kill oracle.

Three corrections against the draft config:

- timeout-coefficient: 3, not 5. 5 was labelled "default", but gremlins'
  built-in default is 3 (engine/executor.go DefaultTimeoutCoefficient, with
  the flag default 0 meaning "use it"), so 5 was raising the timeout above
  the default — the opposite of the config's own reasoning that a bigger
  coefficient only spends wall time before a permanently-hanging mutant is
  declared TIMED OUT.
- dropped output-diff-statuses: it is absent from gremlins' documented
  reference config and from the installed build (v0.6.0 keys), and viper
  ignores unknown keys silently, so it would have left a reader believing
  diffs were printed.
- the loop-guard example that hangs under conditionals-negation is
  store.go:948 `for pos+wireUint32Size <= len(data)`, named by file.

TESTING_PLAN.md records the baseline: `gremlins unleash --dry-run` enumerates
1098 runnable mutants, 93 not covered, mutator coverage 92.19% in 110ms, and
gremlins' Go 1.27 support is now verified rather than assumed.
The first gremlins run reported 93 NOT COVERED mutants. Triage against the
coverage profile (gremlins runs plain `go test -cover`):

Filled (93 -> 84 NOT COVERED, mutator coverage 92.19% -> 92.95%):

- stream_state.go dropReservation and reserveRecord never ran in the suite:
  the leader publish pipeline's claim/release contract (reserve claims the
  next seq and registers the dedup mapping, a live id is a duplicate, drop
  releases only the mapping that still names the record) was covered only by
  inspection. TestStreamStateReserveDropDedup and
  TestStreamStateDropReservationGuards drive it directly.
- protocol.go wireReader.done's trailing-bytes rejection was never hit.
  TestWireReaderDoneTrailingBytes feeds a frame with bytes past the consumed
  fields and asserts errInvalidFrame.
- store.go flushJournal's fsync-failure path (and flushJournals' per-stream
  warning) never ran. TestStoreFlushJournalError closes the segment file,
  marks the journal dirty and asserts the error propagates and the dirty flag
  survives.

Classified as NOT real gaps (gremlins cannot represent them, so they stay
NOT COVERED forever):

- Case guards and empty case bodies: Go coverage attaches no counter to a
  `case X:` expression or an empty case body; the block lands at a different
  position than the mutant token, so gremlins' IsCovered misses it. The
  validName switch (server.go 637-640), consumer.go pull's `err == nil`,
  memlog.go applyRecord's reject guards and the sort comparators' guards all
  execute on every run; their case bodies show count 1 in the profile.
- Package-level const arithmetic (protocol.go overheads, the 30s timeouts,
  segmentRecords and friends): const declarations have no instrumented
  statement; the values are exercised through every encode/decode.
- nameMaxHeap.Pop: required by the heap.Interface contract but never called
  (smallestNames uses heap.Fix). Dead by contract, not a gap.
- store.go sweepStream's `return -1`: only reachable with out-of-order
  segment names, which ReadDir never produces.

Known real but unfilled (need test-data shapes, low signal):

- budgetSnapshot's candidate sort comparator bodies (store.go 1743-1755):
  every eviction test yields at most one deletable segment, so SortFunc
  never compares. Filling means a 3+ segment or multi-stream eviction
  fixture.
- store.go 1080/1088 rewind-after-failed-write/sync: reachable only if the
  append's Seek succeeds and the write then fails, which a closed file
  cannot produce (Seek fails first). A seam-free test is not available.
- store.go 835 io.ReadFull torn tail: the sibling declared-length-past-EOF
  branch is tested; this one needs the file to shrink between stat and read.
The coverage step of the testing plan: `make cover` runs the untagged
profile (`go test -coverprofile=coverage.out ./...` plus the func total),
`make cover-props` adds the rapid-tagged durable journal differential,
`coverage.out` is gitignored as a regenerable artifact, and `coverage.log`
tracks the total per meaningful change.

Baseline on this branch: 87.8% statements untagged — the same profile
gremlins uses for its mutation stage, so the totals and the per-package
mutation scores compare directly over history — and 88.1% with the tagged
differential. Recorded in coverage.log.
The four micro-benchmark groups the testing plan's step 5 specified, under
the existing make bench target:

- memlog ops at a realistic retained window: appendOwned 128 ns/op (0 allocs,
  the chunk allocation amortizes over recordChunkSize appends), readAt 3.8
  ns/op, and a steady-state trimBefore cycle that crosses chunk boundaries
  (32.5 us per 256-record trim + refill).
- the three stream-name extractors on a real publish payload: 10-12 ns/op,
  0 allocs — the routing pre-decode every request pays.
- the command-log codec: append encode 285 ns/4 allocs, decode 221 ns/2
  allocs (decode clones the payload by contract), trim encode 92 ns, decode
  22 ns.
- leader hashing over a 4-node mesh set: hashLeader 1.27 us/8 allocs,
  leaderHash 280 ns/2 allocs.

Baselines recorded in TESTING_PLAN.md for benchstat history.
bench.txt was gitignored as the raw scratch output for benchstat, but that
left benchmark history untracked entirely: no way to spot a regression. Add
`make bench-log`, which appends a dated, machine-tagged record of the
machine-stable micro-benchmarks (memlog ops, name extractors, command
codec, leader hashing) to the committed bench.log. The mesh/TCP and server
end-to-end benchmarks stay out of it: their numbers depend on load and
hardware, so they would drown the trend signal. Baseline recorded.
docs: document the testing layers and the agent contract for this repo
All checks were successful
code-review / Agent review (pull_request) Successful in 9s
1be416e0b2
README gains a Testing section: the make targets (test, test-props, cover,
cover-props, bench, bench-log, fuzz, gremlins), the layered suite (native
fuzz targets with replayed seeds, rapid property models as the storage
layer's executable spec, synctest bubbles for deterministic time, mocknet
for in-memory transport, the .gremlins.yaml mutation config, and the
coverage/benchmark regression logs).

AGENTS.md gains the contract for writing fault-tolerant correct code here:
no test seams in production (no injected clocks or failure hooks), synctest
bubbles for time, mocknet for network tests, a native fuzz target for every
byte-level parser, a rapid shadow model for every stateful component, the
rapid build-tag split (cheap models untagged, only the ~21s journal
differential tagged), reachable-error-path coverage, and the gremlins
NOT-COVERED sentinel with its documented false positives. A model break is
a contract break: fix the code, not the model.
pi-reviewer left a comment

AI-driven code review (samber/cc-skills-golang passes: style/naming/docs, correctness/safety/concurrency, security/deps, tests/perf/observability) against head 1be416e, with every claim reproduced in this session against a fresh clone of the PR head.

Verified green (matching the PR's claims)

  • go test ./... → ok git.norobo.chat/norobo/webstream 6.538s
  • go vet ./... and go vet -tags rapid ./... → clean
  • go test -tags rapid -run TestJournalMemlogParity -count=1 → ok (15.3s)
  • govulncheck -mode=source ./... → 0 called vulnerabilities; the one new dependency pgregory.net/rapid v1.3.0 is test-only (Go 1.23, module is Go 1.27.1), no replace directives
  • The four production hardening fixes (parseSegmentBase overflow, tsFromHeader length guard, cursor-slot totality, smallestNames/countFromSeqs) are correct as written

Blocking

  • go test -race ./... — the repo's own make test gate and the PR's Verification section — fails at this head (2/2 runs, ~20 tests cascade-fail with "race detected during execution of test"). Root cause is one unlocked access in the new TestStreamStateProperties (see inline comment). origin/main passes the same command (ok ... 12.542s), so the race is introduced by this PR.

Suggestions

Seven non-blocking findings, all verified: benchmark window growth, two duplicated doc comments, one dead bound check, an inaccurate mocknet stack comment, Makefile pipe masking of test failures, and the coverage regression on the appendPublish rewind wiring (documented trade-off).

Identity note: this review posts as pi-reviewer, so REQUEST_CHANGES is valid (not the author's account).

AI-driven code review (samber/cc-skills-golang passes: style/naming/docs, correctness/safety/concurrency, security/deps, tests/perf/observability) against head `1be416e`, with every claim reproduced in this session against a fresh clone of the PR head. ## Verified green (matching the PR's claims) - `go test ./...` → `ok git.norobo.chat/norobo/webstream 6.538s` - `go vet ./...` and `go vet -tags rapid ./...` → clean - `go test -tags rapid -run TestJournalMemlogParity -count=1` → ok (15.3s) - `govulncheck -mode=source ./...` → 0 called vulnerabilities; the one new dependency `pgregory.net/rapid v1.3.0` is test-only (Go 1.23, module is Go 1.27.1), no `replace` directives - The four production hardening fixes (parseSegmentBase overflow, tsFromHeader length guard, cursor-slot totality, smallestNames/countFromSeqs) are correct as written ## Blocking - `go test -race ./...` — the repo's own `make test` gate and the PR's Verification section — **fails at this head** (2/2 runs, ~20 tests cascade-fail with "race detected during execution of test"). Root cause is one unlocked access in the new `TestStreamStateProperties` (see inline comment). `origin/main` passes the same command (`ok ... 12.542s`), so the race is introduced by this PR. ## Suggestions Seven non-blocking findings, all verified: benchmark window growth, two duplicated doc comments, one dead bound check, an inaccurate mocknet stack comment, Makefile pipe masking of test failures, and the coverage regression on the appendPublish rewind wiring (documented trade-off). Identity note: this review posts as `pi-reviewer`, so REQUEST_CHANGES is valid (not the author's account).
@ -25,3 +31,3 @@
bench:
go test -run '^$$' -bench=. -benchmem ./...
go test -run '^$$' -bench=. -benchmem ./... | tee bench.txt
Member

🟡 SUGGESTION — go test ... | tee bench.txt masks failures: the pipeline's exit status is tee's, so make bench (and bench-log at line 41) report success even when the benchmark run fails to compile or panics; the previous target returned go test's status directly. Fix: set -o pipefail in the recipe, or check $${PIPESTATUS[0]}.

🟡 SUGGESTION — `go test ... | tee bench.txt` masks failures: the pipeline's exit status is `tee`'s, so `make bench` (and `bench-log` at line 41) report success even when the benchmark run fails to compile or panics; the previous target returned go test's status directly. Fix: `set -o pipefail` in the recipe, or check `$${PIPESTATUS[0]}`.
bench_test.go Outdated
@ -530,0 +585,4 @@
b.ResetTimer()
for range b.N {
log.trimBefore(256)
Member

🟡 SUGGESTION — log.trimBefore(256) is an absolute target, so after the first iteration base is already 256 and every later trim is a no-op; the retained window grows 256 records per iteration instead of staying at a realistic window as the comment claims. Probe (go test -run TestZZTrimWindowProbe, scratch test removed after): windows [256,896) → [256,1152) → [256,1408) → [256,1664). At b.N≈37k (bench.log) the log ends ~9.5M records (≈1.5 GB), the trim cost disappears after iteration 1, and measured per-append cost drifts with memory locality — so this is not a stable micro-benchmark. Fix: log.trimBefore(log.base + 256) to keep the window fixed.

🟡 SUGGESTION — `log.trimBefore(256)` is an absolute target, so after the first iteration base is already 256 and every later trim is a no-op; the retained window grows 256 records per iteration instead of staying at a realistic window as the comment claims. Probe (`go test -run TestZZTrimWindowProbe`, scratch test removed after): windows [256,896) → [256,1152) → [256,1408) → [256,1664). At b.N≈37k (bench.log) the log ends ~9.5M records (≈1.5 GB), the trim cost disappears after iteration 1, and measured per-append cost drifts with memory locality — so this is not a stable micro-benchmark. Fix: `log.trimBefore(log.base + 256)` to keep the window fixed.
@ -32,2 +33,4 @@
}
// mockHosts is meshHosts over libp2p's in-memory mocknet: the same full libp2p
// stack (yamux, noise, identify, gating) without kernel sockets, so mesh tests
Member

🟡 SUGGESTION — the comment claims mockHosts runs 'the same full libp2p stack (yamux, noise, identify, gating) without kernel sockets'. Verified against the go-libp2p v0.49.0 source in the module cache: mocknet.GenPeer → GenPeerWithOptions → AddPeerWithOptions builds bhost.NewHost over a mock peernet with none of yamux/noise/identify, and no connection gater is ever installed (the old TCP setup passed libp2p.ConnectionGater(NewAllowlistGater(...)); the mocknet setup drops it). Transport-level security/muxing/gating regressions would not be caught by these tests — the app-level WithAllowedPeers authorization still is, and TestAllowlistGater + integration_test.go:460 keep the gater covered on real TCP. Tighten the comment (and the PR description's 'same yamux/noise/identify stack' claim) to what the mocknet actually exercises.

🟡 SUGGESTION — the comment claims mockHosts runs 'the same full libp2p stack (yamux, noise, identify, gating) without kernel sockets'. Verified against the go-libp2p v0.49.0 source in the module cache: `mocknet.GenPeer` → `GenPeerWithOptions` → `AddPeerWithOptions` builds `bhost.NewHost` over a mock `peernet` with none of yamux/noise/identify, and no connection gater is ever installed (the old TCP setup passed `libp2p.ConnectionGater(NewAllowlistGater(...))`; the mocknet setup drops it). Transport-level security/muxing/gating regressions would not be caught by these tests — the app-level `WithAllowedPeers` authorization still is, and `TestAllowlistGater` + integration_test.go:460 keep the gater covered on real TCP. Tighten the comment (and the PR description's 'same yamux/noise/identify stack' claim) to what the mocknet actually exercises.
store.go Outdated
@ -880,8 +887,20 @@ func tsFromHeader(buf []byte) (int64, error) {
// countFromSeqs derives a segment's record count from its first and last
Member

🟡 SUGGESTION — duplicated doc comment on countFromSeqs: the old block (lines 888-889, 'derives a segment's record count…') sits directly above the new block (lines 890-893) whose first lines repeat it verbatim. Delete the old block.

🟡 SUGGESTION — duplicated doc comment on `countFromSeqs`: the old block (lines 888-889, 'derives a segment's record count…') sits directly above the new block (lines 890-893) whose first lines repeat it verbatim. Delete the old block.
@ -885,1 +901,4 @@
}
count := span + 1
if count > math.MaxUint32 {
Member

🟡 SUGGESTION — the second check if count > math.MaxUint32 is unreachable: the span guard at line 899 already rejects span >= math.MaxUint32, so count = span+1 ≤ MaxUint32 by construction (the property test encodes the same bound: wantOK := last >= first && last-first < math.MaxUint32). Remove the dead check.

🟡 SUGGESTION — the second check `if count > math.MaxUint32` is unreachable: the span guard at line 899 already rejects `span >= math.MaxUint32`, so count = span+1 ≤ MaxUint32 by construction (the property test encodes the same bound: `wantOK := last >= first && last-first < math.MaxUint32`). Remove the dead check.
@ -1474,11 +1487,20 @@ const maxListPageHeap = 1 << 15
// smallestNames returns the limit smallest strings of names in ascending order,
Member

🟡 SUGGESTION — duplicated doc comment on smallestNames: the original three-line comment (lines 1487-1489) is followed verbatim by a second copy (lines 1490-1492). Keep only the copy documenting the new totality/no-mutation contract.

🟡 SUGGESTION — duplicated doc comment on `smallestNames`: the original three-line comment (lines 1487-1489) is followed verbatim by a second copy (lines 1490-1492). Keep only the copy documenting the new totality/no-mutation contract.
@ -955,0 +949,4 @@
beforeCount := j.open.count
beforeTS := j.open.lastTS
if err := j.open.writeRecord(record{seq: 3, timestamp: 4, data: []byte("x")}); err != nil {
Member

🟡 SUGGESTION — removing the failSync seam also removed the only test that drove appendPublishLocked's real failure wiring: this test now manually performs writeRecord + rewindAppendLocked, verifying the mechanism but not that appendPublish calls it on a failed durable write. Verified with go tool cover: the 'rewinding segment after failed append/sync' branches inside appendPublishLocked (store.go:1079-1093) are 0-hit in the current suite. AGENTS.md documents the seam-free choice as deliberate — acknowledged — but the coverage regression is measurable, so a future regression in the rewind wiring would go unnoticed. Worth recording this specific gap in coverage.log when the baseline is next updated.

🟡 SUGGESTION — removing the `failSync` seam also removed the only test that drove appendPublishLocked's real failure wiring: this test now manually performs `writeRecord` + `rewindAppendLocked`, verifying the mechanism but not that appendPublish calls it on a failed durable write. Verified with `go tool cover`: the 'rewinding segment after failed append/sync' branches inside appendPublishLocked (store.go:1079-1093) are 0-hit in the current suite. AGENTS.md documents the seam-free choice as deliberate — acknowledged — but the coverage regression is measurable, so a future regression in the rewind wiring would go unnoticed. Worth recording this specific gap in `coverage.log` when the baseline is next updated.
@ -0,0 +188,4 @@
}
// The cached minimum cursor must equal the true minimum.
if got := state.minConsumerCursorLocked(); got != model.minCursor() {
Member

🔴 BLOCKING — state.minConsumerCursorLocked() is called without holding state.lock, and the method mutates the minCursor/minCursorValid cache (writes at stream_state.go:800-801, reads at :788-789). When the property draws a maxAge > 0 stream, the production ageLoop goroutine (started at stream_state.go:135) runs trimExpired → trimNowLocked → computeKeepLocked → minConsumerCursorLocked under the lock, so the test's unlocked call races with it on the cache fields. Verified: go test -race ./... fails 2/2 runs with WARNING: DATA RACE (goroutine 18 = ageLoop reading stream_state.go:788; goroutine 17 = TestStreamStateProperties.func1.1.1 at this line writing :800), and ~20 unrelated tests cascade-fail race detected during execution of test; origin/main passes (ok ... 12.542s). This directly contradicts the PR's Verification section. A synctest bubble does not exempt the access: the race detector treats bubble goroutines as concurrent, and make test runs -race. Fix — hold the write lock (the method writes the cache, so RLock would still race the writer): state.lock.Lock(); got := state.minConsumerCursorLocked(); state.lock.Unlock().

🔴 BLOCKING — `state.minConsumerCursorLocked()` is called without holding `state.lock`, and the method mutates the `minCursor`/`minCursorValid` cache (writes at stream_state.go:800-801, reads at :788-789). When the property draws a `maxAge > 0` stream, the production `ageLoop` goroutine (started at stream_state.go:135) runs `trimExpired → trimNowLocked → computeKeepLocked → minConsumerCursorLocked` under the lock, so the test's unlocked call races with it on the cache fields. Verified: `go test -race ./...` fails 2/2 runs with `WARNING: DATA RACE` (goroutine 18 = ageLoop reading stream_state.go:788; goroutine 17 = TestStreamStateProperties.func1.1.1 at this line writing :800), and ~20 unrelated tests cascade-fail `race detected during execution of test`; `origin/main` passes (`ok ... 12.542s`). This directly contradicts the PR's Verification section. A synctest bubble does not exempt the access: the race detector treats bubble goroutines as concurrent, and `make test` runs `-race`. Fix — hold the write lock (the method writes the cache, so RLock would still race the writer): `state.lock.Lock(); got := state.minConsumerCursorLocked(); state.lock.Unlock()`.
pi-agent referenced this pull request from a commit 2026-10-12 20:33:27 +00:00
Blocking:

- TestStreamStateProperties called minConsumerCursorLocked without the
  stream lock. The method writes the minCursor/minCursorValid cache, so on a
  maxAge stream the age loop (which recomputes it under the lock) raced the
  test's unlocked call; go test -race ./... failed 2/2 with a data race on
  stream_state.go:788/800. The check now holds the write lock around the
  call. go test -race ./... passes again.

Suggestions:

- BenchmarkMemlogTrimBefore trimmed to an absolute 256, so after the first
  iteration every trim was a no-op and the retained window grew 256 records
  per iteration (a ~1.5 GB log at b.N=37k). The trim point is now
  base-relative (log.base + 256), so the window stays fixed and every
  iteration pays a real trim; bench.log re-recorded.
- Removed the dead `count > math.MaxUint32` check in countFromSeqs: the
  span guard already rejects span >= MaxUint32, so count = span+1 fits by
  construction.
- Removed two duplicated doc comments (countFromSeqs, smallestNames); the
  inline duplicate of the span-wrap explanation is folded into the doc
  comment.
- Tightened the mockHosts comment: mocknet hosts are plain hosts over a
  mock peernet with no yamux/noise/identify and no connection gater; the
  gater remains covered by TestAllowlistGater and the real-TCP integration
  tests. The PR body's matching claim was corrected too.
- Makefile bench and bench-log recipes now use `set -o pipefail` so a
  failing go test is not masked by tee's exit status.
- coverage.log records the known 0-hit rewind-after-failed-append/sync
  branches in appendPublishLocked (closed file fails the leading Seek
  first; no seam-free test reaches them), per the review's request.
Author
Member

All 8 review findings resolved in f70f71f:

Blocking

  • state.minConsumerCursorLocked() in TestStreamStateProperties was called without the stream lock; the method writes the minCursor/minCursorValid cache, so on a maxAge stream the age loop raced it. The check now holds the write lock around the call. Verified: go test -race ./... passes at head (the reviewer's 2/2 data-race repro is gone; origin/main comparison no longer differs).

Suggestions

  • BenchmarkMemlogTrimBefore now trims at log.base + 256 instead of an absolute 256: every iteration pays a real trim and the window stays fixed (the old form made trims no-ops after iteration 1 and grew the window ~1.5 GB at b.N=37k). bench.log re-recorded: 31.4 µs/op steady-state.
  • countFromSeqs: removed the dead count > math.MaxUint32 check (the span guard makes it unreachable) and the duplicated doc/inline comments.
  • smallestNames: removed the duplicated doc comment.
  • mockHosts comment (and the PR body): mocknet hosts are plain hosts over a mock peernet — no yamux/noise/identify, no connection gater. The gater stays covered by TestAllowlistGater and the real-TCP integration tests.
  • Makefile: set -o pipefail on the bench and bench-log recipes so tee no longer masks a failing go test.
  • The appendPublishLocked rewind-wiring gap (store.go 1079-1093, 0-hit) is recorded in coverage.log per the review's request; a closed file fails the leading Seek first, so no seam-free test reaches it.

Verified at head: go test ./..., go test -race ./..., go vet ./..., golangci-lint run ./... all pass.

All 8 review findings resolved in `f70f71f`: **Blocking** - `state.minConsumerCursorLocked()` in `TestStreamStateProperties` was called without the stream lock; the method writes the `minCursor`/`minCursorValid` cache, so on a maxAge stream the age loop raced it. The check now holds the write lock around the call. Verified: `go test -race ./...` passes at head (the reviewer's 2/2 data-race repro is gone; `origin/main` comparison no longer differs). **Suggestions** - `BenchmarkMemlogTrimBefore` now trims at `log.base + 256` instead of an absolute 256: every iteration pays a real trim and the window stays fixed (the old form made trims no-ops after iteration 1 and grew the window ~1.5 GB at b.N=37k). bench.log re-recorded: 31.4 µs/op steady-state. - `countFromSeqs`: removed the dead `count > math.MaxUint32` check (the span guard makes it unreachable) and the duplicated doc/inline comments. - `smallestNames`: removed the duplicated doc comment. - `mockHosts` comment (and the PR body): mocknet hosts are plain hosts over a mock peernet — no yamux/noise/identify, no connection gater. The gater stays covered by `TestAllowlistGater` and the real-TCP integration tests. - Makefile: `set -o pipefail` on the `bench` and `bench-log` recipes so tee no longer masks a failing `go test`. - The `appendPublishLocked` rewind-wiring gap (store.go 1079-1093, 0-hit) is recorded in `coverage.log` per the review's request; a closed file fails the leading Seek first, so no seam-free test reaches it. Verified at head: `go test ./...`, `go test -race ./...`, `go vet ./...`, `golangci-lint run ./...` all pass.
sam merged commit c54f2e3129 into main 2026-10-12 20:34:23 +00:00
sam deleted branch test/fuzz-property-mutation 2026-10-12 20:34:23 +00:00
Sign in to join this conversation.
No reviewers
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
norobo/webstream!13
No description provided.