test: fuzz, property and mutation test foundation #13
No reviewers
Labels
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
norobo/webstream!13
Loading…
Reference in a new issue
No description provided.
Delete branch "test/fuzz-property-mutation"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Adds the test foundation the library needs to stay correct under edits by people who did not write it: native fuzz targets for the byte-level parsers, rapid property models for the stateful storage layer, and deterministic time/transport for tests.
TESTING_PLAN.md(new) is the inventory and rationale.Bugs found and fixed (all caught by the new tests)
decodeDoubleBufferedpanicked on a short cursor slot sliceFuzzDecodeCursorSlotparseSegmentBaseaccepted overflowing segment base namesFuzzParseSegmentBasetsFromHeaderpanicked on a short bufferFuzzTSFromHeadermemlogrecovery replayed records wrongly (appendRecoveredregression)memloggap-slot byte underflow (gap accounting now starts atmax(next, base))removeTail(MaxUint64)wrap panicsmallestNamesalias-mutated its input and panicked on empty names /limit <= 0TestSmallestNamescountFromSeqs(0, MaxUint64)wraparoundTestCountFromSeqsFuzz targets (native
testing.F, with seeds so plaingo testreplays them)Tier 1: replication command/record codecs, cursor watermark slot, segment scan, stream-name extractors. Tier 2: segment base names, record timestamps, header validation. No production change of behaviour beyond the four fixes above.
Property models (
pgregory.net/rapid, test-only dependency)TestMemlogProperties- append/recover/trim op sequences against a shadow log (window, records, byte accounting).TestJournalMemlogParity- differential: the on-disk journal reloaded must equal the live in-memory log (batched mode, so per-publish fsync does not dominate).TestConsumerStateProperties- leases, cursors and watermarks, in a synctest bubble.TestStreamStateProperties-appendDedup/ consumer create / ack / delete / clock-elapse sequences against a shadow model: dedup table tied to retention, dedup hit/miss per retained window, interest-vs-age trim point, cached minimum cursor invalidation.smallestNames,countFromSeqs,hashLeaderdistribution and name grammars.Test infrastructure
nowclock field (consumer_state.go) and thefailSynchook (store.go) are gone; time-dependent tests usetesting/synctestbubbles with a deterministic fake clock.TestAllowlistGaterand the real-TCP integration tests. Benchmarks keep real TCP because they measure the transport, and the QUIC test keeps it because it tests that transport.rapid.SyncTestfor the bubble models. A rawsynctest.Testinsiderapid.Checkcannot report a failure: it panics out of the bubble's goroutine, killing the process (exit status 2) with no shrunk counterexample. Verified with a falsifiable property.rapidbuild tag, so plaingo testand gremlins skip it; the invariant models run on everygo test(~1s; default suite 5.0s -> 6.1s).make test-propsruns everything.Verification
go test ./...(6.5s),go test -race ./...(make test),go test -tags rapid ./...,go vet ./...,go vet -tags rapid ./...,golangci-lint run ./...andgolangci-lint run --build-tags rapid ./...all pass.Not in this PR: benchmarks, coverage baseline and the gremlins mutation run — those became part of the branch during review and are documented in
TESTING_PLAN.mdand the new README Testing section.AI-driven code review (samber/cc-skills-golang passes: style/naming/docs, correctness/safety/concurrency, security/deps, tests/perf/observability) against head
1be416e, with every claim reproduced in this session against a fresh clone of the PR head.Verified green (matching the PR's claims)
go test ./...→ok git.norobo.chat/norobo/webstream 6.538sgo vet ./...andgo vet -tags rapid ./...→ cleango test -tags rapid -run TestJournalMemlogParity -count=1→ ok (15.3s)govulncheck -mode=source ./...→ 0 called vulnerabilities; the one new dependencypgregory.net/rapid v1.3.0is test-only (Go 1.23, module is Go 1.27.1), noreplacedirectivesBlocking
go test -race ./...— the repo's ownmake testgate and the PR's Verification section — fails at this head (2/2 runs, ~20 tests cascade-fail with "race detected during execution of test"). Root cause is one unlocked access in the newTestStreamStateProperties(see inline comment).origin/mainpasses the same command (ok ... 12.542s), so the race is introduced by this PR.Suggestions
Seven non-blocking findings, all verified: benchmark window growth, two duplicated doc comments, one dead bound check, an inaccurate mocknet stack comment, Makefile pipe masking of test failures, and the coverage regression on the appendPublish rewind wiring (documented trade-off).
Identity note: this review posts as
pi-reviewer, so REQUEST_CHANGES is valid (not the author's account).@ -25,3 +31,3 @@bench:go test -run '^$$' -bench=. -benchmem ./...go test -run '^$$' -bench=. -benchmem ./... | tee bench.txt🟡 SUGGESTION —
go test ... | tee bench.txtmasks failures: the pipeline's exit status istee's, somake bench(andbench-logat line 41) report success even when the benchmark run fails to compile or panics; the previous target returned go test's status directly. Fix:set -o pipefailin the recipe, or check$${PIPESTATUS[0]}.@ -530,0 +585,4 @@b.ResetTimer()for range b.N {log.trimBefore(256)🟡 SUGGESTION —
log.trimBefore(256)is an absolute target, so after the first iteration base is already 256 and every later trim is a no-op; the retained window grows 256 records per iteration instead of staying at a realistic window as the comment claims. Probe (go test -run TestZZTrimWindowProbe, scratch test removed after): windows [256,896) → [256,1152) → [256,1408) → [256,1664). At b.N≈37k (bench.log) the log ends ~9.5M records (≈1.5 GB), the trim cost disappears after iteration 1, and measured per-append cost drifts with memory locality — so this is not a stable micro-benchmark. Fix:log.trimBefore(log.base + 256)to keep the window fixed.@ -32,2 +33,4 @@}// mockHosts is meshHosts over libp2p's in-memory mocknet: the same full libp2p// stack (yamux, noise, identify, gating) without kernel sockets, so mesh tests🟡 SUGGESTION — the comment claims mockHosts runs 'the same full libp2p stack (yamux, noise, identify, gating) without kernel sockets'. Verified against the go-libp2p v0.49.0 source in the module cache:
mocknet.GenPeer→GenPeerWithOptions→AddPeerWithOptionsbuildsbhost.NewHostover a mockpeernetwith none of yamux/noise/identify, and no connection gater is ever installed (the old TCP setup passedlibp2p.ConnectionGater(NewAllowlistGater(...)); the mocknet setup drops it). Transport-level security/muxing/gating regressions would not be caught by these tests — the app-levelWithAllowedPeersauthorization still is, andTestAllowlistGater+ integration_test.go:460 keep the gater covered on real TCP. Tighten the comment (and the PR description's 'same yamux/noise/identify stack' claim) to what the mocknet actually exercises.@ -880,8 +887,20 @@ func tsFromHeader(buf []byte) (int64, error) {// countFromSeqs derives a segment's record count from its first and last🟡 SUGGESTION — duplicated doc comment on
countFromSeqs: the old block (lines 888-889, 'derives a segment's record count…') sits directly above the new block (lines 890-893) whose first lines repeat it verbatim. Delete the old block.@ -885,1 +901,4 @@}count := span + 1if count > math.MaxUint32 {🟡 SUGGESTION — the second check
if count > math.MaxUint32is unreachable: the span guard at line 899 already rejectsspan >= math.MaxUint32, so count = span+1 ≤ MaxUint32 by construction (the property test encodes the same bound:wantOK := last >= first && last-first < math.MaxUint32). Remove the dead check.@ -1474,11 +1487,20 @@ const maxListPageHeap = 1 << 15// smallestNames returns the limit smallest strings of names in ascending order,🟡 SUGGESTION — duplicated doc comment on
smallestNames: the original three-line comment (lines 1487-1489) is followed verbatim by a second copy (lines 1490-1492). Keep only the copy documenting the new totality/no-mutation contract.@ -955,0 +949,4 @@beforeCount := j.open.countbeforeTS := j.open.lastTSif err := j.open.writeRecord(record{seq: 3, timestamp: 4, data: []byte("x")}); err != nil {🟡 SUGGESTION — removing the
failSyncseam also removed the only test that drove appendPublishLocked's real failure wiring: this test now manually performswriteRecord+rewindAppendLocked, verifying the mechanism but not that appendPublish calls it on a failed durable write. Verified withgo tool cover: the 'rewinding segment after failed append/sync' branches inside appendPublishLocked (store.go:1079-1093) are 0-hit in the current suite. AGENTS.md documents the seam-free choice as deliberate — acknowledged — but the coverage regression is measurable, so a future regression in the rewind wiring would go unnoticed. Worth recording this specific gap incoverage.logwhen the baseline is next updated.@ -0,0 +188,4 @@}// The cached minimum cursor must equal the true minimum.if got := state.minConsumerCursorLocked(); got != model.minCursor() {🔴 BLOCKING —
state.minConsumerCursorLocked()is called without holdingstate.lock, and the method mutates theminCursor/minCursorValidcache (writes at stream_state.go:800-801, reads at :788-789). When the property draws amaxAge > 0stream, the productionageLoopgoroutine (started at stream_state.go:135) runstrimExpired → trimNowLocked → computeKeepLocked → minConsumerCursorLockedunder the lock, so the test's unlocked call races with it on the cache fields. Verified:go test -race ./...fails 2/2 runs withWARNING: DATA RACE(goroutine 18 = ageLoop reading stream_state.go:788; goroutine 17 = TestStreamStateProperties.func1.1.1 at this line writing :800), and ~20 unrelated tests cascade-failrace detected during execution of test;origin/mainpasses (ok ... 12.542s). This directly contradicts the PR's Verification section. A synctest bubble does not exempt the access: the race detector treats bubble goroutines as concurrent, andmake testruns-race. Fix — hold the write lock (the method writes the cache, so RLock would still race the writer):state.lock.Lock(); got := state.minConsumerCursorLocked(); state.lock.Unlock().All 8 review findings resolved in
f70f71f:Blocking
state.minConsumerCursorLocked()inTestStreamStatePropertieswas called without the stream lock; the method writes theminCursor/minCursorValidcache, so on a maxAge stream the age loop raced it. The check now holds the write lock around the call. Verified:go test -race ./...passes at head (the reviewer's 2/2 data-race repro is gone;origin/maincomparison no longer differs).Suggestions
BenchmarkMemlogTrimBeforenow trims atlog.base + 256instead of an absolute 256: every iteration pays a real trim and the window stays fixed (the old form made trims no-ops after iteration 1 and grew the window ~1.5 GB at b.N=37k). bench.log re-recorded: 31.4 µs/op steady-state.countFromSeqs: removed the deadcount > math.MaxUint32check (the span guard makes it unreachable) and the duplicated doc/inline comments.smallestNames: removed the duplicated doc comment.mockHostscomment (and the PR body): mocknet hosts are plain hosts over a mock peernet — no yamux/noise/identify, no connection gater. The gater stays covered byTestAllowlistGaterand the real-TCP integration tests.set -o pipefailon thebenchandbench-logrecipes so tee no longer masks a failinggo test.appendPublishLockedrewind-wiring gap (store.go 1079-1093, 0-hit) is recorded incoverage.logper the review's request; a closed file fails the leading Seek first, so no seam-free test reaches it.Verified at head:
go test ./...,go test -race ./...,go vet ./...,golangci-lint run ./...all pass.