feature/server-persistence #1

Merged
sam merged 17 commits from feature/server-persistence into main 2026-09-24 20:46:23 +00:00
Owner
No description provided.
Three durability fixes on the durable server path:

- Recover the in-memory log base from the oldest retained segment so a
  stream that trimmed across a segment boundary no longer restarts at
  base 0, which made fresh consumers pull empty gap records and
  underflowed the trim byte accounting.
- Surface a panicking user codec from Stream.Publish as an error instead
  of a silent (PublishAck{}, nil) success; scoped to the encode call so
  the scratch buffer is returned to the pool exactly once.
- Run the consumer watermark coalescer on its own fixed cadence,
  independent of the publish fsync policy, so watermarks reach disk even
  in the default per-publish-fsync mode and are not flushed only at
  shutdown.

Adds regression tests for each: trimmed-restart base and a fresh
consumer (store-level and full-server recovery), a panicking publish
codec, and the cursor coalescer flushing on cadence without a close.
Merge branch 'main' into feature/server-persistence
All checks were successful
code-review / Agent review (pull_request) Successful in 9s
00159a5698
pi-agent requested changes 2026-09-20 12:25:30 +00:00
Dismissed
pi-agent left a comment

High-signal review of feature/server-persistence (Phase 1-2 durability). The design is thoughtful and well-documented: stream-as-directory, derived base/next, double-buffered CRC watermark, and a thorough crash-ordering analysis. Recovery tests are strong. Two durability issues should be fixed before merge (a data-loss path under the documented crash model, and an exactly-once gap on the publish error path), plus lifecycle and minor concerns.

High-signal review of feature/server-persistence (Phase 1-2 durability). The design is thoughtful and well-documented: stream-as-directory, derived base/next, double-buffered CRC watermark, and a thorough crash-ordering analysis. Recovery tests are strong. Two durability issues should be fixed before merge (a data-loss path under the documented crash model, and an exactly-once gap on the publish error path), plus lifecycle and minor concerns.
@ -249,0 +269,4 @@
return all, nil
}
cursor = resp.streams[len(resp.streams)-1]
Member

If a server ever returns hasMore: true with an empty streams page (a buggy/foreign peer, or a limit underflow), resp.streams[len(resp.streams)-1] panics with index-out-of-range. It relies on a non-local server invariant. Guard the page before indexing: if hasMore && len(resp.streams) == 0 { return nil, errors.New(...) }.

If a server ever returns `hasMore: true` with an empty `streams` page (a buggy/foreign peer, or a limit underflow), `resp.streams[len(resp.streams)-1]` panics with index-out-of-range. It relies on a non-local server invariant. Guard the page before indexing: if `hasMore && len(resp.streams) == 0 { return nil, errors.New(...) }`.
sam marked this conversation as resolved
@ -102,0 +138,4 @@
}
server.store = st
st.start(ctx)
Member

IMPORTANT - Server has no Close and nothing shuts the store down: store.close() is only invoked directly in tests. In production the cursor coalescer and publishLoop goroutines run forever on NewServer's ctx and the open segment/consumer file handles are never closed or flushed, so the storage root cannot be cleanly reopened and fds leak across restarts. Add a Server.Close (or a WithStore-owned shutdown hook) that cancels the loops and flushes + closes every journal.

**IMPORTANT** - `Server` has no `Close` and nothing shuts the store down: `store.close()` is only invoked directly in tests. In production the cursor coalescer and publishLoop goroutines run forever on NewServer's ctx and the open segment/consumer file handles are never closed or flushed, so the storage root cannot be cleanly reopened and fds leak across restarts. Add a `Server.Close` (or a `WithStore`-owned shutdown hook) that cancels the loops and flushes + closes every journal.
sam marked this conversation as resolved
@ -513,0 +625,4 @@
// vanish or resurrect across a restart. Stream creation is rare, so holding
// the server lock for the mount is acceptable.
if s.store != nil {
journal, err := s.store.createStream(req.name, req.maxAge)
Member

createStream mounts the stream directory (Mkdir + writeStreamMeta temp+rename+fsync + two consumer headers + syncStreamsDir) while holding the global s.lock. A slow or failed disk stalls every stream's create and all concurrent operations. The in-code comment acknowledges stream creation is rare, but consider performing the disk mount outside the server lock and reconciling the map under it, so a blocking fsync cannot serialize the whole server.

`createStream` mounts the stream directory (Mkdir + writeStreamMeta temp+rename+fsync + two consumer headers + syncStreamsDir) while holding the global `s.lock`. A slow or failed disk stalls every stream's create and all concurrent operations. The in-code comment acknowledges stream creation is rare, but consider performing the disk mount outside the server lock and reconciling the map under it, so a blocking fsync cannot serialize the whole server.
sam marked this conversation as resolved
server.go Outdated
@ -705,0 +892,4 @@
// latency. The append's own interest/age trims delete segments via
// trimAfterCursorAdvance and the trim hooks; nothing is written for a moved
// base, since base/next are derived from the segment set.
if err := persistPublish(stream, rec); err != nil {
Member

IMPORTANT - appendDedup mutates the in-memory memlog and the dedup map before persistPublish runs. If persistPublish fails (fsync/I-O error), the request returns ErrInternal but the record is already in the log and state.dedup[id] = rec.seq is set. A client retry with the same dedup ID then returns duplicate: true with the stored seq, so the client believes the record is durable when it never reached disk - after a restart the acked record is gone, breaking exactly-once storage. Fix: register the dedup mapping (and roll back the in-memory append if feasible) only after the durable write succeeds, so an error leaves the publish re-doable as a fresh append.

**IMPORTANT** - `appendDedup` mutates the in-memory memlog and the dedup map *before* `persistPublish` runs. If `persistPublish` fails (fsync/I-O error), the request returns ErrInternal but the record is already in the log and `state.dedup[id] = rec.seq` is set. A client retry with the same dedup ID then returns `duplicate: true` with the stored seq, so the client believes the record is durable when it never reached disk - after a restart the acked record is gone, breaking exactly-once storage. Fix: register the dedup mapping (and roll back the in-memory append if feasible) only after the durable write succeeds, so an error leaves the publish re-doable as a fresh append.
sam marked this conversation as resolved
store.go Outdated
@ -0,0 +987,4 @@
// Delete oldest segments while at least two remain (never the newest) and
// the oldest's end (its next segment's base) is at or below base.
for len(j.segs) > 1 && j.segs[0].base+uint64(j.segs[0].count) <= base {
Member

BLOCKING - trimTo derives the oldest sealed segment's end from j.segs[0].base + uint64(j.segs[0].count), but the sealed count header is written in sealLocked via WriteAt + Close with no fsync before the next segment and its directory entry are fsynced. This branch's own crash-ordering spec (plan.md) says the header count "cannot be the authority" for trim and that a sealed segment's end is the next segment's base. If a crash leaves a stale (understated) count on the oldest segment and the trim base later lands inside that segment's true record range, base+count <= base becomes true and the segment is deleted even though records still inside the retained window would be dropped - silent loss of durable, acked records. Fix: derive the end from the next segment's base, e.g. end := j.segs[1].base when len(j.segs) >= 2, deleting while end <= base.

**BLOCKING** - `trimTo` derives the oldest sealed segment's end from `j.segs[0].base + uint64(j.segs[0].count)`, but the sealed `count` header is written in `sealLocked` via WriteAt + Close with **no fsync** before the next segment and its directory entry are fsynced. This branch's own crash-ordering spec (plan.md) says the header count "cannot be the authority" for trim and that a sealed segment's end is the **next segment's base**. If a crash leaves a stale (understated) count on the oldest segment and the trim base later lands inside that segment's true record range, `base+count <= base` becomes true and the segment is deleted even though records still inside the retained window would be dropped - silent loss of durable, acked records. Fix: derive the end from the next segment's base, e.g. `end := j.segs[1].base` when `len(j.segs) >= 2`, deleting while `end <= base`.
sam marked this conversation as resolved
fix: resolve PR review on trim count authority, publish rollback, server close, and mount locking
All checks were successful
code-review / Agent review (pull_request) Successful in 9s
ae2a198997
Resolves all five review comments on feature/server-persistence:

- trimTo: derive a sealed segment's end from the NEXT segment's durable
  filename base, not the oldest's base+count. A sealed count is finalized
  with an unsynced header rewrite before the next segment is fsynced, so it
  can be stale (understated) after a crash; base+count could then delete a
  segment whose surviving records are still inside the retained window
  (silent loss of durable, acked records). Regression test corrupts seg0's
  count to 0 and trims inside it.
- publish: roll back a failed durable write so the publish stays re-doable as
  a fresh append. appendDedup commits to the in-memory log and dedup map
  before persistPublish; on failure, rollbackFailedPublish drops the dedup
  mapping and, when the record is still the tail, removes it, so a retry of
  the same ID is not reported as a durable duplicate. Adds memlog.removeTail.
- lifecycle: add Server.Close and an auto-close watcher so the cursor
  coalescer and publish loop stop, and every segment/consumer file is flushed
  and closed when the NewServer context is cancelled or Close is called.
  store.close is now idempotent.
- listStreams client: guard hasMore with an empty page so a malformed server
  response cannot panic on index out of range.
- createStream: mount the stream directory outside the global server lock,
  reserving the name in s.creating so a concurrent duplicate create still
  resolves to a no-op instead of mounting twice.
pi-agent approved these changes 2026-09-20 13:10:15 +00:00
Dismissed
pi-agent left a comment

Reviewing ae2a198 (head of feature/server-persistence). All five REQUEST_CHANGES comments from review #4 are addressed:

  • #74 (BLOCKING, trimTo): trimTo now derives the oldest sealed segment's end from the NEXT segment's durable filename base (j.segs[1].base) instead of j.segs[0].base + count. The sealed count header is an unsynced WriteAt+Close rewrite and can be stale after a crash, so it is no longer treated as the authority for trim. New regression test TestTrimRecoveryStaleCountKeepsRecords corrupts seg0's count to 0 and confirms the retained records survive (fails on the old code).
  • #75 (IMPORTANT, publish): publish now rolls back on persistPublish failure via streamState.rollbackFailedPublish (drops the dedup mapping and, when the record is still the tail, removes it through the new memlog.removeTail), so a retry of the same dedup ID is a fresh append rather than a misleading duplicate:true. Covered by TestRollbackFailedPublishEnablesRedo.
  • #76 (IMPORTANT, lifecycle): added Server.Close() plus an auto-close watcher on loopCtx.Done in store.start; store.close is idempotent (new closed guard under streamMu), so the coalescer/publishLoop goroutines stop and journals flush/close cleanly.
  • #77 (minor, listStreams): guards hasMore && len(resp.streams)==0 before indexing the page, returning all,nil instead of panicking.
  • #78 (minor, createStream): the durable directory mount moved outside the global s.lock, with a s.creating reservation set so concurrent duplicate creates stay a no-op.

Verified: gofmt, go vet ./..., go build ./..., and go test -count=1 ./... all pass. (Note: this Forgejo instance exposes no API/MCP action to set resolver=true per thread — the resolve routes return 405 — so each thread should be marked resolved in the UI; code-wise the threads are all addressed by this commit.)

Reviewing ae2a198 (head of feature/server-persistence). All five REQUEST_CHANGES comments from review #4 are addressed: - **#74 (BLOCKING, trimTo):** `trimTo` now derives the oldest sealed segment's end from the NEXT segment's durable filename base (`j.segs[1].base`) instead of `j.segs[0].base + count`. The sealed `count` header is an unsynced WriteAt+Close rewrite and can be stale after a crash, so it is no longer treated as the authority for trim. New regression test `TestTrimRecoveryStaleCountKeepsRecords` corrupts seg0's count to 0 and confirms the retained records survive (fails on the old code). - **#75 (IMPORTANT, publish):** `publish` now rolls back on `persistPublish` failure via `streamState.rollbackFailedPublish` (drops the dedup mapping and, when the record is still the tail, removes it through the new `memlog.removeTail`), so a retry of the same dedup ID is a fresh append rather than a misleading `duplicate:true`. Covered by `TestRollbackFailedPublishEnablesRedo`. - **#76 (IMPORTANT, lifecycle):** added `Server.Close()` plus an auto-close watcher on `loopCtx.Done` in `store.start`; `store.close` is idempotent (new `closed` guard under `streamMu`), so the coalescer/publishLoop goroutines stop and journals flush/close cleanly. - **#77 (minor, listStreams):** guards `hasMore && len(resp.streams)==0` before indexing the page, returning `all,nil` instead of panicking. - **#78 (minor, createStream):** the durable directory mount moved outside the global `s.lock`, with a `s.creating` reservation set so concurrent duplicate creates stay a no-op. Verified: `gofmt`, `go vet ./...`, `go build ./...`, and `go test -count=1 ./...` all pass. (Note: this Forgejo instance exposes no API/MCP action to set `resolver=true` per thread — the resolve routes return 405 — so each thread should be marked resolved in the UI; code-wise the threads are all addressed by this commit.)
pi-agent requested changes 2026-09-20 13:13:59 +00:00
Dismissed
pi-agent left a comment

AI code review — feature/server-persistence

Reviewed the durability implementation (store.go, state.go, server.go, memlog.go, protocol.go, conn.go and tests) against the golang skill passes. The design — stream-directory-as-storage-unit, base/next derived from the segment set, double-buffered CRC watermark, *os.Root confinement — is coherent, well documented and the crash-ordering reasoning is mostly airtight. No vulnerabilities or data races found. Two real issues on the durable-write and recovery paths should be addressed before merge; one design nit.

Important

  1. Failed durable-publish rollback does not roll back the journal: a write/fsync failure leaves the on-disk segment ahead of the in-memory log, so the retry reuses the sequence number and a later restart replays duplicate-seq frames (bytes double-counted, records overwritten). The rollback test only covers the nil-journal path.
  2. Recovery never validates a segment frame's seq before appendRecovered sizes its chunk array; a corrupt/tampered seq forces unbounded allocation during startup.

Suggestion
3. Consumer create/delete do disk I/O (incl. fsyncs) under the server's global lock, contradicting the PR's own move of createStream's disk mount outside that lock.

## AI code review — feature/server-persistence Reviewed the durability implementation (store.go, state.go, server.go, memlog.go, protocol.go, conn.go and tests) against the golang skill passes. The design — stream-directory-as-storage-unit, base/next derived from the segment set, double-buffered CRC watermark, `*os.Root` confinement — is coherent, well documented and the crash-ordering reasoning is mostly airtight. No vulnerabilities or data races found. Two real issues on the durable-write and recovery paths should be addressed before merge; one design nit. **Important** 1. Failed durable-publish rollback does not roll back the journal: a write/fsync failure leaves the on-disk segment ahead of the in-memory log, so the retry reuses the sequence number and a later restart replays duplicate-seq frames (bytes double-counted, records overwritten). The rollback test only covers the nil-journal path. 2. Recovery never validates a segment frame's `seq` before `appendRecovered` sizes its chunk array; a corrupt/tampered seq forces unbounded allocation during startup. **Suggestion** 3. Consumer create/delete do disk I/O (incl. fsyncs) under the server's global lock, contradicting the PR's own move of `createStream`'s disk mount outside that lock.
@ -166,0 +197,4 @@
// A never-acked consumer's only persistent state is watermark == 0, exactly
// what re-creating it produces, so create is write-behind and self-healing.
state.persistConsumerCreate(name)
Member

🟡 SUGGESTION — consumer create/delete do disk I/O while holding the server's global lock.

addConsumerIfRoom calls persistConsumerCreate → consumerSlot (store.go:1415) and deleteConsumer calls persistConsumerDelete → deleteConsumer, both of which do file writes and fsyncs — and these run while the caller (Server.createConsumer/deleteConsumer) holds s.lock. This reintroduces exactly what the PR fixed for createStream, whose directory mount is moved outside the server lock so "disk I/O does not block every stream operation" (createStream comment). Not a correctness bug, but a global-latency regression. Consider reserving the slot under the lock and doing the file write/fsync after releasing it.

🟡 SUGGESTION — consumer create/delete do disk I/O while holding the server's global lock. `addConsumerIfRoom` calls `persistConsumerCreate` → `consumerSlot` (store.go:1415) and `deleteConsumer` calls `persistConsumerDelete` → `deleteConsumer`, both of which do file writes and fsyncs — and these run while the caller (`Server.createConsumer`/`deleteConsumer`) holds `s.lock`. This reintroduces exactly what the PR fixed for `createStream`, whose directory mount is moved outside the server lock so "disk I/O does not block every stream operation" (createStream comment). Not a correctness bug, but a global-latency regression. Consider reserving the slot under the lock and doing the file write/fsync after releasing it.
sam marked this conversation as resolved
@ -293,0 +361,4 @@
// publish already grew past it, only the dedup entry is removed; the leftover
// non-durable record is retained in memory for the retained window rather than
// corrupt the log.
func (state *streamState) rollbackFailedPublish(rec record) {
Member

🟠 IMPORTANT — failed durable-publish rollback does not roll back the journal, desyncing the on-disk log from the in-memory log.

rollbackFailedPublish only restores the memlog tail and the dedup entry; it never touches state.journal. But appendPublish (store.go:841) has already written the frame to the open segment and bumped open.count (writeRecord, store.go:811) before syncLocked (store.go:941) is the step that can fail. On a write/fsync failure the segment object and file are left one record ahead of the memlog, which has been rolled back to rec.seq.

The retry (server.go:938) then re-publishes the same dedup ID as a fresh append, and the memlog reuses rec.seq (its tail was removed), so a second frame with the same seq is appended to the segment. After a restart, loadRecords replays both frames into appendRecovered, which overwrites the duplicate record and double-counts retainedBytes — corrupting byte-budget accounting and violating the "sequence number never reused" invariant the recovery logic depends on.

Condition: any failed durable write/sync — precisely the failure this rollback exists to handle. TestRollbackFailedPublishEnablesRedo builds a nil-journal stream, so it never exercises the desync.

Fix: on failure also rewind the journal segment (reset open.count, truncate the file back to the pre-append offset, clear dirty), and add a test with a real *os.Root/segment; or stop serving the stream once a publish cannot be made durable.

🟠 IMPORTANT — failed durable-publish rollback does not roll back the journal, desyncing the on-disk log from the in-memory log. `rollbackFailedPublish` only restores the memlog tail and the dedup entry; it never touches `state.journal`. But `appendPublish` (store.go:841) has already written the frame to the open segment and bumped `open.count` (`writeRecord`, store.go:811) before `syncLocked` (store.go:941) is the step that can fail. On a write/fsync failure the segment object and file are left one record ahead of the memlog, which has been rolled back to `rec.seq`. The retry (server.go:938) then re-publishes the same dedup ID as a fresh append, and the memlog reuses `rec.seq` (its tail was removed), so a second frame with the *same* seq is appended to the segment. After a restart, `loadRecords` replays both frames into `appendRecovered`, which overwrites the duplicate record and double-counts `retainedBytes` — corrupting byte-budget accounting and violating the "sequence number never reused" invariant the recovery logic depends on. Condition: any failed durable write/sync — precisely the failure this rollback exists to handle. `TestRollbackFailedPublishEnablesRedo` builds a nil-journal stream, so it never exercises the desync. Fix: on failure also rewind the journal segment (reset `open.count`, truncate the file back to the pre-append offset, clear `dirty`), and add a test with a real `*os.Root`/segment; or stop serving the stream once a publish cannot be made durable.
sam marked this conversation as resolved
@ -0,0 +768,4 @@
return nil, err
}
log.appendRecovered(rec)
Member

🟠 IMPORTANT — recovery does not validate a frame's seq before feeding it to appendRecovered, so a corrupt/tampered segment can OOM the server during startup.

appendRecovered sizes its chunk array to the slot rec.seq lands on (for len(chunks) <= pos/size { append }) and its doc comment explicitly says the caller must validate rec.seq first ("an out-of-range value would size the chunk array to match"). This caller, loadRecords, never bounds rec.seq. A corrupt frame with seq far above next allocates a huge chunk array; one with seq below base makes position() (seq - base + base%size) underflow uint64 to a huge index, also forcing a massive allocation.

That contradicts the documented contract that a corrupt frame "refuses that stream rather than misreading it" — only a decode error refuses; a well-formed frame with a bad seq is accepted.

Fix: before appendRecovered, reject any rec.seq outside the segment's expected window (bound it) and return a corruption error.

🟠 IMPORTANT — recovery does not validate a frame's `seq` before feeding it to `appendRecovered`, so a corrupt/tampered segment can OOM the server during startup. `appendRecovered` sizes its chunk array to the slot `rec.seq` lands on (`for len(chunks) <= pos/size { append }`) and its doc comment explicitly says the caller must validate `rec.seq` first ("an out-of-range value would size the chunk array to match"). This caller, `loadRecords`, never bounds `rec.seq`. A corrupt frame with `seq` far above `next` allocates a huge chunk array; one with `seq` below `base` makes `position()` (`seq - base + base%size`) underflow uint64 to a huge index, also forcing a massive allocation. That contradicts the documented contract that a corrupt frame "refuses that stream rather than misreading it" — only a decode error refuses; a well-formed frame with a bad seq is accepted. Fix: before `appendRecovered`, reject any `rec.seq` outside the segment's expected window (bound it) and return a corruption error.
sam marked this conversation as resolved
Address review comments 85-87:

- appendPublish now captures the pre-append file offset/count/lastTS and,
  on a failed write or durable sync, truncates the open segment back to
  exactly its prior state via rewindAppendLocked. A failed publish no
  longer leaves the on-disk log a frame ahead of the in-memory log, so a
  retry reuses the same sequence number as a fresh append and a restart
  cannot replay two frames with the same seq (double-counted bytes /
  overwritten record). A test-only failSync seam reproduces the failure
  on a real segment; TestPublishSyncFailureRewindsJournalChain covers it.

- loadRecords: validate each frame's seq against the segment's
  [base, base+segmentRecords) window before appendRecovered, refusing a
  corrupt/tampered seq that would otherwise force an unbounded chunk
  allocation at startup. TestRecoveryRejectsOutOfWindowSeq covers it.

- createConsumer/deleteConsumer: the in-memory slot reservation now
  happens under the server's global lock but the durable slot write,
  tombstone, and watermark fsyncs run after it is released, so a slow
  disk does not serialize every stream operation. consumerStore ops are
  nil-safe against the concurrent deleteStream-close this allows.
  addConsumerIfRoom reports created; deleteConsumer reports existed/
  moved/base for the caller to persist.
pi-agent approved these changes 2026-09-20 13:20:40 +00:00
Dismissed
pi-agent left a comment

Reviewing d609bc4 (head of feature/server-persistence). The three REQUEST_CHANGES comments from review #6 are addressed:

  • #85 (IMPORTANT, publish journal desync): appendPublish now captures the pre-append file offset/count/lastTS and, on a failed write or durable sync, calls rewindAppendLocked to truncate the open segment back to exactly its prior state. A failed publish no longer leaves the on-disk log a frame ahead of the in-memory log, so a retry reuses the same sequence number as a fresh append and a restart cannot replay two frames with the same seq (bytes double-counted / record overwritten). A test-only failSync seam in syncLocked reproduces the durable-sync failure on a real *os.Root segment; new TestPublishSyncFailureRewindsJournalChain confirms the rewind and that recovery yields exactly one seq-3 frame.
  • #86 (IMPORTANT, recovery seq validation): loadRecords now validates each frame's seq against the segment's [base, base+segmentRecords) window before appendRecovered, returning a corruption error for an out-of-window seq instead of forcing an unbounded chunk allocation at startup. New TestRecoveryRejectsOutOfWindowSeq covers it (seq chosen to pass the segment-level count scan so only the per-frame validation catches it).
  • #87 (SUGGESTION, consumer I/O under global lock): createConsumer/deleteConsumer now reserve the slot under the server's global lock but run the durable slot write, tombstone, and watermark fsyncs after releasing it, matching the PR's own move of createStream's disk mount. addConsumerIfRoom reports whether the consumer was created; deleteConsumer reports existed/moved/base for the caller to persist. consumerStore.consumerSlot/deleteConsumer are now nil-safe against the concurrent deleteStream-close this allows.

Verified: gofmt, go vet ./..., go build ./..., and go test -count=1 -timeout 120s . all pass (the full package suite completes in ~1s; the env QUIC/libp2p hang is unrelated). (Same resolver caveat as before: this Forgejo build exposes no API/MCP action to set resolver=true per thread, so mark the threads resolved in the UI; code-wise all are addressed.)

Reviewing d609bc4 (head of feature/server-persistence). The three REQUEST_CHANGES comments from review #6 are addressed: - **#85 (IMPORTANT, publish journal desync):** `appendPublish` now captures the pre-append file offset/count/lastTS and, on a failed write or durable sync, calls `rewindAppendLocked` to truncate the open segment back to exactly its prior state. A failed publish no longer leaves the on-disk log a frame ahead of the in-memory log, so a retry reuses the same sequence number as a fresh append and a restart cannot replay two frames with the same seq (bytes double-counted / record overwritten). A test-only `failSync` seam in `syncLocked` reproduces the durable-sync failure on a real `*os.Root` segment; new `TestPublishSyncFailureRewindsJournalChain` confirms the rewind and that recovery yields exactly one seq-3 frame. - **#86 (IMPORTANT, recovery seq validation):** `loadRecords` now validates each frame's `seq` against the segment's `[base, base+segmentRecords)` window before `appendRecovered`, returning a corruption error for an out-of-window seq instead of forcing an unbounded chunk allocation at startup. New `TestRecoveryRejectsOutOfWindowSeq` covers it (seq chosen to pass the segment-level count scan so only the per-frame validation catches it). - **#87 (SUGGESTION, consumer I/O under global lock):** `createConsumer`/`deleteConsumer` now reserve the slot under the server's global lock but run the durable slot write, tombstone, and watermark fsyncs after releasing it, matching the PR's own move of `createStream`'s disk mount. `addConsumerIfRoom` reports whether the consumer was created; `deleteConsumer` reports existed/moved/base for the caller to persist. `consumerStore.consumerSlot`/`deleteConsumer` are now nil-safe against the concurrent `deleteStream`-close this allows. Verified: `gofmt`, `go vet ./...`, `go build ./...`, and `go test -count=1 -timeout 120s .` all pass (the full package suite completes in ~1s; the env QUIC/libp2p hang is unrelated). (Same resolver caveat as before: this Forgejo build exposes no API/MCP action to set `resolver=true` per thread, so mark the threads resolved in the UI; code-wise all are addressed.)
A consume loop exits by releasing its lease, but the loop's final
in-flight pull (already sent to the server) can be processed after that
release lands. acquireLease saw the freshly-zeroed lease and re-acquired
the same token, orphaning the consumer: the replacement loop's pull then
got ErrConsumerBusy, which is terminal, so the loop died and the test
hung waiting for it to take over.

Track a retiredLease token on the server. Once a loop releases its
lease that token can never be re-acquired; a stale pull carrying it is
refused as busy. A genuine replacement loop (fresh token) still acquires
normally.

Also harden the takeover test: capture the first loop's own error channel
before the second loop replaces it, and surface a replacement-loop
terminal error instead of hanging on it.
pi-agent requested changes 2026-09-20 13:34:00 +00:00
Dismissed
pi-agent left a comment

Root-caused and fixed the flaky TestSingleConsumeLoopOverTheWire hang.

Root cause (real, not environmental): when a consume loop exits it sends an explicit releaseLease. But its final in-flight pull (already written to the server) can be processed after the release lands. acquireLease saw the freshly-zeroed lease and re-acquired the same token, orphaning the lease. The replacement loop's pull then got ErrConsumerBusy, which is terminal in the loop, so it died and the test hung waiting for takeover.

Fix (state.go): track a retiredLease token. Once a loop releases its lease, that token can never be acquired again; a stale pull carrying it is refused as busy. A genuine replacement loop (fresh token) still acquires normally.

Also hardened the test (integration_test.go): capture the first loop's own error channel before the second loop replaces it (so the wait is on the first loop's stop/release, not the already-closed second channel), and surface a replacement-loop terminal error instead of hanging.

Verification: go build, go vet, gofmt clean. Full suite passed 30 consecutive runs at -parallel 4, 15 at -parallel 8, plus 20 solo runs of the target test. The prior flake (previously dismissed as transport/libp2p noise) no longer reproduces.

Root-caused and fixed the flaky `TestSingleConsumeLoopOverTheWire` hang. **Root cause (real, not environmental):** when a consume loop exits it sends an explicit `releaseLease`. But its *final in-flight pull* (already written to the server) can be processed *after* the release lands. `acquireLease` saw the freshly-zeroed lease and re-acquired the same token, **orphaning** the lease. The replacement loop's pull then got `ErrConsumerBusy`, which is *terminal* in the loop, so it died and the test hung waiting for takeover. **Fix (`state.go`):** track a `retiredLease` token. Once a loop releases its lease, that token can never be acquired again; a stale pull carrying it is refused as busy. A genuine replacement loop (fresh token) still acquires normally. **Also hardened the test (`integration_test.go`):** capture the first loop's own error channel before the second loop replaces it (so the wait is on the *first* loop's stop/release, not the already-closed second channel), and surface a replacement-loop terminal error instead of hanging. **Verification:** `go build`, `go vet`, `gofmt` clean. Full suite passed 30 consecutive runs at `-parallel 4`, 15 at `-parallel 8`, plus 20 solo runs of the target test. The prior flake (previously dismissed as transport/libp2p noise) no longer reproduces.
@ -102,0 +288,4 @@
// Nothing is recovered at construction: a stream opens on first access.
if server.storeRoot != nil {
st := newStore(server.storeRoot, server.logger, server.flushInterval)
server.store = st
Member

IMPORTANT - nil deref on shutdown. In acquire, a first-touch inserts a residentStream with stream == nil into s.streams before store.openStreamState completes. If Server.Close() runs in that window, rs.stream is nil and stream.stopAgeTrim() panics (nil receiver). Fix: guard the loop (skip nil streams); the still-loading journal is covered by store.close/flush.

IMPORTANT - nil deref on shutdown. In acquire, a first-touch inserts a residentStream with stream == nil into s.streams before store.openStreamState completes. If Server.Close() runs in that window, rs.stream is nil and stream.stopAgeTrim() panics (nil receiver). Fix: guard the loop (skip nil streams); the still-loading journal is covered by store.close/flush.
sam marked this conversation as resolved
@ -64,0 +185,4 @@
ch := rs.loading
s.lock.Unlock()
select {
Member

SUGGESTION - evictIdleStreams has no runtime caller: only tests invoke it directly. So the eviction half of lazy residency is inert in production - streams load on first touch but nothing ever evicts them, leaving idleEvictable/residentStream.evict dead at runtime and resident memory unbounded on a long-running server. If auto-eviction is intended (Phase 4/5), wire a scheduler; otherwise document that eviction is test-only for now.

SUGGESTION - evictIdleStreams has no runtime caller: only tests invoke it directly. So the eviction half of lazy residency is inert in production - streams load on first touch but nothing ever evicts them, leaving idleEvictable/residentStream.evict dead at runtime and resident memory unbounded on a long-running server. If auto-eviction is intended (Phase 4/5), wire a scheduler; otherwise document that eviction is test-only for now.
sam marked this conversation as resolved
@ -228,3 +500,4 @@
target = rs.stream
}
}
Member

BLOCKING - deleteStream builds a filesystem path from an unvalidated req.name, unlike createStream (line 431) and createConsumer (line 665) which call validName. On a durable server a non-resident (cold) stream falls through to s.store.removeStream(req.name) -> removeQuiet(streamsDir + '/' + name) -> root.RemoveAll('streams/'+name). Go's os.Root.splitPath strips empty and '.' path components, so a frame carrying name='.' (or an empty name) resolves to the streams/ directory itself and removes every stream - a single unauthenticated delete frame destroys all durable data. Fix: reject with validName(req.name) (errInvalidFrame) at the top of deleteStream (and make getStream/streamExists reject too), mirroring createStream.

BLOCKING - deleteStream builds a filesystem path from an unvalidated req.name, unlike createStream (line 431) and createConsumer (line 665) which call validName. On a durable server a non-resident (cold) stream falls through to s.store.removeStream(req.name) -> removeQuiet(streamsDir + '/' + name) -> root.RemoveAll('streams/'+name). Go's os.Root.splitPath strips empty and '.' path components, so a frame carrying name='.' (or an empty name) resolves to the streams/ directory itself and removes every stream - a single unauthenticated delete frame destroys all durable data. Fix: reject with validName(req.name) (errInvalidFrame) at the top of deleteStream (and make getStream/streamExists reject too), mirroring createStream.
sam marked this conversation as resolved
@ -0,0 +416,4 @@
}
log, err := j.loadRecords(ctx)
if err != nil {
Member

IMPORTANT - journal/FD leak on recovery failure. openStream registers the journal in store.streams and leaves the consumer slot files + open segment file open; openStreamState then returns an error from readStreamMeta/loadRecords without closing it. On the caller side, Server.acquire only deletes the server-level streams entry on error - it never closes the journal or removes it from store.streams. So a persistently corrupt/torn stream leaks one journal + open FDs into the store live set on every touch, and flushConsumers/flushJournals keep syncing it. Fix: deregister and j.close() on the post-registration error path (or register the journal only after the full open succeeds).

IMPORTANT - journal/FD leak on recovery failure. openStream registers the journal in store.streams and leaves the consumer slot files + open segment file open; openStreamState then returns an error from readStreamMeta/loadRecords without closing it. On the caller side, Server.acquire only deletes the server-level streams entry on error - it never closes the journal or removes it from store.streams. So a persistently corrupt/torn stream leaks one journal + open FDs into the store live set on every touch, and flushConsumers/flushJournals keep syncing it. Fix: deregister and j.close() on the post-registration error path (or register the journal only after the full open succeeds).
sam marked this conversation as resolved
feat: lazy stream residency (phase 3)
All checks were successful
code-review / Agent review (pull_request) Successful in 9s
433d315f96
Replace eager startup recovery with a stream manager that opens a stream
only on first access and evicts cold streams.

- NewServer no longer recovers every stream; nothing is read from the
  store until a stream is touched.
- A resident stream carries a refcount, single-flight loading (concurrent
  first-touches coalesce into one open) and a last-used timestamp.
- Every request path acquires a stream handle and releases it when done;
  a blocked pull holds a handle across its wait.
- Idle durable streams are evicted (flushed, file handles closed, age
  loop stopped) and reload as a fresh object on the next touch. In-memory
  streams are never evicted.
- Consumer objects are lazy: a stream open loads only the consumer index
  and the resident watermark array; a consumerState materializes on first
  use. Interest trim reads the durable watermark array so it still sees
  never-materialized consumers.
- The removed eager recovery helpers (store.recover/recoverStream) are
  replaced by store.openStreamState and store.removeStream.

Phase 3 tests cover not-loaded-until-touch, single-flight, eviction and
identical reload, eviction blocked by a live lease, lazy consumers,
touch-only residency across restart, and a full consume loop surviving
eviction without redelivering an acked record.
pi-agent requested changes 2026-09-20 17:29:52 +00:00
Dismissed
pi-agent left a comment

AI code review of feature/server-persistence at head 433d315 (Phase 3 lazy residency on top of the already-reviewed Phases 1-2). Reviewed the full current diff (store.go, state.go, server.go, memlog.go, protocol.go, conn.go, stream.go + tests) against the golang review passes. The design is coherent and well-documented: stream-directory-as-storage-unit, derived base/next, double-buffered CRC watermark, *os.Root confinement, single-flight lazy acquire/release, and strong recovery tests. Prior review threads (publish-journal desync, recovery seq validation, consumer I/O under global lock, trimTo stale-count, lifecycle) all appear addressed in the current head.

New findings in the Phase 3 lazy-residency surface:

  • BLOCKING: deleteStream builds a filesystem path from an unvalidated name — a dot/empty name empties the whole storage tree.
  • IMPORTANT: Server.Close dereferences a nil stream when a load is in flight.
  • IMPORTANT: a stream that fails to open leaks its journal + FDs into the store live set on every touch.
  • SUGGESTION: evictIdleStreams is never wired to a runtime scheduler.

No go.mod changes (no new dependencies; no CVE surface added).

AI code review of feature/server-persistence at head 433d315 (Phase 3 lazy residency on top of the already-reviewed Phases 1-2). Reviewed the full current diff (store.go, state.go, server.go, memlog.go, protocol.go, conn.go, stream.go + tests) against the golang review passes. The design is coherent and well-documented: stream-directory-as-storage-unit, derived base/next, double-buffered CRC watermark, *os.Root confinement, single-flight lazy acquire/release, and strong recovery tests. Prior review threads (publish-journal desync, recovery seq validation, consumer I/O under global lock, trimTo stale-count, lifecycle) all appear addressed in the current head. New findings in the Phase 3 lazy-residency surface: - BLOCKING: deleteStream builds a filesystem path from an unvalidated name — a dot/empty name empties the whole storage tree. - IMPORTANT: Server.Close dereferences a nil stream when a load is in flight. - IMPORTANT: a stream that fails to open leaks its journal + FDs into the store live set on every touch. - SUGGESTION: evictIdleStreams is never wired to a runtime scheduler. No go.mod changes (no new dependencies; no CVE surface added).
@ -102,0 +288,4 @@
// Nothing is recovered at construction: a stream opens on first access.
if server.storeRoot != nil {
st := newStore(server.storeRoot, server.logger, server.flushInterval)
server.store = st
Member

IMPORTANT - nil deref on shutdown. In acquire, a first-touch inserts a residentStream with stream == nil into s.streams before store.openStreamState completes. If Server.Close() runs in that window, rs.stream is nil and stream.stopAgeTrim() panics (nil receiver). Fix: guard the loop (skip nil streams); the still-loading journal is covered by store.close/flush.

IMPORTANT - nil deref on shutdown. In acquire, a first-touch inserts a residentStream with stream == nil into s.streams before store.openStreamState completes. If Server.Close() runs in that window, rs.stream is nil and stream.stopAgeTrim() panics (nil receiver). Fix: guard the loop (skip nil streams); the still-loading journal is covered by store.close/flush.
sam marked this conversation as resolved
@ -64,0 +185,4 @@
ch := rs.loading
s.lock.Unlock()
select {
Member

SUGGESTION - evictIdleStreams has no runtime caller: only tests invoke it directly. So the eviction half of lazy residency is inert in production - streams load on first touch but nothing ever evicts them, leaving idleEvictable/residentStream.evict dead at runtime and resident memory unbounded on a long-running server. If auto-eviction is intended (Phase 4/5), wire a scheduler; otherwise document that eviction is test-only for now.

SUGGESTION - evictIdleStreams has no runtime caller: only tests invoke it directly. So the eviction half of lazy residency is inert in production - streams load on first touch but nothing ever evicts them, leaving idleEvictable/residentStream.evict dead at runtime and resident memory unbounded on a long-running server. If auto-eviction is intended (Phase 4/5), wire a scheduler; otherwise document that eviction is test-only for now.
sam marked this conversation as resolved
@ -228,3 +500,4 @@
target = rs.stream
}
}
Member

BLOCKING - deleteStream builds a filesystem path from an unvalidated req.name, unlike createStream (line 431) and createConsumer (line 665) which call validName. On a durable server a non-resident (cold) stream falls through to s.store.removeStream(req.name) -> removeQuiet(streamsDir + '/' + name) -> root.RemoveAll('streams/'+name). Go's os.Root.splitPath strips empty and '.' path components, so a frame carrying name='.' (or an empty name) resolves to the streams/ directory itself and removes every stream - a single unauthenticated delete frame destroys all durable data. Fix: reject with validName(req.name) (errInvalidFrame) at the top of deleteStream (and make getStream/streamExists reject too), mirroring createStream.

BLOCKING - deleteStream builds a filesystem path from an unvalidated req.name, unlike createStream (line 431) and createConsumer (line 665) which call validName. On a durable server a non-resident (cold) stream falls through to s.store.removeStream(req.name) -> removeQuiet(streamsDir + '/' + name) -> root.RemoveAll('streams/'+name). Go's os.Root.splitPath strips empty and '.' path components, so a frame carrying name='.' (or an empty name) resolves to the streams/ directory itself and removes every stream - a single unauthenticated delete frame destroys all durable data. Fix: reject with validName(req.name) (errInvalidFrame) at the top of deleteStream (and make getStream/streamExists reject too), mirroring createStream.
sam marked this conversation as resolved
@ -0,0 +416,4 @@
}
log, err := j.loadRecords(ctx)
if err != nil {
Member

IMPORTANT - journal/FD leak on recovery failure. openStream registers the journal in store.streams and leaves the consumer slot files + open segment file open; openStreamState then returns an error from readStreamMeta/loadRecords without closing it. On the caller side, Server.acquire only deletes the server-level streams entry on error - it never closes the journal or removes it from store.streams. So a persistently corrupt/torn stream leaks one journal + open FDs into the store live set on every touch, and flushConsumers/flushJournals keep syncing it. Fix: deregister and j.close() on the post-registration error path (or register the journal only after the full open succeeds).

IMPORTANT - journal/FD leak on recovery failure. openStream registers the journal in store.streams and leaves the consumer slot files + open segment file open; openStreamState then returns an error from readStreamMeta/loadRecords without closing it. On the caller side, Server.acquire only deletes the server-level streams entry on error - it never closes the journal or removes it from store.streams. So a persistently corrupt/torn stream leaks one journal + open FDs into the store live set on every touch, and flushConsumers/flushJournals keep syncing it. Fix: deregister and j.close() on the post-registration error path (or register the journal only after the full open succeeds).
sam marked this conversation as resolved
fix: address phase 3 code-review findings
All checks were successful
code-review / Agent review (pull_request) Successful in 9s
f7c2dca371
- BLOCKING (deleteStream): a delete stream naming '.' or an empty name
  fell through to store.removeStream on a cold stream and would RemoveAll
  the whole streams/ tree. deleteStream and getStream now reject a name
  that fails validName (errInvalidFrame), mirroring createStream.
- IMPORTANT (Server.Close nil deref): Close while a first-touch load was
  still in flight (resident stream == nil) dereferenced nil. It now skips
  nil streams; the still-loading journal is covered by store.close.
- IMPORTANT (journal/FD leak on failed open): openStream now closes the
  consumer slot files when loadLog/consumers.load fails, and
  openStreamState deregisters (journal.evict) + closes the journal when a
  post-registration read fails, so a corrupt stream no longer leaks a
  journal plus open FDs into the store's live set on every touch.
- SUGGESTION (eviction was inert): wired a residencyLoop goroutine that
  periodically evicts resident streams idle beyond streamIdleEvictThreshold,
  so a long-running server reclaims cold streams; Phase 4 replaces the idle
  timeout with the storage budget as the eviction driver.

New tests: TestDeleteStreamRejectsInvalidName,
TestServerCloseSkipsLoadingStream, TestOpenStreamFailureDoesNotLeakJournal.
pi-agent approved these changes 2026-09-20 17:33:39 +00:00
pi-agent left a comment

All four findings from the Phase 3 review (head 433d315) are addressed in f7c2dca.

  • BLOCKING #96 (deleteStream path traversal): deleteStream and getStream now reject any name failing validName with errInvalidFrame, mirroring createStream. A frame naming '.'/empty can no longer reach store.removeStream/RemoveAll and wipe the storage tree. Covered by TestDeleteStreamRejectsInvalidName (deletes '. ``, empty, .., a/b, .hidden` are refused; the valid stream survives).
  • IMPORTANT #97 (Close nil deref): Server.Close now skips resident streams whose load is still in flight (stream == nil); the still-loading journal is covered by store.close. Covered by TestServerCloseSkipsLoadingStream.
  • IMPORTANT #98 (journal/FD leak on failed open): openStream closes the consumer slot files if loadLog/consumers.load fails, and openStreamState deregisters (journal.evict) + closes the journal when a post-registration read (readStreamMeta/loadRecords) fails. A corrupt stream no longer leaks a journal + open FDs into the store live set on every touch. Covered by TestOpenStreamFailureDoesNotLeakJournal (corrupts stream.meta, asserts the journal is not left registered).
  • SUGGESTION #99 (eviction inert): wired a residencyLoop goroutine launched in NewServer (durable servers only) that periodically evicts resident streams idle beyond streamIdleEvictThreshold (2m), so a long-running server reclaims cold streams. Phase 4 will replace the fixed idle timeout with the node storage budget as the eviction driver.

Verified: gofmt, go vet ./..., go build ./..., and the full package suite (5 runs at -parallel 8) all pass.

All four findings from the Phase 3 review (head 433d315) are addressed in f7c2dca. - **BLOCKING #96 (deleteStream path traversal):** `deleteStream` and `getStream` now reject any name failing `validName` with `errInvalidFrame`, mirroring `createStream`. A frame naming `'.'`/empty can no longer reach `store.removeStream`/`RemoveAll` and wipe the storage tree. Covered by `TestDeleteStreamRejectsInvalidName` (deletes `'. ``, empty, `..`, `a/b`, `.hidden` are refused; the valid stream survives). - **IMPORTANT #97 (Close nil deref):** `Server.Close` now skips resident streams whose load is still in flight (`stream == nil`); the still-loading journal is covered by `store.close`. Covered by `TestServerCloseSkipsLoadingStream`. - **IMPORTANT #98 (journal/FD leak on failed open):** `openStream` closes the consumer slot files if `loadLog`/`consumers.load` fails, and `openStreamState` deregisters (`journal.evict`) + closes the journal when a post-registration read (`readStreamMeta`/`loadRecords`) fails. A corrupt stream no longer leaks a journal + open FDs into the store live set on every touch. Covered by `TestOpenStreamFailureDoesNotLeakJournal` (corrupts stream.meta, asserts the journal is not left registered). - **SUGGESTION #99 (eviction inert):** wired a `residencyLoop` goroutine launched in `NewServer` (durable servers only) that periodically evicts resident streams idle beyond `streamIdleEvictThreshold` (2m), so a long-running server reclaims cold streams. Phase 4 will replace the fixed idle timeout with the node storage budget as the eviction driver. Verified: `gofmt`, `go vet ./...`, `go build ./...`, and the full package suite (5 runs at `-parallel 8`) all pass.
pi-agent left a comment

Reviewed the full diff (store.go, server.go, state.go, memlog.go, conn.go, protocol.go, and the new test files) against the golang-* skills (safety, concurrency, error handling, correctness). The durable store is generally well built: careful validation of every on-disk header, double-buffered CRC watermarks, single-flight lazy residency, rollback on failed durable writes, and strong coverage (recovery, lazy residency, store tests).

Two issues to address before merge — a data-integrity bug in the durable publish path (#1) and a budget gap for durable + lazy-residency deployments (#2) — plus one minor suggestion. The rest of the changes look correct. Not approving yet; fix #1 first.

Reviewed the full diff (store.go, server.go, state.go, memlog.go, conn.go, protocol.go, and the new test files) against the golang-* skills (safety, concurrency, error handling, correctness). The durable store is generally well built: careful validation of every on-disk header, double-buffered CRC watermarks, single-flight lazy residency, rollback on failed durable writes, and strong coverage (recovery, lazy residency, store tests). Two issues to address before merge — a data-integrity bug in the durable publish path (#1) and a budget gap for durable + lazy-residency deployments (#2) — plus one minor suggestion. The rest of the changes look correct. Not approving yet; fix #1 first.
@ -64,0 +147,4 @@
s.streams[name] = rs
s.lock.Unlock()
stream, err := s.store.openStreamState(ctx, name)
Member

🟡 SUGGESTION — a request racing a durable createStream can get a spurious error. acquire (first-touch) does not consult s.creating, so a publish/get/pull for a name whose directory mount is still in flight calls openStreamState while the directory or stream.meta may not exist yet, returning ErrStreamNotFound (or failing mid-creation). Transient, not corrupting, but a client that issues a publish concurrently with a create on another connection can observe a spurious error. Consider returning a get-or-create-style no-op / retry while the name is in s.creating.

🟡 SUGGESTION — a request racing a durable `createStream` can get a spurious error. `acquire` (first-touch) does not consult `s.creating`, so a publish/get/pull for a name whose directory mount is still in flight calls `openStreamState` while the directory or `stream.meta` may not exist yet, returning `ErrStreamNotFound` (or failing mid-creation). Transient, not corrupting, but a client that issues a publish concurrently with a create on another connection can observe a spurious error. Consider returning a get-or-create-style no-op / retry while the name is in `s.creating`.
sam marked this conversation as resolved
@ -191,10 +500,10 @@ func (s *Server) enforceStorageBudget() {
func (s *Server) totalRetainedBytesLocked() uint64 {
var total uint64
Member

🟠 IMPORTANT — on a durable server the storage budget is only enforced against resident streams. totalRetainedBytesLocked sums only s.streams (resident), and this PR evicts cold streams to disk (evictIdleStreams), so a durable server's idle streams keep bytes on disk without counting against the defaultStorageBytes (256 MiB) bound. enforceStorageBudget's re-scan also reads the resident total, so it sees ≤ budget and never evicts. Net effect: the advertised 256 MiB default is not a real bound once streams are lazy-evicted. This looks like the documented Phase-4 "global storage budget" disk sweep, but it's worth confirming it is consciously deferred in this PR — a deployment relying on the default bound on a durable server won't actually get it.

🟠 IMPORTANT — on a durable server the storage budget is only enforced against resident streams. `totalRetainedBytesLocked` sums only `s.streams` (resident), and this PR evicts cold streams to disk (`evictIdleStreams`), so a durable server's idle streams keep bytes on disk without counting against the `defaultStorageBytes` (256 MiB) bound. `enforceStorageBudget`'s re-scan also reads the resident total, so it sees ≤ budget and never evicts. Net effect: the advertised 256 MiB default is not a real bound once streams are lazy-evicted. This looks like the documented Phase-4 "global storage budget" disk sweep, but it's worth confirming it is consciously deferred in this PR — a deployment relying on the default bound on a durable server won't actually get it.
sam marked this conversation as resolved
@ -687,0 +1150,4 @@
// publish is on disk. The write happens outside the stream lock; the
// per-stream journal serializes publishes to the same stream on disk
// latency. The append's own interest/age trims delete segments via
// trimAfterCursorAdvance and the trim hooks; nothing is written for a moved
Member

🔴 BLOCKING — concurrent publishes to the same stream can write the journal out of order and corrupt (or refuse) the stream on restart. Sequence numbers are assigned under stream.lock inside appendDedup (server.go ~1141), but the durable write happens later under the journal's own journal.mu (persistPublish → appendPublish, store.go ~921). Two different mutexes, so the order in which they are acquired is not tied to the order sequence numbers were allocated.

Concretely: two connections publish to "orders" concurrently. G1 is assigned seq N, G2 seq N+1 (in-memory order N, N+1). But G2 can take journal.mu first and persist N+1, then G1 persists N — the segment file holds [N+1, N]. appendPublish writes whatever rec.seq it is given, and rollLocked rolls a new segment at rec.seq, with no check that rec.seq == journal.next().

• If the stream was empty (open==nil), G2 rolls a segment at base N+1, then G1 writes a frame with seq N into it. On restart loadRecords rejects rec.seq < seg.base and the whole stream refuses to load (unreadable).
• Otherwise the segment ends with non-monotonic frames [N+1, N]; on recovery appendRecovered reads N+1 (next=N+2) then N (next regresses to N+1), leaving a record at/above next and a wrong next after restart.

Fix: serialize the in-memory append and the durable append under one per-stream writer so on-disk order always equals allocation order — e.g. hold a per-stream mutex across appendDedup + persistPublish for durable streams, or have appendPublish verify rec.seq == j.next() and roll/refuse otherwise. The existing rollback path handles a failed write but not an out-of-order one.

🔴 BLOCKING — concurrent publishes to the same stream can write the journal out of order and corrupt (or refuse) the stream on restart. Sequence numbers are assigned under `stream.lock` inside `appendDedup` (server.go ~1141), but the durable write happens later under the journal's own `journal.mu` (`persistPublish` → `appendPublish`, store.go ~921). Two different mutexes, so the order in which they are acquired is not tied to the order sequence numbers were allocated. Concretely: two connections publish to "orders" concurrently. G1 is assigned seq N, G2 seq N+1 (in-memory order N, N+1). But G2 can take `journal.mu` first and persist N+1, then G1 persists N — the segment file holds [N+1, N]. `appendPublish` writes whatever `rec.seq` it is given, and `rollLocked` rolls a new segment at `rec.seq`, with no check that `rec.seq == journal.next()`. • If the stream was empty (open==nil), G2 rolls a segment at base N+1, then G1 writes a frame with seq N into it. On restart `loadRecords` rejects `rec.seq < seg.base` and the whole stream refuses to load (unreadable). • Otherwise the segment ends with non-monotonic frames [N+1, N]; on recovery `appendRecovered` reads N+1 (next=N+2) then N (next regresses to N+1), leaving a record at/above `next` and a wrong `next` after restart. Fix: serialize the in-memory append and the durable append under one per-stream writer so on-disk order always equals allocation order — e.g. hold a per-stream mutex across `appendDedup` + `persistPublish` for durable streams, or have `appendPublish` verify `rec.seq == j.next()` and roll/refuse otherwise. The existing rollback path handles a failed write but not an out-of-order one.
sam marked this conversation as resolved
Enforce the WithStorage budget against the whole node, including cold
(non-resident) streams, by sweeping the storage directory rather than
summing the resident map.

- store: budgetSnapshot sweeps every stream's segment files and metadata
  (stat + header-only reads, never loading records) to yield the node total
  and the globally-oldest sealed segments; deleteSegment removes a cold
  stream's sealed segment.
- server: durable budget enforcement seeds cacheTotal from a sweep on first
  publish and evicts the node's oldest sealed segments when over budget,
  reconciling a resident stream in place via releaseBudgetTrim and deleting
  cold streams' segments directly. The in-memory path is unchanged.
- state: releaseBudgetTrim reconciles a resident stream whose oldest segment
  was budget-evicted.
- tests: node_budget_test.go (reserve/metadata counting, non-resident
  eviction, sweep does not materialize streams) and BenchmarkBudgetSweepNonResident.
fix: serialize durable publish, wait for in-flight create (review #10)
All checks were successful
code-review / Agent review (pull_request) Successful in 10s
30649142c1
Addresses review comments on PR #1 head f7c2dca:

- BLOCKING (#105): concurrent publishes to a stream no longer write the
  journal out of order. The durable publish path now holds the journal's
  per-stream writer lock across both the in-memory append (seq allocation)
  and the durable write, so on-disk order always equals allocation order;
  previously the append took stream.lock and the write took journal.mu,
  two unrelated mutexes, so a later-allocated record could reach disk first
  and leave non-monotonic frames (or a rolled segment above an earlier seq)
  that refuse to load or regress next on restart. appendPublishLocked
  exposes the locked body for this path. The failed-write rollback is now
  exact, since no concurrent publish can have grown past the record.
- IMPORTANT (#106): the durable storage budget now sweeps the whole node
  (cold, non-resident streams included) — implemented in Phase 4.
- SUGGESTION (#107): createStream reserves the name as a single-flight
  loading residentStream (the same marker acquire waits on) instead of a
  separate s.creating map, so a request racing a durable create waits for
  the mount instead of erroring ErrStreamNotFound mid-creation. Drops the
  s.creating field.

Tests: TestConcurrentDurablePublishOnDiskOrder (concurrent publishes leave
strict monotonic seqs after restart), TestAcquireWaitsForStreamBeingCreated
(acquire waits for an in-flight create).
Member

All three review-#10 comments are addressed in the new head 3064914 (on top of the Phase 4 storage-budget commit). Please mark the threads resolved in the UI (this Forgejo build exposes no resolver API).

#105 (BLOCKING — concurrent publish journal ordering). The durable publish path now holds the journal's per-stream writer lock (journal.mu) across both the in-memory append (appendDedup, which allocates the seq under stream.lock) and the durable write, so the order records reach disk always equals the order their seqs were allocated. The old code took two unrelated mutexes and could let a later-allocated seq reach disk first. appendPublish is split into appendPublishLocked for this path. Regression test TestConcurrentDurablePublishOnDiskOrder runs 16 goroutines × 100 publishes and asserts strict seq order plus correct next after a restart.

#106 (IMPORTANT — budget only against resident streams). Addressed by the preceding Phase 4 commit (72b8c51): WithStorage now enforces the node total via a disk sweep over cold, non-resident streams (store.budgetSnapshot), and evicts the globally-oldest sealed segments, so the 256 MiB default is a real bound under lazy residency. Covered by node_budget_test.go.

#107 (SUGGESTION — acquire races a durable create). createStream now reserves the name as a single-flight loading residentStream (the same marker acquire already waits on) instead of a separate s.creating map, so a publish/get/pull racing a durable create waits for the mount instead of erroring ErrStreamNotFound mid-creation. The creating field is dropped. Covered by TestAcquireWaitsForStreamBeingCreated.

Verified: gofmt, go vet ./..., go build ./..., and the full suite (go test -count=1 ./...) pass; the new concurrency test is stable across -count=3.

All three review-#10 comments are addressed in the new head `3064914` (on top of the Phase 4 storage-budget commit). Please mark the threads resolved in the UI (this Forgejo build exposes no resolver API). **#105 (BLOCKING — concurrent publish journal ordering).** The durable publish path now holds the journal's per-stream writer lock (`journal.mu`) across both the in-memory append (`appendDedup`, which allocates the seq under `stream.lock`) and the durable write, so the order records reach disk always equals the order their seqs were allocated. The old code took two unrelated mutexes and could let a later-allocated seq reach disk first. `appendPublish` is split into `appendPublishLocked` for this path. Regression test `TestConcurrentDurablePublishOnDiskOrder` runs 16 goroutines × 100 publishes and asserts strict seq order plus correct `next` after a restart. #106 (IMPORTANT — budget only against resident streams). Addressed by the preceding Phase 4 commit (`72b8c51`): `WithStorage` now enforces the node total via a disk sweep over cold, non-resident streams (`store.budgetSnapshot`), and evicts the globally-oldest sealed segments, so the 256 MiB default is a real bound under lazy residency. Covered by `node_budget_test.go`. #107 (SUGGESTION — acquire races a durable create). `createStream` now reserves the name as a single-flight loading residentStream (the same marker `acquire` already waits on) instead of a separate `s.creating` map, so a publish/get/pull racing a durable create waits for the mount instead of erroring `ErrStreamNotFound` mid-creation. The `creating` field is dropped. Covered by `TestAcquireWaitsForStreamBeingCreated`. Verified: `gofmt`, `go vet ./...`, `go build ./...`, and the full suite (`go test -count=1 ./...`) pass; the new concurrency test is stable across `-count=3`.
pi-agent left a comment

Code review — line-by-line, with reproduction

Reviewed pull request #1 of norobo/webstream (feature/server-persistence, head 3064914, base main 894f169; +6681/−545 across 24 files) against plan.md — the design document for this "server persistence" work that lives at the repository root and specifies the on-disk layout (one directory per stream under streams/), the durability table, lazy residency, and the two consumer slot files.

Verification performed: gofmt -l clean, go vet ./... clean, go test -count=1 ./... passes. Findings marked [reproduced] were confirmed with throwaway Go tests added to the package and then removed (working tree clean); each reproduction is described inline so it can be re-added as a regression test.

Terms used below. A watermark (consumerState.processedBefore) is the durable, exclusive record sequence number up to which a consumer has confirmed processing. A segment is one file of a stream's record log, named by the sequence number of its first record (20-digit zero padded). A slot is a fixed-stride entry in consumers.meta (the consumer-name → slot index) or in cursors.meta (the CRC-protected watermark values). "Resident" means materialized in the server's in-memory stream map.


BLOCKING

B1. consumerStore.deleteConsumer can free a slot a concurrent create just took — durable consumer entry and watermark lost

store.go:1948

The function mutates the free list and zeroes the cursor slot under cs.mu, then releases the mutex in order to fsync cursors.meta, then re-acquires it and writes the free state byte at idxOff. In the gap a concurrent consumerSlot (store.go:1818) pops that same slot from freeSlots, writes its own entry at the same idxOff, and sets liveSlots[slot] = true — after which the delete's re-acquired lock writes consumerFree over the new entry.

[reproduced] A 64-pair concurrent consumerSlot / deleteConsumer probe (all names distinct) left 39 of 64 newly created consumers live in RAM (nameToSlot, liveSlots) but absent from consumers.meta after a reload. The window is an fsync wide, so this is easy to hit under load.

The impact is exactly the failure the design forbids: on restart that consumer vanishes, the client re-creates it at watermark 0, and every record in [base, W) is reprocessed — losing the one value the store exists to protect (plan.md: "losing a watermark would duplicate processing"). The reverse interleaving also exists: the delete's liveSlots[slot] = false can land after the create's.

Fix: keep the free-list mutation, both WriteAt calls and their ordering inside one lock hold, and move only the Sync calls outside it — or re-verify !cs.liveSlots[slot] (and that no name maps to slot) before writing free, skipping the write when the slot was reclaimed.
Regression test: concurrent create/delete, then reopen the store and assert the RAM live set equals the reloaded one.

B2. Unvalidated req.stream reaches the filesystem

server.go:135 (acquire), store.go:368 (openStream)

createStream (server.go:884), deleteStream (server.go:983) and getStream (server.go:1088) call validName; publish, pull, ack, createConsumer, deleteConsumer, getConsumer and releaseConsumerLease do not. They pass req.stream straight to acquire → store.openStreamState → streamsDir + "/" + name with O_CREATE on consumers.meta / cursors.meta.

[reproduced] Against a durable store:

openStreamState(".")  -> error (consumers.meta: reading file header: EOF)
openStreamState("..") -> error (consumers.meta: reading file header: EOF)
streams/ entry: consumers.meta (dir=false)
streams/ entry: cursors.meta (dir=false)
root entry:    consumers.meta (dir=false)
root entry:    cursors.meta (dir=false)

*os.Root prevents escaping the storage root (so this is not a sandbox break, and only names whose parent already exists resolve at all), but a wire-supplied field creates files in streams/ and in the storage root, where only stream directories belong. Fix at the root cause: validate once in acquire, which every request path funnels through.

B3. listStorePage treats limit == 0 as limit == 1

server.go:1070

listed, err := s.store.listStreams(req.cursor, int(req.limit)+1)
hasMore := req.limit > 0 && len(listed) > int(req.limit)

listStreamsRequest.limit is documented as "zero means no cap"; with limit == 0 the +1 probe asks for exactly one name, truncates to one, and reports hasMore == false — an uncapped request silently returns a single name from a durable server. The in-memory path (listStreamNames, server.go:1040) handles 0 correctly, so the two backends disagree on the same wire message.


IMPORTANT

I1. Cold-stream delete is not tombstoned

store.go:459 (store.removeStream)

journal.remove (store.go:1170) performs the documented rename to .<name> + syncStreamsDir + best-effort RemoveAll. removeStream, used by deleteStream when the stream is not resident (server.go:1000-1005), does a bare RemoveAll + sync. A crash mid-RemoveAll leaves a half-deleted live directory: listStreams / getStream report the stream as existing (it is a valid-named directory) while openStreamState fails on the missing stream.meta. Both plan.md ("Delete: rename the stream directory to a tombstone name…") and the README claim the tombstone path. Share one helper between the two.

I2. journal.closed is written and never read; a publish racing delete recreates the stream directory

store.go:162 (declaration), store.go:1181 (assignment), store.go:937 (appendPublishLocked)

remove nils j.open and sets closed under j.mu, then releases j.mu before the rename and RemoveAll. A publish that already holds a stream handle (refs > 0, so eviction is excluded by design) can take j.mu after that, hit j.open == nil, and enter rollLocked (store.go:1003) → MkdirAll("streams/<name>/log") + create a segment — recreating streams/<name>/ after the rename, racing the RemoveAll of the tombstone. The result is a phantom stream that lists and stats as existing but fails to open. consumerSlot / deleteConsumer already guard this with idxFile == nil; the segment path has no equivalent guard, and closed is the field that was clearly intended to be it. Return an error from appendPublishLocked / flushJournal when j.closed, and keep the rename plus RemoveAll inside j.mu.

I3. Budget enforcement does whole-node disk I/O and per-segment fsyncs under the global server write lock

server.go:514 (enforceDurableStorageBudget), server.go:546 (evictDurableToBudgetLocked), store.go:1350 (budgetSnapshot)

  • budgetSnapshot readdir + stat + header-reads every stream directory and segment, and it is called while s.lock is held (server.go:519 for the seed, server.go:527 for enforcement).
  • Each cold segment eviction calls deleteSegment (store.go:1545), which fsyncs streams/<name>/log/; each resident one calls releaseBudgetTrim → persistWatermark → trimTo → syncLogDir, another fsync — still under s.lock.

One publish that trips the budget therefore stalls every other stream's publish/pull/ack for the duration of N fsyncs plus a full-directory sweep. The snapshot needs no lock (it re-reads disk) and the deletions only need the per-stream journal.mu; take the write lock for the resident reconciliation only, and bound the batch.

I4. The min-cursor cache is gone for durable streams

store.go:1917 (minLiveWatermark), reached from state.minConsumerCursorLocked

The journal != nil early return makes every trim — that is, every publish and every ack — take cs.mu and iterate the whole nameToSlot map, while the in-memory path keeps minCursor / minCursorValid precisely to avoid this. With maxConsumersPerStream raised to 65535, one record now costs up to 65k map iterations plus a mutex that ack also contends on. The invalidation rule already written for the cached path (watermarks only rise; create/delete invalidate) applies unchanged to the resident watermark array.

I5. A segment header's lastTS is never finalized — the budget's global order key is wrong

store.go:1059 (sealLocked), store.go:1003-1046 (rollLocked), store.go:1533 (segmentLastTS)

rollLocked writes the header with firstTS == lastTS == rec.timestamp, the timestamp of the segment's first record. writeRecord (store.go:907) updates lastTS in memory only, and sealLocked rewrites only the count at offset 16. So for every sealed segment, offset 28 holds a first-record timestamp, and budgetSnapshot sorts its "globally oldest segments" candidates by that value. A segment that ends newer than another stream's segment can therefore be evicted first, diverging from the DiscardOld order the plan specifies. loadSealedSegment recovers the same wrong value. Fix: write lastTS at offset 28 in sealLocked, which already rewrites the header; add a test asserting a sealed header's count and timestamps.


MINOR / CODE QUALITY

  1. Duplicated doc comment. The streamExists comment is repeated at store.go:1565 glued onto the consumerStore doc block (// streamExists reports whether… consumers.meta (the … type consumerStore struct). Looks like a botched merge. Related: journal.next() (store.go:916) exists only for store_test.go:276 — production API kept alive by a test.
  2. consumerStore.load validation gaps (store.go:1673): nameLen < 0 is dead code (the value comes from a uint8); nameLen == 0 yields a live, reachable consumer literally named ""; cursor-slot uniqueness is not checked, so two live index entries can claim the same slot and liveSlots[slot] then governs the wrong consumer's watermark. The sanitizing WriteAt of non-live slots is also not fsynced and leaves watermarks[i] / slotStates[i] at zero without marking the copy state, so "zeroed before the slot is handed out again" rests on writeback luck rather than the stated invariant.
  3. scanSegment allocates an unvalidated length (store.go:699): make([]byte, length) from the 4-byte prefix before bounding it against the remaining file size (which loadRecords does do at store.go:840). A corrupt tail of garbage rather than zeros allocates up to 4 GiB. The guard is also length < wireByteSize, and wireByteSize == 1 (protocol.go:32) while a record's minimum encoded size is 24 bytes, so a truncated frame is refused as corruption instead of being truncated as a torn tail.
  4. maxStreamsPerServer counts only resident streams (server.go:281, checked at server.go:908). Under lazy residency with idle eviction, a durable node's on-disk stream count is effectively unbounded. The compensating control the plan promises — "a per-stream reserve counts toward the budget" — does not exist in the code: sweepStream (store.go:1435) sums real file sizes only (48 bytes of headers per stream), so the 256 MiB default admits millions of empty streams.
  5. Server.Close writes s.store = nil unguarded (server.go:339) while acquire reads that field (server.go:135), a data race once a handler is in flight (the doc says the server must not be used afterwards, but -race will flag it). Also store.openStream does not check store.closed, so a load starting during close registers a journal after the close loop's snapshot and is never flushed or closed.
  6. listResponseOverhead = 9 (protocol.go:54) versus an actual fixed size of 5 (a uint32 count plus the hasMore byte). Conservative, but the comment claims exactness.
  7. listStreams reads and sorts the whole directory per page (store.go:1251): limit is applied after ReadDir(-1) and slices.Sort, so paging an unbounded set is O(n log n) per page.

Reviewed and found correct

  • Lock order: the only nesting is server.lock → state.lock and journal.mu → state.lock; every persistWatermark* / trimTo call happens after releasing state.lock, and the publish error path unlocks journal.mu before rollbackFailedPublish. No inversion, no deadlock.
  • Durable publish ordering: holding journal.mu across appendDedup plus appendPublishLocked is the right shape, and rewindAppendLocked correctly restores offset, count and lastTS so a retry reuses the same sequence number. The roll itself is not undone, but stays consistent because the new segment's base is derived from rec.seq.
  • memlog: position / trimBefore bookkeeping holds for a non-chunk-aligned segment base (base 300 ⇒ offset 44); removeTail's tail check is exact; the new chunk-array bounds guards are necessary.
  • Double-buffered watermark: decodeDoubleBuffered (higher valid copy wins, both torn ⇒ 0 ⇒ redelivery) and encodeStaleCopy's alternating copy are correct; a torn write cannot read high within a slot.
  • advanceIf keeps cursor and watermark in one critical section, and the MaxUint64 ack guard prevents cursor wrap.
  • Wire change (dropping resp.cursor, adding hasMore) is consistent across encode, decode and the fuzz seeds; maxRecordSize / listResponseOverhead bound the frames.

Suggested regression tests: concurrent create/delete-consumer asserting RAM == disk live sets (B1); limit == 0 for both backends (B3); sealed-segment header assertions for count and lastTS (I5); delete versus concurrent publish asserting no stream directory is recreated (I2).

## Code review — line-by-line, with reproduction Reviewed pull request #1 of `norobo/webstream` (`feature/server-persistence`, head `3064914`, base `main` `894f169`; +6681/−545 across 24 files) against `plan.md` — the design document for this "server persistence" work that lives at the repository root and specifies the on-disk layout (one directory per stream under `streams/`), the durability table, lazy residency, and the two consumer slot files. **Verification performed:** `gofmt -l` clean, `go vet ./...` clean, `go test -count=1 ./...` passes. Findings marked **[reproduced]** were confirmed with throwaway Go tests added to the package and then removed (working tree clean); each reproduction is described inline so it can be re-added as a regression test. **Terms used below.** A *watermark* (`consumerState.processedBefore`) is the durable, exclusive record sequence number up to which a consumer has confirmed processing. A *segment* is one file of a stream's record log, named by the sequence number of its first record (20-digit zero padded). A *slot* is a fixed-stride entry in `consumers.meta` (the consumer-name → slot index) or in `cursors.meta` (the CRC-protected watermark values). "Resident" means materialized in the server's in-memory stream map. --- ## BLOCKING ### B1. `consumerStore.deleteConsumer` can free a slot a concurrent create just took — durable consumer entry and watermark lost `store.go:1948` The function mutates the free list and zeroes the cursor slot under `cs.mu`, then **releases the mutex in order to fsync `cursors.meta`**, then re-acquires it and writes the `free` state byte at `idxOff`. In the gap a concurrent `consumerSlot` (`store.go:1818`) pops that same slot from `freeSlots`, writes its own entry at the same `idxOff`, and sets `liveSlots[slot] = true` — after which the delete's re-acquired lock writes `consumerFree` over the *new* entry. **[reproduced]** A 64-pair concurrent `consumerSlot` / `deleteConsumer` probe (all names distinct) left **39 of 64** newly created consumers live in RAM (`nameToSlot`, `liveSlots`) but absent from `consumers.meta` after a reload. The window is an fsync wide, so this is easy to hit under load. The impact is exactly the failure the design forbids: on restart that consumer vanishes, the client re-creates it at watermark 0, and every record in `[base, W)` is reprocessed — losing the one value the store exists to protect (`plan.md`: "losing a watermark would duplicate processing"). The reverse interleaving also exists: the delete's `liveSlots[slot] = false` can land after the create's. **Fix:** keep the free-list mutation, both `WriteAt` calls and their ordering inside one lock hold, and move only the `Sync` calls outside it — or re-verify `!cs.liveSlots[slot]` (and that no name maps to `slot`) before writing `free`, skipping the write when the slot was reclaimed. **Regression test:** concurrent create/delete, then reopen the store and assert the RAM live set equals the reloaded one. ### B2. Unvalidated `req.stream` reaches the filesystem `server.go:135` (`acquire`), `store.go:368` (`openStream`) `createStream` (`server.go:884`), `deleteStream` (`server.go:983`) and `getStream` (`server.go:1088`) call `validName`; `publish`, `pull`, `ack`, `createConsumer`, `deleteConsumer`, `getConsumer` and `releaseConsumerLease` do not. They pass `req.stream` straight to `acquire` → `store.openStreamState` → `streamsDir + "/" + name` with `O_CREATE` on `consumers.meta` / `cursors.meta`. **[reproduced]** Against a durable store: ``` openStreamState(".") -> error (consumers.meta: reading file header: EOF) openStreamState("..") -> error (consumers.meta: reading file header: EOF) streams/ entry: consumers.meta (dir=false) streams/ entry: cursors.meta (dir=false) root entry: consumers.meta (dir=false) root entry: cursors.meta (dir=false) ``` `*os.Root` prevents escaping the storage root (so this is not a sandbox break, and only names whose parent already exists resolve at all), but a wire-supplied field creates files in `streams/` and in the storage root, where only stream directories belong. Fix at the root cause: validate once in `acquire`, which every request path funnels through. ### B3. `listStorePage` treats `limit == 0` as `limit == 1` `server.go:1070` ```go listed, err := s.store.listStreams(req.cursor, int(req.limit)+1) hasMore := req.limit > 0 && len(listed) > int(req.limit) ``` `listStreamsRequest.limit` is documented as "zero means no cap"; with `limit == 0` the `+1` probe asks for exactly one name, truncates to one, and reports `hasMore == false` — an uncapped request silently returns a single name from a durable server. The in-memory path (`listStreamNames`, `server.go:1040`) handles 0 correctly, so the two backends disagree on the same wire message. --- ## IMPORTANT ### I1. Cold-stream delete is not tombstoned `store.go:459` (`store.removeStream`) `journal.remove` (`store.go:1170`) performs the documented rename to `.<name>` + `syncStreamsDir` + best-effort `RemoveAll`. `removeStream`, used by `deleteStream` when the stream is not resident (`server.go:1000-1005`), does a bare `RemoveAll` + sync. A crash mid-`RemoveAll` leaves a half-deleted *live* directory: `listStreams` / `getStream` report the stream as existing (it is a valid-named directory) while `openStreamState` fails on the missing `stream.meta`. Both `plan.md` ("Delete: rename the stream directory to a tombstone name…") and the README claim the tombstone path. Share one helper between the two. ### I2. `journal.closed` is written and never read; a publish racing delete recreates the stream directory `store.go:162` (declaration), `store.go:1181` (assignment), `store.go:937` (`appendPublishLocked`) `remove` nils `j.open` and sets `closed` under `j.mu`, then releases `j.mu` before the rename and `RemoveAll`. A publish that already holds a stream handle (`refs > 0`, so eviction is excluded by design) can take `j.mu` after that, hit `j.open == nil`, and enter `rollLocked` (`store.go:1003`) → `MkdirAll("streams/<name>/log")` + create a segment — recreating `streams/<name>/` *after* the rename, racing the `RemoveAll` of the tombstone. The result is a phantom stream that lists and stats as existing but fails to open. `consumerSlot` / `deleteConsumer` already guard this with `idxFile == nil`; the segment path has no equivalent guard, and `closed` is the field that was clearly intended to be it. Return an error from `appendPublishLocked` / `flushJournal` when `j.closed`, and keep the rename plus `RemoveAll` inside `j.mu`. ### I3. Budget enforcement does whole-node disk I/O and per-segment fsyncs under the global server write lock `server.go:514` (`enforceDurableStorageBudget`), `server.go:546` (`evictDurableToBudgetLocked`), `store.go:1350` (`budgetSnapshot`) - `budgetSnapshot` readdir + stat + header-reads **every** stream directory and segment, and it is called while `s.lock` is held (`server.go:519` for the seed, `server.go:527` for enforcement). - Each cold segment eviction calls `deleteSegment` (`store.go:1545`), which fsyncs `streams/<name>/log/`; each resident one calls `releaseBudgetTrim` → `persistWatermark` → `trimTo` → `syncLogDir`, another fsync — still under `s.lock`. One publish that trips the budget therefore stalls every other stream's publish/pull/ack for the duration of N fsyncs plus a full-directory sweep. The snapshot needs no lock (it re-reads disk) and the deletions only need the per-stream `journal.mu`; take the write lock for the resident reconciliation only, and bound the batch. ### I4. The min-cursor cache is gone for durable streams `store.go:1917` (`minLiveWatermark`), reached from `state.minConsumerCursorLocked` The `journal != nil` early return makes every trim — that is, **every publish and every ack** — take `cs.mu` and iterate the whole `nameToSlot` map, while the in-memory path keeps `minCursor` / `minCursorValid` precisely to avoid this. With `maxConsumersPerStream` raised to 65535, one record now costs up to 65k map iterations plus a mutex that ack also contends on. The invalidation rule already written for the cached path (watermarks only rise; create/delete invalidate) applies unchanged to the resident watermark array. ### I5. A segment header's `lastTS` is never finalized — the budget's global order key is wrong `store.go:1059` (`sealLocked`), `store.go:1003-1046` (`rollLocked`), `store.go:1533` (`segmentLastTS`) `rollLocked` writes the header with `firstTS == lastTS == rec.timestamp`, the timestamp of the segment's *first* record. `writeRecord` (store.go:907) updates `lastTS` in memory only, and `sealLocked` rewrites only the count at offset 16. So for every sealed segment, offset 28 holds a first-record timestamp, and `budgetSnapshot` sorts its "globally oldest segments" candidates by that value. A segment that ends newer than another stream's segment can therefore be evicted first, diverging from the DiscardOld order the plan specifies. `loadSealedSegment` recovers the same wrong value. Fix: write `lastTS` at offset 28 in `sealLocked`, which already rewrites the header; add a test asserting a sealed header's count *and* timestamps. --- ## MINOR / CODE QUALITY 1. **Duplicated doc comment.** The `streamExists` comment is repeated at `store.go:1565` glued onto the `consumerStore` doc block (`// streamExists reports whether… consumers.meta (the` … `type consumerStore struct`). Looks like a botched merge. Related: `journal.next()` (`store.go:916`) exists only for `store_test.go:276` — production API kept alive by a test. 2. **`consumerStore.load` validation gaps** (`store.go:1673`): `nameLen < 0` is dead code (the value comes from a `uint8`); `nameLen == 0` yields a live, reachable consumer literally named `""`; cursor-slot uniqueness is not checked, so two live index entries can claim the same slot and `liveSlots[slot]` then governs the wrong consumer's watermark. The sanitizing `WriteAt` of non-live slots is also not fsynced and leaves `watermarks[i]` / `slotStates[i]` at zero without marking the copy state, so "zeroed before the slot is handed out again" rests on writeback luck rather than the stated invariant. 3. **`scanSegment` allocates an unvalidated length** (`store.go:699`): `make([]byte, length)` from the 4-byte prefix before bounding it against the remaining file size (which `loadRecords` does do at store.go:840). A corrupt tail of garbage rather than zeros allocates up to 4 GiB. The guard is also `length < wireByteSize`, and `wireByteSize == 1` (`protocol.go:32`) while a record's minimum encoded size is 24 bytes, so a truncated frame is refused as corruption instead of being truncated as a torn tail. 4. **`maxStreamsPerServer` counts only resident streams** (`server.go:281`, checked at `server.go:908`). Under lazy residency with idle eviction, a durable node's on-disk stream count is effectively unbounded. The compensating control the plan promises — "a per-stream reserve counts toward the budget" — does not exist in the code: `sweepStream` (`store.go:1435`) sums real file sizes only (48 bytes of headers per stream), so the 256 MiB default admits millions of empty streams. 5. **`Server.Close` writes `s.store = nil` unguarded** (`server.go:339`) while `acquire` reads that field (`server.go:135`), a data race once a handler is in flight (the doc says the server must not be used afterwards, but `-race` will flag it). Also `store.openStream` does not check `store.closed`, so a load starting during `close` registers a journal after the close loop's snapshot and is never flushed or closed. 6. **`listResponseOverhead = 9`** (`protocol.go:54`) versus an actual fixed size of 5 (a `uint32` count plus the `hasMore` byte). Conservative, but the comment claims exactness. 7. **`listStreams` reads and sorts the whole directory per page** (`store.go:1251`): `limit` is applied after `ReadDir(-1)` and `slices.Sort`, so paging an unbounded set is O(n log n) per page. --- ## Reviewed and found correct - **Lock order**: the only nesting is `server.lock → state.lock` and `journal.mu → state.lock`; every `persistWatermark*` / `trimTo` call happens *after* releasing `state.lock`, and the publish error path unlocks `journal.mu` before `rollbackFailedPublish`. No inversion, no deadlock. - **Durable publish ordering**: holding `journal.mu` across `appendDedup` plus `appendPublishLocked` is the right shape, and `rewindAppendLocked` correctly restores offset, count and `lastTS` so a retry reuses the same sequence number. The roll itself is not undone, but stays consistent because the new segment's base is derived from `rec.seq`. - **memlog**: `position` / `trimBefore` bookkeeping holds for a non-chunk-aligned segment base (base 300 ⇒ offset 44); `removeTail`'s tail check is exact; the new chunk-array bounds guards are necessary. - **Double-buffered watermark**: `decodeDoubleBuffered` (higher valid copy wins, both torn ⇒ 0 ⇒ redelivery) and `encodeStaleCopy`'s alternating copy are correct; a torn write cannot read high within a slot. - **`advanceIf`** keeps cursor and watermark in one critical section, and the `MaxUint64` ack guard prevents cursor wrap. - **Wire change** (dropping `resp.cursor`, adding `hasMore`) is consistent across encode, decode and the fuzz seeds; `maxRecordSize` / `listResponseOverhead` bound the frames. **Suggested regression tests:** concurrent create/delete-consumer asserting RAM == disk live sets (B1); `limit == 0` for both backends (B3); sealed-segment header assertions for count *and* `lastTS` (I5); delete versus concurrent publish asserting no stream directory is recreated (I2).
@ -64,0 +132,4 @@
// caller must call release(name) when done. The stream is not evictable while a
// handle is held, so in-flight code can safely use the returned *streamState
// across the acquire boundary.
func (s *Server) acquire(ctx context.Context, name string) (*streamState, error) {
Member

BLOCKING — acquire is the funnel for every request path that carries a stream name, yet it never calls validName (createStream, deleteStream and getStream do). req.stream of . or .. therefore reaches openStreamState → streamsDir + "/" + name with O_CREATE, which I reproduced creating consumers.meta/cursors.meta in streams/ and in the storage root. *os.Root confines it, so it is not a sandbox escape, but a wire field must not create files outside a stream directory. Validate here, once.

**BLOCKING** — `acquire` is the funnel for every request path that carries a stream name, yet it never calls `validName` (`createStream`, `deleteStream` and `getStream` do). `req.stream` of `.` or `..` therefore reaches `openStreamState` → `streamsDir + "/" + name` with `O_CREATE`, which I reproduced creating `consumers.meta`/`cursors.meta` in `streams/` *and* in the storage root. `*os.Root` confines it, so it is not a sandbox escape, but a wire field must not create files outside a stream directory. Validate here, once.
sam marked this conversation as resolved
@ -189,0 +543,4 @@
// directly. A segment whose stream is mid-load is left for a later sweep, since
// the in-flight load would otherwise race the deletion. The caller holds
// s.lock.
func (s *Server) evictDurableToBudgetLocked(total uint64, deletable []budgetSegment) {
Member

IMPORTANT — called with s.lock held, this performs per-segment fsyncs (deleteSegment, and releaseBudgetTrim → persistWatermark → trimTo → syncLogDir) and, via the caller, a full-node budgetSnapshot readdir/stat sweep. A single publish that trips the budget stalls every other stream's publish/pull/ack for N fsyncs plus that sweep. The snapshot needs no lock; the deletions need only the per-stream journal.mu.

**IMPORTANT** — called with `s.lock` held, this performs per-segment fsyncs (`deleteSegment`, and `releaseBudgetTrim` → `persistWatermark` → `trimTo` → `syncLogDir`) and, via the caller, a full-node `budgetSnapshot` readdir/stat sweep. A single publish that trips the budget stalls every other stream's publish/pull/ack for N fsyncs plus that sweep. The snapshot needs no lock; the deletions need only the per-stream `journal.mu`.
sam marked this conversation as resolved
@ -557,0 +1067,4 @@
// listStorePage lists a page of stream names from the storage directory,
// fetching limit+1 to detect the page boundary.
func (s *Server) listStorePage(req listStreamsRequest) ([]string, bool, error) {
Member

BLOCKING — int(req.limit)+1 turns the documented "zero means no cap" into a one-name page: with limit == 0 this asks listStreams for 1 name, truncates to it, and reports hasMore == false. The in-memory path (listStreamNames) handles 0 correctly, so the two backends disagree on the same request.

**BLOCKING** — `int(req.limit)+1` turns the documented "zero means no cap" into a one-name page: with `limit == 0` this asks `listStreams` for 1 name, truncates to it, and reports `hasMore == false`. The in-memory path (`listStreamNames`) handles 0 correctly, so the two backends disagree on the same request.
sam marked this conversation as resolved
@ -0,0 +456,4 @@
// removeStream deletes a stream directory for a stream that is not resident, so
// a delete of a cold stream does not resurrect on the next load.
func (s *store) removeStream(name string) {
Member

IMPORTANT — unlike journal.remove (store.go:1170), this cold-stream delete skips the tombstone rename: a crash mid-RemoveAll leaves a half-deleted directory that listStreams/getStream report as existing (valid name, is a directory) while openStreamState fails on the missing stream.meta. plan.md and the README both promise the tombstone path; share one helper.

**IMPORTANT** — unlike `journal.remove` (store.go:1170), this cold-stream delete skips the tombstone rename: a crash mid-`RemoveAll` leaves a half-deleted directory that `listStreams`/`getStream` report as existing (valid name, is a directory) while `openStreamState` fails on the missing `stream.meta`. `plan.md` and the README both promise the tombstone path; share one helper.
sam marked this conversation as resolved
@ -0,0 +1056,4 @@
// file, turning it into a sealed (non-newest) segment. The count is advisory; a
// sealed segment's end is derived from the next segment's base. The caller holds
// journal.mu.
func (j *journal) sealLocked() error {
Member

IMPORTANT — sealLocked finalizes only the count (offset 16). The header's lastTS (offset 28) is written once in rollLocked with the first record's timestamp and never updated, yet segmentLastTS (store.go:1533) feeds it to budgetSnapshot as the globally-oldest ordering key, so eviction order is wrong across streams. Write lastTS here too and assert it in a test.

**IMPORTANT** — `sealLocked` finalizes only the count (offset 16). The header's `lastTS` (offset 28) is written once in `rollLocked` with the *first* record's timestamp and never updated, yet `segmentLastTS` (store.go:1533) feeds it to `budgetSnapshot` as the globally-oldest ordering key, so eviction order is wrong across streams. Write `lastTS` here too and assert it in a test.
sam marked this conversation as resolved
@ -0,0 +1167,4 @@
// remove deletes the stream directory (tombstone + RemoveAll) so a deleted
// stream does not resurrect, and drops it from the store's live set.
func (j *journal) remove() {
Member

IMPORTANT — closed is set here and read nowhere, and the rename/RemoveAll run after j.mu is released. A publish that already holds a stream handle (refs > 0, so eviction is excluded by design) can then enter rollLocked → MkdirAll("streams/<name>/log") and recreate the directory after the rename, producing a phantom stream that lists as existing but fails to open. Guard the segment write path on j.closed as consumerSlot/deleteConsumer guard on idxFile == nil.

**IMPORTANT** — `closed` is set here and read nowhere, and the rename/`RemoveAll` run after `j.mu` is released. A publish that already holds a stream handle (refs > 0, so eviction is excluded by design) can then enter `rollLocked` → `MkdirAll("streams/<name>/log")` and recreate the directory after the rename, producing a phantom stream that lists as existing but fails to open. Guard the segment write path on `j.closed` as `consumerSlot`/`deleteConsumer` guard on `idxFile == nil`.
sam marked this conversation as resolved
@ -0,0 +1945,4 @@
// and fsyncs cursors.meta, then marks the consumers.meta entry free and fsyncs
// it, before the ack returns. A crash between leaves a live consumer with a
// zeroed watermark (safe redelivery), never a freed slot carrying a watermark.
func (cs *consumerStore) deleteConsumer(name string) {
Member

BLOCKING — the free-list mutation and the free state byte are written under two separate lock holds, with an fsync in between. A concurrent consumerSlot can pop the slot and write its own entry at the same idxOff in that gap, and this function then overwrites it with consumerFree. Reproduced: 39 of 64 concurrent create/delete pairs left a consumer live in RAM but absent from consumers.meta after reload — its watermark is lost, so the client restarts at 0 and reprocesses [base, W). Keep both writes and the free-list mutation in one critical section (move only the Syncs out), or re-check !cs.liveSlots[slot] before writing free.

**BLOCKING** — the free-list mutation and the `free` state byte are written under two *separate* lock holds, with an fsync in between. A concurrent `consumerSlot` can pop the slot and write its own entry at the same `idxOff` in that gap, and this function then overwrites it with `consumerFree`. Reproduced: 39 of 64 concurrent create/delete pairs left a consumer live in RAM but absent from `consumers.meta` after reload — its watermark is lost, so the client restarts at 0 and reprocesses `[base, W)`. Keep both writes and the free-list mutation in one critical section (move only the `Sync`s out), or re-check `!cs.liveSlots[slot]` before writing `free`.
sam marked this conversation as resolved
@ -0,0 +1914,4 @@
// zero when none is live. It is the durable source of interest trim, which must
// see every consumer (including ones never materialized as a consumerState): a
// never-materialized consumer still holds records a trim must not drop.
func (cs *consumerStore) minLiveWatermark() uint64 {
Member

IMPORTANT — this is reached from streamState.minConsumerCursorLocked on every trim, i.e. every publish and every ack, taking cs.mu and iterating the whole nameToSlot map. The in-memory path caches the minimum (minCursor/minCursorValid) for exactly this reason; at the new 65535-consumer cap a single record can cost 65k iterations plus lock contention against acks.

**IMPORTANT** — this is reached from `streamState.minConsumerCursorLocked` on *every* trim, i.e. every publish and every ack, taking `cs.mu` and iterating the whole `nameToSlot` map. The in-memory path caches the minimum (`minCursor`/`minCursorValid`) for exactly this reason; at the new 65535-consumer cap a single record can cost 65k iterations plus lock contention against acks.
sam marked this conversation as resolved
Addresses all BLOCKING/IMPORTANT findings and the actionable MINOR items
from the line-by-line review (id 20) of feature/server-persistence at
3064914. Verified: gofmt, go vet, golangci-lint (0 new issues vs the PR
head), and the full package suite pass.

BLOCKING
- B1 consumerStore.deleteConsumer: free-list mutation, cursor zero and the
  free state byte now share one lock hold; only the two fsyncs run outside
  it, so a delete cannot overwrite an entry a concurrent create just wrote
  (the old second lock hold cost a watermark on restart). Regression:
  TestConcurrentConsumerCreateDeleteKeepsRAMDiskConsistent.
- B2 Server.acquire validates the stream name once, the funnel every request
  path goes through; "." / ".." can no longer create consumers.meta /
  cursors.meta outside a stream directory. Regression:
  TestPublishRejectsInvalidStreamName.
- B3 listStorePage no longer turns limit==0 ("no cap") into a one-name page:
  the +1 probe is only added to a positive limit. Regression:
  TestListStreamsUncappedBothBackends.

IMPORTANT
- I1 store.removeStream now tombstones via the shared removeStreamDir helper
  (rename + fsync + RemoveAll), matching journal.remove, so a crash
  mid-delete never leaves a half-deleted live stream.
- I2 journal.remove sets closed under j.mu and keeps the rename and RemoveAll
  under the same hold; appendPublishLocked refuses when closed, so a publish
  racing a delete cannot recreate the stream directory via rollLocked's
  MkdirAll. Regression: TestDeleteRacingPublishDoesNotRecreateStream.
- I3 budget enforcement sweeps and deletes cold segments outside the server
  write lock (partitioned under it), batches at 64 segments, and takes the
  lock only for resident reconciliation.
- I4 consumerStore caches the minimum live watermark (minWatermark), seeded
  on load and invalidated by create/delete and a raise of the minimum's slot,
  so every publish/ack trim reads one value instead of sweeping up to 65k
  consumers. Regression: TestMinLiveWatermarkTracksCreateDeletePut.
- I5 sealLocked finalizes the header's lastTS (offset 28) together with the
  count, so budget eviction orders by the segment's real end, not the
  first-record timestamp. Regression:
  TestSealedSegmentHeaderFinalizesCountAndLastTS.

MINOR
- Duplicated streamExists doc comment removed; journal.next() (kept alive
  only by a test) deleted.
- consumerStore.load rejects empty names and duplicate cursor slots, and
  fsyncs the slot sanitize writes so zeroing is durable before reuse.
- scanSegment bounds the frame allocation by the remaining file size and
  treats sub-minimum or oversized tails as torn (truncate), not corruption.
  Regression: TestScanSegmentTruncatesGarbageLength.
- budget sweep counts a per-stream reserve so storageBytes bounds the stream
  count under lazy residency.
- Server.Close nils s.store under the server lock; store.openStream refuses
  registration once store.closed, so a load racing close cannot leak a
  journal. Regressions: TestColdStreamRemoveDoesNotLeaveLiveDirectory,
  TestOpenStreamStateAfterCloseRefusesRegistration.
- listResponseOverhead comment documents the exact 9+1 accounting.
- listStreams pages with a bounded max-heap (O(n log limit)) instead of a
  full sort per page.
Member

Review #20 resolved — commit c4b69a0

All eight inline comments plus the MINOR items from the review description are addressed on the updated head (c4b69a0, base 894f169). Verified: gofmt clean, go vet ./... clean, golangci-lint reports 0 new issues vs the reviewed head, and the full package suite passes.

BLOCKING

  • B1 (#429, store.go deleteConsumer): the free-list mutation, the cursor zero and the free state byte now share one lock hold; only the two Sync calls run outside it. A concurrent consumerSlot can no longer pop a slot in the fsync gap and have its new entry overwritten with consumerFree. Regression: TestConcurrentConsumerCreateDeleteKeepsRAMDiskConsistent (64 concurrent create/delete pairs, RAM live set == reloaded set).
  • B2 (#430, server.go acquire): acquire validates the stream name once — it is the funnel for every request path, so . / .. can no longer create consumers.meta/cursors.meta outside a stream directory. Regression: TestPublishRejectsInvalidStreamName (asserts no slot files in streams/ or the storage root).
  • B3 (#431, server.go listStorePage): the +1 boundary probe is only added to a positive limit, so limit == 0 (no cap) returns every stream instead of a one-name page. Regression: TestListStreamsUncappedBothBackends (durable + in-memory).

IMPORTANT

  • I1 (#432, store.go removeStream): cold-stream delete now goes through the shared tombstone helper (rename + fsync streams/ + best-effort RemoveAll), matching journal.remove, so a crash mid-delete never leaves a half-deleted live stream.
  • I2 (#433, store.go journal.remove/appendPublishLocked): closed is set under j.mu and the rename + RemoveAll stay under the same hold; appendPublishLocked returns an error when closed, so a publish racing a delete cannot recreate the stream directory via rollLocked's MkdirAll. Regression: TestDeleteRacingPublishDoesNotRecreateStream.
  • I3 (#434, server.go enforceDurableStorageBudget): the disk sweep and cold-segment deletions run outside the server write lock (partitioned under it), batched at 64 segments per enforcement; the lock is taken only for resident reconciliation. A publish that trips the budget no longer stalls every stream behind whole-node stat sweeps and per-segment fsyncs.
  • I4 (#435, store.go minLiveWatermark): the durable minimum watermark is cached (minWatermark/minWatermarkSlot), seeded on load and invalidated by create/delete and by a raise of the minimum's slot — the every-publish/ack trim reads one value instead of sweeping up to 65 535 consumers. Regression: TestMinLiveWatermarkTracksCreateDeletePut.
  • I5 (#436, store.go sealLocked): the sealed header now finalizes lastTS (offset 28) together with the count, so budgetSnapshot orders global eviction by a segment's real end rather than its first-record timestamp. Regression: TestSealedSegmentHeaderFinalizesCountAndLastTS.

MINOR (review description)

  1. Duplicated streamExists doc comment removed; journal.next() (kept alive only by a test) deleted.
  2. consumerStore.load rejects empty names and duplicate cursor slots; the sanitizing slot writes are fsynced before load returns, so a freed slot is durably zero before reuse.
  3. scanSegment bounds the frame allocation by the remaining file size and treats sub-minimum or oversized tails as torn (truncated) rather than allocating up to 4 GiB. Regression: TestScanSegmentTruncatesGarbageLength.
  4. The budget sweep counts a per-stream reserve, so storageBytes bounds the stream count under lazy residency.
  5. Server.Close nils s.store under the server lock (matching the read in acquire); store.openStream refuses registration once store.closed, so a load racing close cannot leak a journal. Regression: TestOpenStreamStateAfterCloseRefusesRegistration.
  6. listResponseOverhead comment now documents the exact 9 + 1 accounting (the reviewer's "5" missed the response envelope: code + message-length + count + hasMore).
  7. listStreams pages with a bounded max-heap (O(n log limit)) instead of sorting the whole directory per page.

Note: this Forgejo build still exposes no API/MCP action to set resolver=true per thread (the resolve routes return 405), so the eight threads should be marked resolved in the UI; code-wise they are all addressed by this commit.

## Review #20 resolved — commit c4b69a0 All eight inline comments plus the MINOR items from the review description are addressed on the updated head (`c4b69a0`, base `894f169`). Verified: `gofmt` clean, `go vet ./...` clean, `golangci-lint` reports 0 new issues vs the reviewed head, and the full package suite passes. **BLOCKING** - **B1 (#429, store.go deleteConsumer):** the free-list mutation, the cursor zero and the `free` state byte now share one lock hold; only the two `Sync` calls run outside it. A concurrent `consumerSlot` can no longer pop a slot in the fsync gap and have its new entry overwritten with `consumerFree`. Regression: `TestConcurrentConsumerCreateDeleteKeepsRAMDiskConsistent` (64 concurrent create/delete pairs, RAM live set == reloaded set). - **B2 (#430, server.go acquire):** `acquire` validates the stream name once — it is the funnel for every request path, so `.` / `..` can no longer create `consumers.meta`/`cursors.meta` outside a stream directory. Regression: `TestPublishRejectsInvalidStreamName` (asserts no slot files in `streams/` or the storage root). - **B3 (#431, server.go listStorePage):** the `+1` boundary probe is only added to a positive limit, so `limit == 0` (no cap) returns every stream instead of a one-name page. Regression: `TestListStreamsUncappedBothBackends` (durable + in-memory). **IMPORTANT** - **I1 (#432, store.go removeStream):** cold-stream delete now goes through the shared tombstone helper (`rename` + fsync `streams/` + best-effort `RemoveAll`), matching `journal.remove`, so a crash mid-delete never leaves a half-deleted live stream. - **I2 (#433, store.go journal.remove/appendPublishLocked):** `closed` is set under `j.mu` and the rename + `RemoveAll` stay under the same hold; `appendPublishLocked` returns an error when `closed`, so a publish racing a delete cannot recreate the stream directory via `rollLocked`'s `MkdirAll`. Regression: `TestDeleteRacingPublishDoesNotRecreateStream`. - **I3 (#434, server.go enforceDurableStorageBudget):** the disk sweep and cold-segment deletions run outside the server write lock (partitioned under it), batched at 64 segments per enforcement; the lock is taken only for resident reconciliation. A publish that trips the budget no longer stalls every stream behind whole-node stat sweeps and per-segment fsyncs. - **I4 (#435, store.go minLiveWatermark):** the durable minimum watermark is cached (`minWatermark`/`minWatermarkSlot`), seeded on load and invalidated by create/delete and by a raise of the minimum's slot — the every-publish/ack trim reads one value instead of sweeping up to 65 535 consumers. Regression: `TestMinLiveWatermarkTracksCreateDeletePut`. - **I5 (#436, store.go sealLocked):** the sealed header now finalizes `lastTS` (offset 28) together with the count, so `budgetSnapshot` orders global eviction by a segment's real end rather than its first-record timestamp. Regression: `TestSealedSegmentHeaderFinalizesCountAndLastTS`. **MINOR (review description)** 1. Duplicated `streamExists` doc comment removed; `journal.next()` (kept alive only by a test) deleted. 2. `consumerStore.load` rejects empty names and duplicate cursor slots; the sanitizing slot writes are fsynced before load returns, so a freed slot is durably zero before reuse. 3. `scanSegment` bounds the frame allocation by the remaining file size and treats sub-minimum or oversized tails as torn (truncated) rather than allocating up to 4 GiB. Regression: `TestScanSegmentTruncatesGarbageLength`. 4. The budget sweep counts a per-stream reserve, so `storageBytes` bounds the stream count under lazy residency. 5. `Server.Close` nils `s.store` under the server lock (matching the read in `acquire`); `store.openStream` refuses registration once `store.closed`, so a load racing close cannot leak a journal. Regression: `TestOpenStreamStateAfterCloseRefusesRegistration`. 6. `listResponseOverhead` comment now documents the exact 9 + 1 accounting (the reviewer's "5" missed the response envelope: code + message-length + count + hasMore). 7. `listStreams` pages with a bounded max-heap (O(n log limit)) instead of sorting the whole directory per page. Note: this Forgejo build still exposes no API/MCP action to set `resolver=true` per thread (the resolve routes return 405), so the eight threads should be marked resolved in the UI; code-wise they are all addressed by this commit.
sam merged commit 74c236dcba into main 2026-09-24 20:46:23 +00:00
sam deleted branch feature/server-persistence 2026-09-24 20:46:23 +00:00
Sign in to join this conversation.
No reviewers
No labels
code-review
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
norobo/webstream!1
No description provided.